profile| Runs pylint against external code, generating profile-heatmaps

[doublethefish]

Steps

• [-] Add yourself to CONTRIBUTORS if you are a new contributor.
• [-] Add a ChangeLog entry describing what your PR does.
• [-] If it's a new feature or an important bug fix, add a What's New entry in doc/whatsnew/<current release.rst>.
• Write a good description on what the PR does.

Description

Runs pylint against numpy and supports adding other external repos for testing.

Will generate a .svg in the .tox/prof. Here's an example:
https://pylint.s3.amazonaws.com/doublethefish/pylint/output.svg

Other considerations:

• Use https gitub uri so we do not need credentials
• Shallow clone so we don't pull more data than we need
• Ensure we skip external profiling unless explicitly wanted

Type of Changes

	Type
✓	🔨 Refactoring

Related Issue

[coveralls]

Coverage remained the same at 90.836% when pulling 899156a on doublethefish:profile/pylint_against_numpy into 3d712f2 on PyCQA:master.

[coveralls]

Coverage remained the same at 90.454% when pulling a1ad60e on doublethefish:profile/pylint_against_numpy into 67d0572 on PyCQA:master.

[Pierre-Sassoulas]

Nice addition to the code base. I think we could also clone the local pylint repository at a fixed commit (maybe the 2.4.0 tag). This way we can launch this test without having access to the internet. Most pylint developpers should also have astroid in pylint/astroid/ (we advice to clone it in the pylint directory), so that would make another repository to test on without the need to download anything.

[doublethefish]

Hey @AWhetter @PCManticore are you able to review this or shall I close the PR?

[AWhetter]

Apologies for taking so long to get around to this. I love the change! It'll make profiling much easier.
I've made a couple of suggested changes. Let me know what you think.

[brycepg]

Hey could you re-up the svg? I was hoping to get an overview before I dig in

[doublethefish]

Aha traction on all this stuff! Great stuff. Also thanks @brycepg for manually sorting out the other PR, it has been quite frustrating getting all these PRs sorted, and I lost motivation to contribute.

The file is long gone I'm afraid, but here's an example run against test_pylint_runners: svg. Having kept the other one up for months, I'm not going to be able to keep this one up longer than a couple of weeks.

That example was easy to gen on this branch (well my sept 1st rebased version anyway, I CBA to rebase again right the now). On this branch run:

python -Wi -m pytest --exitfirst \
                         --profile-svg \
                         tests/test_pylint_runners.py

If you want to run the numpy version (which wasn't working for me in 2.5), in your linux shell-like, run:

PYTEST_PROFILE_EXTERNAL=1 \
                        python -Wi -m pytest --exitfirst \
                         --profile-svg \
                         tests/profile/test_profile_against_externals.py

I would strongly recommend you run the command locally as since Pylint 2.5 (IIRC) something changed in astroid that caused a run against numpy to break, at least, that was the state of it last month.

I am super interested to find out if running pylint against numpy works again and what caused the regression (sadface) in the astroid code.

Finally, if this goes in, I may revisit some of my other patches looking at the mult-proc threading code performance which seems bottlenecked by astroid repeating work.

[doublethefish]

For the sake of clarity. My current understanding is that running pylint against numpy is broken due to some regression in astroid/pylint and this PR shouldn't go in until that regression is fixed.

Obviously, if this PR had gone in in April the regression would have likely been caught at point of merge.

[doublethefish]

Whilst logging bug #3913 I spent a bit of time looking at where the regression came in, and it is on the dev branch only (👍)

[Pierre-Sassoulas]

Sorry for the long delay, I did not see that you made the changes requested by @AWhetter ! Great merge request !