Cache requires-python checks & skip debug logging

[ichard26]

The requires-python check is pretty fast, but when performed for 10000 links, the checks consume a nontrival amount of time. For example, while installing (pre-cached + --dry-run) a pared down list of homeassistant dependencies (117 in total), link evaluation took 15% of the total runtime, with check_requires_python() accounting for half (7.5%) of that.

The cache can be kept pretty small as requires-python specifiers often repeat, and when they do change, it's often in chunks (or between entirely different packages). For example, setuptools has like 1500 links, but only ~30 different requires-python specifiers.

In addition, _log_skipped_link() is a hot method and unfortunately expensive as it hashes the link on every call. Fortunately, we can return early when debug logging is not enabled. In the same homeassistant run, this saves 0.7% of the runtime.

---

Command: python -m cProfile -o profile.pstats -m pip install -r temp/homeassistant/requirements.txt --dry-run

Before

After

The Link.from_json() classmethod is surprisingly expensive. I'll take a look at speeding that up next.

[ichard26]

I noticed while writing a janky demo of installing build dependencies in-process that setuptools was still taking forever to download and install. It turns out setuptools' index page is so huge that requires-python evaluation is pretty slow, taking ~100 ms on my reasonably fast machine. (For reference, the HTTP request took ~500ms.)

Command: python -m cProfile -o profile.pstats -m pip install setuptools --ignore-installed

[notatallshaw]

LGTM

[ichard26]

I'll give some time for Stéphane to take a look, but I'm planning to merge this by EoD tomorrow.

[ichard26]

I threw an old list of homeassistants' full list of dependencies at this branch. The cache was very oversized even for this insanely large dependency set. I've knocked down the maxsize to 32 from 96, and less than 1% of all checks miss the cache.

CacheInfo(hits=86380, misses=829, maxsize=32, currsize=32)
CacheInfo(hits=85938, misses=1271, maxsize=16, currsize=16)
CacheInfo(hits=85540, misses=1669, maxsize=10, currsize=10)