feat: update python versions as part of update_dependencies

[mayeut]

As mentioned in #490, a method to update CPython & PyPy on Windows/macOS.
Per @YannickJadoul criteria, the idea behind it is "clean" however, the implementation here is far from being clean and thus is opened as draft to give ideas/pointers to others as I probably won't have time to work on this.

[mayeut]

If this helps or gives other ideas, I started to rework a bit the "CPython from python.org" part for source packages in manylinux.
It's not committed anywhere yet.

from collections import namedtuple

import packaging.version
import requests


_CandidateRelease = namedtuple('_CandidateRelease', [
    'version',
    'release',
])


def _get_id(resource_uri):
    return int(resource_uri.rstrip('/').split('/')[-1])


def _get_os():
    response = requests.get('https://www.python.org/api/v2/downloads/os/?slug=source')
    response.raise_for_status()
    os_info = response.json()
    assert len(os_info) == 1
    return _get_id(os_info[0]['resource_uri'])


def _get_update(bin, os_):
    current_version = bin[0].version
    candidates = list(bin[1:])
    candidates.sort(key=lambda x: x.version, reverse=True)
    for prerelease in {False, current_version.is_prerelease}:
        for candidate in candidates:
            if candidate.version.is_prerelease != prerelease:
                continue
            # manylinux hardcodes URL now. Only the version is mutable.
            expected_url = f'https://www.python.org/ftp/python/{candidate.version.base_version}/Python-{candidate.version}.tgz'
            response = requests.get(f'https://www.python.org/api/v2/downloads/release_file/?os={os_}&release={candidate.release}')
            response.raise_for_status()
            file_info = response.json()
            for file in file_info:
                if file['url'] == expected_url:
                    return candidate.version

    return current_version


def _get_release(current_versions, os_):
    response = requests.get('https://www.python.org/api/v2/downloads/release/?version=3&is_published=true')
    response.raise_for_status()
    release_info = response.json()
    bin = {}
    for current_version in current_versions:
        bin_key = f'{current_version.major}.{current_version.minor}'
        bin[bin_key] = [_CandidateRelease(current_version, None)]
    for release in release_info:
        parts = release['name'].split()
        if parts[0].lower() != 'python':
            continue
        assert len(parts) == 2
        version = packaging.version.parse(parts[1])
        bin_key = f'{version.major}.{version.minor}'
        if bin_key not in bin.keys():
            continue
        current = bin[bin_key][0]
        if version <= current.version:
            continue
        if version.is_postrelease or version.is_devrelease:
            continue
        if version.is_prerelease and not current.version.is_prerelease:
            continue
        bin[bin_key].append(_CandidateRelease(version, _get_id(release['resource_uri'])))
    updates = []
    for current_version in current_versions:
        bin_key = f'{current_version.major}.{current_version.minor}'
        updates.append(_get_update(bin[bin_key], os_))
    return list(zip(current_versions, updates))


os_ = _get_os()
current_versions = list([packaging.version.parse(v) for v in ['3.5.1', '3.6.2', '3.7.5', '3.8.4', '3.9.0', '3.10.0a0']])
print(_get_release(current_versions, os_))

It prints:

[(<Version('3.5.1')>, <Version('3.5.10')>), (<Version('3.6.2')>, <Version('3.6.12')>), (<Version('3.7.5')>, <Version('3.7.9')>), (<Version('3.8.4')>, <Version('3.8.7')>), (<Version('3.9.0')>, <Version('3.9.1')>), (<Version('3.10.0a0')>, <Version('3.10.0a3')>)]

[joerick]

Thanks @mayeut ! I didn't know about the python.org downloads API, that certainly simplifies things! The code is here for the curious- https://github.com/python/pythondotorg/blob/master/downloads/api.py , it's a simple Django Rest Framework API.

As for integrating these into the code, I think a human/machine readable TOML config file would be best, similar to the manylinux one at cibuildwheel/resources/pinned_docker_images.cfg, instead of modifying the code in this script. @YannickJadoul I remember you had a similar idea in #431.

[mayeut]

As for integrating these into the code, I think a human/machine readable TOML config file would be best, similar to the manylinux one at cibuildwheel/resources/pinned_docker_images.cfg, instead of modifying the code in this script. @YannickJadoul I remember you had a similar idea in #431.

I did not saw (or remember seeing) #431 before but that's certainly one option which matches one of the few comment in the commit (# hugly search pattern, package configuration shall probably done otherwise if we want to do this)

[YannickJadoul]

I already completely forgot about #431. For the record, that was just a loose thought, and nothing concrete; so at any rate, we'd have to discuss a bit on how to best approach that, but yes, I agree the combination of these two ideas could be interesting!

[henryiii]

A few comments after I rewrote it a bit:

• Needs a macOS version
• I think we should require cibuildwheel dev install, so we can use things in utils, and things we normally require, like click
• rich is beautiful for debugging printouts of structures. :)

Should we have a way to opt-in to dev versions and include them in the list? Just like CIBW_MIN_PYTHON, having an upper limit or --dev flag or some other way we could control opt-in to dev would allow users to start testing 3.10a4 now, if they wanted too, and would smooth out having to turn it on in a new release. Just a thought, because we have to filter out dev versions now.

[henryiii]

Okay, think it's ready, checks macOS and Windows. Example of rich output:

Should be general enough to include Universal2 with one extra (example included) addition.

[henryiii]

@joerick GitLab CI seems to be both required, and stuck. :/

[joerick]

@joerick GitLab CI seems to be both required, and stuck. :/

I've 'fixed' it by removing that CI from being required. The way Github does the auto merge is pretty odd. I have to choose the CI checks that are 'required' in settings, which are matched by display name to the checks being run. So, as we see in #484, because I changed the name of the macos job, it's forever waiting for the old job to report. Sigh.

Maybe something 3rd party would fit better... really I think we just want 'anything that's started, wait for that to be successful' rather than choosing individual checks.

[joerick]

Wow, quite an impressive bit of work @henryiii !

general points

• I'm not sure how the 'prereleases' flag works, at the moment... should it even be a global flag? Generally we don't want them, but sometimes we want to release a beta version of cibuildwheel with a new version of Python e.g. 3.10 support - then we'd want prerelease support but only on 3.10.x versions.
• There might be a simpler way to write this script, by starting with a read of the build-platforms.toml, and then iterating through each configuration, trying to find a patch release that's newer than the one listed. This would allow us to manually edit the build-platforms.toml to add e.g. 3.10 support, or something else, and would avoid the hard-coding of 2.7 versions here. What do you think?

[henryiii]

Redesigned based on the idea of having the input file drive the search.

[joerick]

Hope you don't mind, I had a little play with this @henryiii, to give nice diff outputs. Just an excuse to play with rich, really!

Anyway, so long as you're happy with my changes, LGTM!

[henryiii]

I'm making one last style change (removing a comma), and looking to see if there's anything else I missed or any comments that might be useful. Then I'll push and trigger a pending merge!

[henryiii]

This is the "magic" comma in Black, which lets you force multiline. I didn't want to do that here, so I removed it.

PS: I originally tried to write this without Black, but the first time a ran I had 20 or so Flake8 errors; I had no interest in waisting time trying to fix them by hand, so I ran black bin/update_python.py, and they all went away. So this is our one Black'ed file. :)

[henryiii]

MyPy's handling of TypedDict's is pretty buggy. I had to do this, and I also don't know how to type narrow on a Union of TypedDicts, as assert isinstance is not allowed, so this takes a normal Dict for now. 🤷