Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a workflow to update dependencies #490

Merged
merged 7 commits into from
Dec 21, 2020
Merged

Add a workflow to update dependencies #490

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
336db95
Add a workflow to update dependencies
mayeut Dec 20, 2020
3b465d8
Fix a few things in update workflow
mayeut Dec 20, 2020
17798c5
Use '--no-docker' to run dependency update using local pythons
mayeut Dec 20, 2020
5ffc7e4
Only run the dependency update workflow manually
mayeut Dec 20, 2020
55048c7
Update .github/workflows/update-dependencies.yml
mayeut Dec 20, 2020
13b7ed3
Create constraints for all python versions
mayeut Dec 20, 2020
d0b61cd
Use manylinux2010_x86_64:latest to update dependencies
mayeut Dec 20, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions .github/workflows/update-dependencies.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
name: Update dependencies

on: workflow_dispatch

jobs:
update-dependencies:
name: Update dependencies
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
name: Install Python 3.9
with:
python-version: 3.9
architecture: x64
- name: Install dependencies
run: python -m pip install requests pip-tools
- name: Run update
run: python ./bin/update_dependencies.py
- name: Create Pull Request
if: github.ref == 'refs/heads/master'
uses: peter-evans/create-pull-request@v3
with:
commit-message: Update dependencies
title: '[Bot] Update dependencies'
body: |
Update the versions of our dependencies.

PR generated by "Update dependencies" [workflow](https://github.com/${{github.repository}}/actions/runs/${{github.run_id}}).
branch: update-dependencies-pr
delete-branch: true
50 changes: 35 additions & 15 deletions bin/update_dependencies.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@

import configparser
import os
import shutil
import subprocess
import sys
from collections import namedtuple

import requests
Expand All @@ -12,21 +14,39 @@

# CUSTOM_COMPILE_COMMAND is a pip-compile option that tells users how to
# regenerate the constraints files
os.environ['CUSTOM_COMPILE_COMMAND'] = "bin/update_constraints.py"
subprocess.check_call([
'pip-compile',
'--allow-unsafe',
'--upgrade',
'cibuildwheel/resources/constraints.in',
])
for python_version in ['27', '35', '36', '37']:
subprocess.check_call([
f'./env{python_version}/bin/pip-compile',
'--allow-unsafe',
'--upgrade',
'cibuildwheel/resources/constraints.in',
'--output-file', f'cibuildwheel/resources/constraints-python{python_version}.txt'
])
os.environ['CUSTOM_COMPILE_COMMAND'] = "bin/update_dependencies.py"

PYTHON_VERSIONS = ['27', '35', '36', '37', '38', '39']

if '--no-docker' in sys.argv:
for python_version in PYTHON_VERSIONS:
subprocess.check_call([
f'./env{python_version}/bin/pip-compile',
'--allow-unsafe',
'--upgrade',
'cibuildwheel/resources/constraints.in',
'--output-file', f'cibuildwheel/resources/constraints-python{python_version}.txt'
])
else:
image = 'quay.io/pypa/manylinux2010_x86_64:latest'
subprocess.check_call(['docker', 'pull', image])
for python_version in PYTHON_VERSIONS:
abi_flags = '' if int(python_version) >= 38 else 'm'
python_path = f'/opt/python/cp{python_version}-cp{python_version}{abi_flags}/bin/'
subprocess.check_call([
'docker', 'run', '--rm',
'-e', 'CUSTOM_COMPILE_COMMAND',
'-v', f'{os.getcwd()}:/volume',
'--workdir', '/volume', image,
'bash', '-c',
f'{python_path}pip install pip-tools &&'
f'{python_path}pip-compile --allow-unsafe --upgrade '
'cibuildwheel/resources/constraints.in '
f'--output-file cibuildwheel/resources/constraints-python{python_version}.txt'
])

# default constraints.txt
shutil.copyfile(f'cibuildwheel/resources/constraints-python{PYTHON_VERSIONS[-1]}.txt', 'cibuildwheel/resources/constraints.txt')

Image = namedtuple('Image', [
'manylinux_version',
Expand Down
10 changes: 5 additions & 5 deletions cibuildwheel/resources/constraints-python27.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,25 @@
# This file is autogenerated by pip-compile
# To update, run:
#
# bin/update_constraints.py
# bin/update_dependencies.py
#
appdirs==1.4.4 # via virtualenv
configparser==4.0.2 # via importlib-metadata
contextlib2==0.6.0.post1 # via importlib-metadata, importlib-resources
delocate==0.8.2 # via -r cibuildwheel/resources/constraints.in
distlib==0.3.1 # via virtualenv
filelock==3.0.12 # via virtualenv
importlib-metadata==2.1.0 # via virtualenv
importlib-metadata==2.1.1 # via virtualenv
importlib-resources==3.3.0 # via virtualenv
pathlib2==2.3.5 # via importlib-metadata, importlib-resources, virtualenv
scandir==1.10.0 # via pathlib2
singledispatch==3.4.0.3 # via importlib-resources
six==1.15.0 # via pathlib2, virtualenv
typing==3.7.4.3 # via importlib-resources
virtualenv==20.2.1 # via -r cibuildwheel/resources/constraints.in
wheel==0.35.1 # via -r cibuildwheel/resources/constraints.in, delocate
virtualenv==20.2.2 # via -r cibuildwheel/resources/constraints.in
wheel==0.36.2 # via -r cibuildwheel/resources/constraints.in, delocate
zipp==1.2.0 # via importlib-metadata, importlib-resources

# The following packages are considered to be unsafe in a requirements file:
pip==20.2.4 # via -r cibuildwheel/resources/constraints.in
pip==20.3.3 # via -r cibuildwheel/resources/constraints.in
setuptools==44.1.1 # via -r cibuildwheel/resources/constraints.in
10 changes: 5 additions & 5 deletions cibuildwheel/resources/constraints-python35.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,19 @@
# This file is autogenerated by pip-compile
# To update, run:
#
# bin/update_constraints.py
# bin/update_dependencies.py
#
appdirs==1.4.4 # via virtualenv
delocate==0.8.2 # via -r cibuildwheel/resources/constraints.in
distlib==0.3.1 # via virtualenv
filelock==3.0.12 # via virtualenv
importlib-metadata==2.1.0 # via virtualenv
importlib-metadata==2.1.1 # via virtualenv
importlib-resources==3.2.1 # via virtualenv
six==1.15.0 # via virtualenv
virtualenv==20.2.1 # via -r cibuildwheel/resources/constraints.in
wheel==0.35.1 # via -r cibuildwheel/resources/constraints.in, delocate
virtualenv==20.2.2 # via -r cibuildwheel/resources/constraints.in
wheel==0.36.2 # via -r cibuildwheel/resources/constraints.in, delocate
zipp==1.2.0 # via importlib-metadata, importlib-resources

# The following packages are considered to be unsafe in a requirements file:
pip==20.2.4 # via -r cibuildwheel/resources/constraints.in
pip==20.3.3 # via -r cibuildwheel/resources/constraints.in
setuptools==50.3.2 # via -r cibuildwheel/resources/constraints.in
13 changes: 7 additions & 6 deletions cibuildwheel/resources/constraints-python36.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,20 @@
# This file is autogenerated by pip-compile
# To update, run:
#
# bin/update_constraints.py
# bin/update_dependencies.py
#
appdirs==1.4.4 # via virtualenv
delocate==0.8.2 # via -r cibuildwheel/resources/constraints.in
distlib==0.3.1 # via virtualenv
filelock==3.0.12 # via virtualenv
importlib-metadata==3.1.0 # via virtualenv
importlib-metadata==3.3.0 # via virtualenv
importlib-resources==3.3.0 # via virtualenv
six==1.15.0 # via virtualenv
virtualenv==20.2.1 # via -r cibuildwheel/resources/constraints.in
wheel==0.35.1 # via -r cibuildwheel/resources/constraints.in, delocate
typing-extensions==3.7.4.3 # via importlib-metadata
virtualenv==20.2.2 # via -r cibuildwheel/resources/constraints.in
wheel==0.36.2 # via -r cibuildwheel/resources/constraints.in, delocate
zipp==3.4.0 # via importlib-metadata, importlib-resources

# The following packages are considered to be unsafe in a requirements file:
pip==20.2.4 # via -r cibuildwheel/resources/constraints.in
setuptools==50.3.2 # via -r cibuildwheel/resources/constraints.in
pip==20.3.3 # via -r cibuildwheel/resources/constraints.in
setuptools==51.0.0 # via -r cibuildwheel/resources/constraints.in
13 changes: 7 additions & 6 deletions cibuildwheel/resources/constraints-python37.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,19 @@
# This file is autogenerated by pip-compile
# To update, run:
#
# bin/update_constraints.py
# bin/update_dependencies.py
#
appdirs==1.4.4 # via virtualenv
delocate==0.8.2 # via -r cibuildwheel/resources/constraints.in
distlib==0.3.1 # via virtualenv
filelock==3.0.12 # via virtualenv
importlib-metadata==3.1.0 # via virtualenv
importlib-metadata==3.3.0 # via virtualenv
six==1.15.0 # via virtualenv
virtualenv==20.2.1 # via -r cibuildwheel/resources/constraints.in
wheel==0.35.1 # via -r cibuildwheel/resources/constraints.in, delocate
typing-extensions==3.7.4.3 # via importlib-metadata
virtualenv==20.2.2 # via -r cibuildwheel/resources/constraints.in
wheel==0.36.2 # via -r cibuildwheel/resources/constraints.in, delocate
zipp==3.4.0 # via importlib-metadata

# The following packages are considered to be unsafe in a requirements file:
pip==20.2.4 # via -r cibuildwheel/resources/constraints.in
setuptools==50.3.2 # via -r cibuildwheel/resources/constraints.in
pip==20.3.3 # via -r cibuildwheel/resources/constraints.in
setuptools==51.0.0 # via -r cibuildwheel/resources/constraints.in
17 changes: 17 additions & 0 deletions cibuildwheel/resources/constraints-python38.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
#
# This file is autogenerated by pip-compile
# To update, run:
#
# bin/update_dependencies.py
#
appdirs==1.4.4 # via virtualenv
delocate==0.8.2 # via -r cibuildwheel/resources/constraints.in
distlib==0.3.1 # via virtualenv
filelock==3.0.12 # via virtualenv
six==1.15.0 # via virtualenv
virtualenv==20.2.2 # via -r cibuildwheel/resources/constraints.in
wheel==0.36.2 # via -r cibuildwheel/resources/constraints.in, delocate

# The following packages are considered to be unsafe in a requirements file:
pip==20.3.3 # via -r cibuildwheel/resources/constraints.in
setuptools==51.0.0 # via -r cibuildwheel/resources/constraints.in
17 changes: 17 additions & 0 deletions cibuildwheel/resources/constraints-python39.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
#
# This file is autogenerated by pip-compile
# To update, run:
#
# bin/update_dependencies.py
#
appdirs==1.4.4 # via virtualenv
delocate==0.8.2 # via -r cibuildwheel/resources/constraints.in
distlib==0.3.1 # via virtualenv
filelock==3.0.12 # via virtualenv
six==1.15.0 # via virtualenv
virtualenv==20.2.2 # via -r cibuildwheel/resources/constraints.in
wheel==0.36.2 # via -r cibuildwheel/resources/constraints.in, delocate

# The following packages are considered to be unsafe in a requirements file:
pip==20.3.3 # via -r cibuildwheel/resources/constraints.in
setuptools==51.0.0 # via -r cibuildwheel/resources/constraints.in
10 changes: 5 additions & 5 deletions cibuildwheel/resources/constraints.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,16 @@
# This file is autogenerated by pip-compile
# To update, run:
#
# bin/update_constraints.py
# bin/update_dependencies.py
#
appdirs==1.4.4 # via virtualenv
delocate==0.8.2 # via -r cibuildwheel/resources/constraints.in
distlib==0.3.1 # via virtualenv
filelock==3.0.12 # via virtualenv
six==1.15.0 # via virtualenv
virtualenv==20.2.1 # via -r cibuildwheel/resources/constraints.in
wheel==0.35.1 # via -r cibuildwheel/resources/constraints.in, delocate
virtualenv==20.2.2 # via -r cibuildwheel/resources/constraints.in
wheel==0.36.2 # via -r cibuildwheel/resources/constraints.in, delocate

# The following packages are considered to be unsafe in a requirements file:
pip==20.2.4 # via -r cibuildwheel/resources/constraints.in
setuptools==50.3.2 # via -r cibuildwheel/resources/constraints.in
pip==20.3.3 # via -r cibuildwheel/resources/constraints.in
setuptools==51.0.0 # via -r cibuildwheel/resources/constraints.in
20 changes: 10 additions & 10 deletions cibuildwheel/resources/pinned_docker_images.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,22 +1,22 @@
[x86_64]
manylinux1 = quay.io/pypa/manylinux1_x86_64:2020-11-11-0f1f128
manylinux2010 = quay.io/pypa/manylinux2010_x86_64:2020-12-03-912b0de
manylinux2014 = quay.io/pypa/manylinux2014_x86_64:2020-11-11-bc8ce45
manylinux1 = quay.io/pypa/manylinux1_x86_64:2020-12-18-c48c073
manylinux2010 = quay.io/pypa/manylinux2010_x86_64:2020-12-19-8df9e2d
manylinux2014 = quay.io/pypa/manylinux2014_x86_64:2020-12-18-cb453dc

[i686]
manylinux1 = quay.io/pypa/manylinux1_i686:2020-11-11-0f1f128
manylinux2010 = quay.io/pypa/manylinux2010_i686:2020-12-03-912b0de
manylinux2014 = quay.io/pypa/manylinux2014_i686:2020-11-11-bc8ce45
manylinux1 = quay.io/pypa/manylinux1_i686:2020-12-18-c48c073
manylinux2010 = quay.io/pypa/manylinux2010_i686:2020-12-19-8df9e2d
manylinux2014 = quay.io/pypa/manylinux2014_i686:2020-12-18-cb453dc

[pypy_x86_64]
manylinux2010 = pypywheels/manylinux2010-pypy_x86_64:2020-11-21-a03b9e9
manylinux2010 = pypywheels/manylinux2010-pypy_x86_64:2020-12-11-f1e0e80

[aarch64]
manylinux2014 = quay.io/pypa/manylinux2014_aarch64:2020-11-11-bc8ce45
manylinux2014 = quay.io/pypa/manylinux2014_aarch64:2020-12-18-cb453dc

[ppc64le]
manylinux2014 = quay.io/pypa/manylinux2014_ppc64le:2020-11-11-bc8ce45
manylinux2014 = quay.io/pypa/manylinux2014_ppc64le:2020-12-18-cb453dc

[s390x]
manylinux2014 = quay.io/pypa/manylinux2014_s390x:2020-11-11-bc8ce45
manylinux2014 = quay.io/pypa/manylinux2014_s390x:2020-12-18-cb453dc

3 changes: 2 additions & 1 deletion unit_test/dependency_constraints_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@ def test_defaults():
resources_dir = project_root / 'cibuildwheel' / 'resources'

assert dependency_constraints.base_file_path.samefile(resources_dir / 'constraints.txt')
assert dependency_constraints.get_for_python_version('3.8').samefile(resources_dir / 'constraints.txt')
assert dependency_constraints.get_for_python_version('3.99').samefile(resources_dir / 'constraints.txt')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason to still have a default constraints.txt file?

I thought the reason it's there is because 3.8 and 3.9 currently have the exact same version file?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see @joerick comments #490 (comment) on his review.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I saw, but to start customizing your own constraints, you could also start from constraints-python39.txt? constraints.txt feels like a historical remainder, somehow. But ofc, it's not wrong; I was just thinking if it's not better to get a reminder (in the form of a failing CI), when we add 3.10, and future versions to cibuildwheel?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's simpler just to have one per version. "But just copying the latest Python we support should do" I think is exactly what @YannickJadoul is referring to here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we're coming from a situation at the start where we only had one constraints.txt. Then 2.7 started acting up, and we had constraints-python2.7.txt, then 3.5, then 3.6, ... and now, there seems to be the realization that just having a separate file for all is better.

So why not just ditch the "default", then, if falling back to the default is a mistake?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not that crucial, though. Just wondering if there's a good reason. (There's nothing else to nitpick about, so I need to find something? I very much like the idea and rest of the PR :-) )

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@YannickJadoul, @henryiii, I agree and proposed to do so. Final decision left to @joerick.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. Sorry; should have added my comment to that thread, then :-)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy to go with that - that we don't bundle a constraints.txt. I think as @YannickJadoul says, the version-specific ones appeared to solve specific problems, but version-specific is probably a more robust solution. We'll have to remove this line but that's not a huge deal, I think.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm, looking at that code, I'm now seeing it's because we want to keep supporting a default constraints.txt for users.

I'm fine either way, then! As you see fit, @mayeut. Sorry about the confusion!

assert dependency_constraints.get_for_python_version('3.9').samefile(resources_dir / 'constraints-python39.txt')
assert dependency_constraints.get_for_python_version('3.6').samefile(resources_dir / 'constraints-python36.txt')
assert dependency_constraints.get_for_python_version('3.5').samefile(resources_dir / 'constraints-python35.txt')
assert dependency_constraints.get_for_python_version('2.7').samefile(resources_dir / 'constraints-python27.txt')