From 730e4d8b97c5d8f24eaffb3b637fe4d64f50ec32 Mon Sep 17 00:00:00 2001 From: Lukas Puehringer Date: Tue, 18 Feb 2020 12:16:00 +0100 Subject: [PATCH] Revise requirements files again Follows up on #978, which had the following problems: - too many requirements files (cc @trishankatdatadog ;) - used extra tooling around pip-compile that - didn't take into account requirement markers (see comments in requirements.txt in this commit), and - confused Dependabot, which expects the hashed requirements file in a certain format, as pip-compile would generate it without custom tooling (see #979). This commit restructures the requirements files as follows: - Merges requirements-tox.txt and requirements-test.txt. The separation was semantically correct but operationally irrelevant. - Removes the hashed requirements file, which doesn't add much security, especially with PEP 458 on the way (see python/peps#1306), but extra maintenance (see notes about requirements.txt in #978 and about Dependabot above) - Manually adds environment markers to requirements-pinned.txt (see comments in requirements.txt in this commit). Signed-off-by: Lukas Puehringer --- requirements-dev.txt | 2 +- requirements-pinned.txt | 17 ++-- requirements-test.txt | 12 ++- requirements-tox.txt | 12 --- requirements.in | 46 ----------- requirements.txt | 169 +++++++++++----------------------------- tox.ini | 4 +- 7 files changed, 66 insertions(+), 196 deletions(-) delete mode 100644 requirements-tox.txt delete mode 100644 requirements.in diff --git a/requirements-dev.txt b/requirements-dev.txt index 2501c49cf8..f8748752a5 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -1,5 +1,5 @@ # Install tuf in editable mode and requirements for local testing with tox, # and also for running test suite or individual tests manually tox --r requirements-tox.txt +-r requirements-test.txt -e . diff --git a/requirements-pinned.txt b/requirements-pinned.txt index 2668ea25b2..d06af9fc4f 100644 --- a/requirements-pinned.txt +++ b/requirements-pinned.txt @@ -1,18 +1,17 @@ -# Auto-generated (see requirements.in) certifi==2019.11.28 # via requests -cffi==1.13.2 # via cryptography, pynacl +cffi==1.14.0 # via cryptography, pynacl chardet==3.0.4 # via requests -colorama==0.4.3 -cryptography==2.8 -enum34==1.1.6 # via cryptography +colorama==0.4.3 # via securesystemslib +cryptography==2.8 # via securesystemslib +enum34==1.1.6 ; python_version < '3' # via cryptography idna==2.8 # via requests -ipaddress==1.0.23 # via cryptography +ipaddress==1.0.23 ; python_version < '3' # via cryptography iso8601==0.1.12 pycparser==2.19 # via cffi -pynacl==1.3.0 +pynacl==1.3.0 # via securesystemslib python-dateutil==2.8.1 # via securesystemslib requests==2.22.0 -securesystemslib==0.14.0 +securesystemslib[colors,crypto,pynacl]==0.14.0 six==1.14.0 -subprocess32==3.5.4 # via securesystemslib +subprocess32==3.5.4 ; python_version < '3' # via securesystemslib urllib3==1.25.8 # via requests diff --git a/requirements-test.txt b/requirements-test.txt index e05724b5eb..b3ee09c4f2 100644 --- a/requirements-test.txt +++ b/requirements-test.txt @@ -1,2 +1,12 @@ -# Install test requirements (see 'tests_require' field in setup.py) +# Install requirements needed in each tox environment + +# pinned tuf runtime dependencies (should auto-update and -trigger ci/cd) +-r requirements-pinned.txt + +# test runtime dependencies (see 'tests_require' field in setup.py) mock; python_version < "3.3" + +# additional test tools for linting and coverage measurement +coverage +pylint +bandit diff --git a/requirements-tox.txt b/requirements-tox.txt deleted file mode 100644 index 34d4dd9cf5..0000000000 --- a/requirements-tox.txt +++ /dev/null @@ -1,12 +0,0 @@ -# Install requirements needed in each tox environment - -# pinned tuf runtime dependencies (should auto-update and -trigger ci/cd) --r requirements-pinned.txt - -# test runtime dependencies --r requirements-test.txt - -# additional test tools for linting and coverage measurement -coverage -pylint -bandit diff --git a/requirements.in b/requirements.in deleted file mode 100644 index 9ad2487875..0000000000 --- a/requirements.in +++ /dev/null @@ -1,46 +0,0 @@ -# Template for tuf runtime requirements files including optional dependencies -# -# Copy-paste script below (without leading '#') to bash when requirements are -# added or removed to generate: -# -# - 'requirements.txt': lists pinned dependencies with hashes, used for -# client-side integrity checks -# -# - 'requirements-pinned.txt': lists pinned dependencies without hashes, used -# for ci/cd builds, where 'requirements.txt' can't be used together with -# other (non-runtime) requirements that don't list hashes (see pypa/pip#4995) -# -# Both files are automatically updated on GitHub with Dependabot. -# -# ----------------------------------------------------------------------------- -# # Gather pip-compile results for each supported Python version -# for v in 2.7 3.5 3.6 3.7 3.8; do -# mkvirtualenv tuf-env-${v} -p python${v}; -# pip install pip-tools; -# pip-compile requirements.in -n 2>&1 | grep -v "^#" >> requirements.combined; -# # Keep one venv as we need it below to add hashes -# if [ $v != 3.8 ]; then -# deactivate; -# rmvirtualenv tuf-env-${v}; -# fi -# done; -# -# # Create requirements-pinned.txt -# echo "# Auto-generated (see requirements.in)" > requirements-pinned.txt -# cat requirements.combined | grep -v "^Dry-run," | sort -u >> requirements-pinned.txt -# rm requirements.combined -# -# # Create requirements.txt with hashes -# pip-compile --generate-hashes -o requirements.txt requirements-pinned.txt -# -# # Remove remaining venv -# deactivate -# rmvirtualenv tuf-env-3.8 -# ----------------------------------------------------------------------------- -securesystemslib -cryptography -colorama -pynacl -requests -six -iso8601 diff --git a/requirements.txt b/requirements.txt index 1f0227cd9e..c4bcce87c6 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,128 +1,47 @@ +# TUF runtime requirements plus securesystemslib with extra dependencies # -# This file is autogenerated by pip-compile -# To update, run: +# This file together with 'pip-compile' is used to generate a pinned +# requirements file with all immediate and transitive dependencies. # -# pip-compile --generate-hashes --output-file=requirements.txt requirements-pinned.txt +# 'requirements-pinned.txt' is updated on GitHub with Dependabot, which +# triggers CI/CD builds to automatically test against updated dependencies. # -certifi==2019.11.28 \ - --hash=sha256:017c25db2a153ce562900032d5bc68e9f191e44e9a0f762f373977de9df1fbb3 \ - --hash=sha256:25b64c7da4cd7479594d035c08c2d809eb4aab3a26e5a990ea98cc450c320f1f -cffi==1.13.2 \ - --hash=sha256:0b49274afc941c626b605fb59b59c3485c17dc776dc3cc7cc14aca74cc19cc42 \ - --hash=sha256:0e3ea92942cb1168e38c05c1d56b0527ce31f1a370f6117f1d490b8dcd6b3a04 \ - --hash=sha256:135f69aecbf4517d5b3d6429207b2dff49c876be724ac0c8bf8e1ea99df3d7e5 \ - --hash=sha256:19db0cdd6e516f13329cba4903368bff9bb5a9331d3410b1b448daaadc495e54 \ - --hash=sha256:2781e9ad0e9d47173c0093321bb5435a9dfae0ed6a762aabafa13108f5f7b2ba \ - --hash=sha256:291f7c42e21d72144bb1c1b2e825ec60f46d0a7468f5346841860454c7aa8f57 \ - --hash=sha256:2c5e309ec482556397cb21ede0350c5e82f0eb2621de04b2633588d118da4396 \ - --hash=sha256:2e9c80a8c3344a92cb04661115898a9129c074f7ab82011ef4b612f645939f12 \ - --hash=sha256:32a262e2b90ffcfdd97c7a5e24a6012a43c61f1f5a57789ad80af1d26c6acd97 \ - --hash=sha256:3c9fff570f13480b201e9ab69453108f6d98244a7f495e91b6c654a47486ba43 \ - --hash=sha256:415bdc7ca8c1c634a6d7163d43fb0ea885a07e9618a64bda407e04b04333b7db \ - --hash=sha256:42194f54c11abc8583417a7cf4eaff544ce0de8187abaf5d29029c91b1725ad3 \ - --hash=sha256:4424e42199e86b21fc4db83bd76909a6fc2a2aefb352cb5414833c030f6ed71b \ - --hash=sha256:4a43c91840bda5f55249413037b7a9b79c90b1184ed504883b72c4df70778579 \ - --hash=sha256:599a1e8ff057ac530c9ad1778293c665cb81a791421f46922d80a86473c13346 \ - --hash=sha256:5c4fae4e9cdd18c82ba3a134be256e98dc0596af1e7285a3d2602c97dcfa5159 \ - --hash=sha256:5ecfa867dea6fabe2a58f03ac9186ea64da1386af2159196da51c4904e11d652 \ - --hash=sha256:62f2578358d3a92e4ab2d830cd1c2049c9c0d0e6d3c58322993cc341bdeac22e \ - --hash=sha256:6471a82d5abea994e38d2c2abc77164b4f7fbaaf80261cb98394d5793f11b12a \ - --hash=sha256:6d4f18483d040e18546108eb13b1dfa1000a089bcf8529e30346116ea6240506 \ - --hash=sha256:71a608532ab3bd26223c8d841dde43f3516aa5d2bf37b50ac410bb5e99053e8f \ - --hash=sha256:74a1d8c85fb6ff0b30fbfa8ad0ac23cd601a138f7509dc617ebc65ef305bb98d \ - --hash=sha256:7b93a885bb13073afb0aa73ad82059a4c41f4b7d8eb8368980448b52d4c7dc2c \ - --hash=sha256:7d4751da932caaec419d514eaa4215eaf14b612cff66398dd51129ac22680b20 \ - --hash=sha256:7f627141a26b551bdebbc4855c1157feeef18241b4b8366ed22a5c7d672ef858 \ - --hash=sha256:8169cf44dd8f9071b2b9248c35fc35e8677451c52f795daa2bb4643f32a540bc \ - --hash=sha256:aa00d66c0fab27373ae44ae26a66a9e43ff2a678bf63a9c7c1a9a4d61172827a \ - --hash=sha256:ccb032fda0873254380aa2bfad2582aedc2959186cce61e3a17abc1a55ff89c3 \ - --hash=sha256:d754f39e0d1603b5b24a7f8484b22d2904fa551fe865fd0d4c3332f078d20d4e \ - --hash=sha256:d75c461e20e29afc0aee7172a0950157c704ff0dd51613506bd7d82b718e7410 \ - --hash=sha256:dcd65317dd15bc0451f3e01c80da2216a31916bdcffd6221ca1202d96584aa25 \ - --hash=sha256:e570d3ab32e2c2861c4ebe6ffcad6a8abf9347432a37608fe1fbd157b3f0036b \ - --hash=sha256:fd43a88e045cf992ed09fa724b5315b790525f2676883a6ea64e3263bae6549d -chardet==3.0.4 \ - --hash=sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae \ - --hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691 -colorama==0.4.3 \ - --hash=sha256:7d73d2a99753107a36ac6b455ee49046802e59d9d076ef8e47b61499fa29afff \ - --hash=sha256:e96da0d330793e2cb9485e9ddfd918d456036c7149416295932478192f4436a1 -cryptography==2.8 \ - --hash=sha256:02079a6addc7b5140ba0825f542c0869ff4df9a69c360e339ecead5baefa843c \ - --hash=sha256:1df22371fbf2004c6f64e927668734070a8953362cd8370ddd336774d6743595 \ - --hash=sha256:369d2346db5934345787451504853ad9d342d7f721ae82d098083e1f49a582ad \ - --hash=sha256:3cda1f0ed8747339bbdf71b9f38ca74c7b592f24f65cdb3ab3765e4b02871651 \ - --hash=sha256:44ff04138935882fef7c686878e1c8fd80a723161ad6a98da31e14b7553170c2 \ - --hash=sha256:4b1030728872c59687badcca1e225a9103440e467c17d6d1730ab3d2d64bfeff \ - --hash=sha256:58363dbd966afb4f89b3b11dfb8ff200058fbc3b947507675c19ceb46104b48d \ - --hash=sha256:6ec280fb24d27e3d97aa731e16207d58bd8ae94ef6eab97249a2afe4ba643d42 \ - --hash=sha256:7270a6c29199adc1297776937a05b59720e8a782531f1f122f2eb8467f9aab4d \ - --hash=sha256:73fd30c57fa2d0a1d7a49c561c40c2f79c7d6c374cc7750e9ac7c99176f6428e \ - --hash=sha256:7f09806ed4fbea8f51585231ba742b58cbcfbfe823ea197d8c89a5e433c7e912 \ - --hash=sha256:90df0cc93e1f8d2fba8365fb59a858f51a11a394d64dbf3ef844f783844cc793 \ - --hash=sha256:971221ed40f058f5662a604bd1ae6e4521d84e6cad0b7b170564cc34169c8f13 \ - --hash=sha256:a518c153a2b5ed6b8cc03f7ae79d5ffad7315ad4569b2d5333a13c38d64bd8d7 \ - --hash=sha256:b0de590a8b0979649ebeef8bb9f54394d3a41f66c5584fff4220901739b6b2f0 \ - --hash=sha256:b43f53f29816ba1db8525f006fa6f49292e9b029554b3eb56a189a70f2a40879 \ - --hash=sha256:d31402aad60ed889c7e57934a03477b572a03af7794fa8fb1780f21ea8f6551f \ - --hash=sha256:de96157ec73458a7f14e3d26f17f8128c959084931e8997b9e655a39c8fde9f9 \ - --hash=sha256:df6b4dca2e11865e6cfbfb708e800efb18370f5a46fd601d3755bc7f85b3a8a2 \ - --hash=sha256:ecadccc7ba52193963c0475ac9f6fa28ac01e01349a2ca48509667ef41ffd2cf \ - --hash=sha256:fb81c17e0ebe3358486cd8cc3ad78adbae58af12fc2bf2bc0bb84e8090fa5ce8 -enum34==1.1.6 \ - --hash=sha256:2d81cbbe0e73112bdfe6ef8576f2238f2ba27dd0d55752a776c41d38b7da2850 \ - --hash=sha256:644837f692e5f550741432dd3f223bbb9852018674981b1664e5dc339387588a \ - --hash=sha256:6bd0f6ad48ec2aa117d3d141940d484deccda84d4fcd884f5c3d93c23ecd8c79 \ - --hash=sha256:8ad8c4783bf61ded74527bffb48ed9b54166685e4230386a9ed9b1279e2df5b1 -idna==2.8 \ - --hash=sha256:c357b3f628cf53ae2c4c05627ecc484553142ca23264e593d327bcde5e9c3407 \ - --hash=sha256:ea8b7f6188e6fa117537c3df7da9fc686d485087abf6ac197f9c46432f7e4a3c -ipaddress==1.0.23 \ - --hash=sha256:6e0f4a39e66cb5bb9a137b00276a2eff74f93b71dcbdad6f10ff7df9d3557fcc \ - --hash=sha256:b7f8e0369580bb4a24d5ba1d7cc29660a4a6987763faf1d8a8046830e020e7e2 -iso8601==0.1.12 \ - --hash=sha256:210e0134677cc0d02f6028087fee1df1e1d76d372ee1db0bf30bf66c5c1c89a3 \ - --hash=sha256:49c4b20e1f38aa5cf109ddcd39647ac419f928512c869dc01d5c7098eddede82 \ - --hash=sha256:bbbae5fb4a7abfe71d4688fd64bff70b91bbd74ef6a99d964bab18f7fdf286dd -pycparser==2.19 \ - --hash=sha256:a988718abfad80b6b157acce7bf130a30876d27603738ac39f140993246b25b3 -pynacl==1.3.0 \ - --hash=sha256:05c26f93964373fc0abe332676cb6735f0ecad27711035b9472751faa8521255 \ - --hash=sha256:0c6100edd16fefd1557da078c7a31e7b7d7a52ce39fdca2bec29d4f7b6e7600c \ - --hash=sha256:0d0a8171a68edf51add1e73d2159c4bc19fc0718e79dec51166e940856c2f28e \ - --hash=sha256:1c780712b206317a746ace34c209b8c29dbfd841dfbc02aa27f2084dd3db77ae \ - --hash=sha256:2424c8b9f41aa65bbdbd7a64e73a7450ebb4aa9ddedc6a081e7afcc4c97f7621 \ - --hash=sha256:2d23c04e8d709444220557ae48ed01f3f1086439f12dbf11976e849a4926db56 \ - --hash=sha256:30f36a9c70450c7878053fa1344aca0145fd47d845270b43a7ee9192a051bf39 \ - --hash=sha256:37aa336a317209f1bb099ad177fef0da45be36a2aa664507c5d72015f956c310 \ - --hash=sha256:4943decfc5b905748f0756fdd99d4f9498d7064815c4cf3643820c9028b711d1 \ - --hash=sha256:53126cd91356342dcae7e209f840212a58dcf1177ad52c1d938d428eebc9fee5 \ - --hash=sha256:57ef38a65056e7800859e5ba9e6091053cd06e1038983016effaffe0efcd594a \ - --hash=sha256:5bd61e9b44c543016ce1f6aef48606280e45f892a928ca7068fba30021e9b786 \ - --hash=sha256:6482d3017a0c0327a49dddc8bd1074cc730d45db2ccb09c3bac1f8f32d1eb61b \ - --hash=sha256:7d3ce02c0784b7cbcc771a2da6ea51f87e8716004512493a2b69016326301c3b \ - --hash=sha256:a14e499c0f5955dcc3991f785f3f8e2130ed504fa3a7f44009ff458ad6bdd17f \ - --hash=sha256:a39f54ccbcd2757d1d63b0ec00a00980c0b382c62865b61a505163943624ab20 \ - --hash=sha256:aabb0c5232910a20eec8563503c153a8e78bbf5459490c49ab31f6adf3f3a415 \ - --hash=sha256:bd4ecb473a96ad0f90c20acba4f0bf0df91a4e03a1f4dd6a4bdc9ca75aa3a715 \ - --hash=sha256:bf459128feb543cfca16a95f8da31e2e65e4c5257d2f3dfa8c0c1031139c9c92 \ - --hash=sha256:e2da3c13307eac601f3de04887624939aca8ee3c9488a0bb0eca4fb9401fc6b1 \ - --hash=sha256:f67814c38162f4deb31f68d590771a29d5ae3b1bd64b75cf232308e5c74777e0 -python-dateutil==2.8.1 \ - --hash=sha256:73ebfe9dbf22e832286dafa60473e4cd239f8592f699aa5adaf10050e6e1823c \ - --hash=sha256:75bb3f31ea686f1197762692a9ee6a7550b59fc6ca3a1f4b5d7e32fb98e2da2a -requests==2.22.0 \ - --hash=sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4 \ - --hash=sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31 -securesystemslib==0.14.0 \ - --hash=sha256:414a722547876294764813f7a3579bba273db6969de81bda2f46f60519e14e3e \ - --hash=sha256:6cbd5ad0b2ae160a2de0800950757d6beea33a8aad15b41d6cff788b0a2ba926 -six==1.14.0 \ - --hash=sha256:236bdbdce46e6e6a3d61a337c0f8b763ca1e8717c03b369e87a7ec7ce1319c0a \ - --hash=sha256:8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c -subprocess32==3.5.4 \ - --hash=sha256:88e37c1aac5388df41cc8a8456bb49ebffd321a3ad4d70358e3518176de3a56b \ - --hash=sha256:eb2937c80497978d181efa1b839ec2d9622cf9600a039a79d0e108d1f9aec79d -urllib3==1.25.8 \ - --hash=sha256:2f3db8b19923a873b3e5256dc9c2dedfa883e33d87c690d9c7913e1f40673cdc \ - --hash=sha256:87716c2d2a7121198ebcb7ce7cccf6ce5e9ba539041cfbaeecfb641dc0bf6acc +# +# NOTE: 'pip-compile' only adds dependencies relevant for the Python version, +# in which it is executed. Moreover, it does not add environment markers of +# transitive dependencies. +# The official recommendation for cross-environment usage of pip-compile tends +# towards separate requirements files for each environment (see +# jazzband/pip-tools#651), this seem like an overkill for tuf, where we only +# have a few conditional dependencies, i.e. dependencies that are required on +# Python < 3 only. +# +# +# Below instructions can be used to re-generate 'requirements-pinned.txt', e.g. +# if: +# - requirements are added or removed from this file +# - Python version support is changed +# - CI/CD build breaks due to updates (e.g. transitive dependency conflicts) +# +# 1. Use this script to create a pinned requirements file for each Python +# version +# ``` +# for v in 2.7 3.5 3.6 3.7 3.8; do +# mkvirtualenv tuf-env-${v} -p python${v}; +# pip install pip-tools; +# pip-compile --no-header -o requirements-${v}.txt requirements.txt; +# deactivate; +# rmvirtualenv tuf-env-${v}; +# done; +# +# ``` +# 2. Use this command to merge per-version files +# `sort -o requirements-pinned.txt -u requirements-?.?.txt` +# 2. Manually add environment markers to requirements-pinned.txt +# 3. Use this command to remove per-version files +# `rm requirements-?.?.txt` +# +securesystemslib[colors, crypto, pynacl] +requests +six +iso8601 diff --git a/tox.ini b/tox.ini index d874efc99b..ccdac9fb35 100644 --- a/tox.ini +++ b/tox.ini @@ -20,7 +20,7 @@ commands = coverage report -m --fail-under 97 deps = - -r{toxinidir}/requirements-tox.txt + -r{toxinidir}/requirements-test.txt # Install TUF in editable mode, instead of tox default virtual environment # installation (see `skipsdist`), to get relative paths in coverage reports --editable {toxinidir} @@ -33,7 +33,7 @@ install_command = pip install --pre {opts} {packages} [testenv:with-sslib-master] deps = --editable git+http://github.com/secure-systems-lab/securesystemslib.git@master#egg=securesystemslib[crypto,pynacl] - -r{toxinidir}/requirements-tox.txt + -r{toxinidir}/requirements-test.txt --editable {toxinidir} commands =