Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Align development requirements with Tidelift's Security-advised PyPI catalog #5762

Closed
aclark4life opened this issue Oct 14, 2021 · 11 comments
Closed
Assignees
Milestone

Comments

@aclark4life
Copy link
Member

FYI I'm going to be setting up a GitHub Action for Tidelift for this:

In the Tidelift parlance, we're going to make sure our development environment "aligns with Security-advised PyPI catalog". While it's obviously not critical that folks use the latest docutils to develop Pillow, it may be interesting to be able to say our development requirements "meet certain licensing, security or other standards". If nothing else, it will definitely help Tidelift as they try to improve "how lifters work with Tidelift and their subscribers". 👍

Screen Shot 2021-10-14 at 1 34 01 PM

@aclark4life
Copy link
Member Author

Done, pending merge of #5763

@hugovk
Copy link
Member

hugovk commented Oct 14, 2021

https://support.tidelift.com/hc/en-us/articles/4406286154004 says of the type:

Whether it's used at runtime or development

The screenshot shows only runtime, but these are all for development (maybe excepting olefile, which is an optional dependency, but must be installed separately and we only install for testing).

Is there a way to mark them as development?

@aclark4life
Copy link
Member Author

@hugovk I haven't considered doing it for runtime yet because to create a project on Tidelift, we need a requirements.txt file (or Pipfile) and all of our dependencies are in setup.py right? Not opposed to doing runtime … but this is a WIP.

@aclark4life
Copy link
Member Author

@hugovk Should I put a conditional on this? Seems to be running a lot 😂 5c69dc7

@hugovk
Copy link
Member

hugovk commented Oct 14, 2021

@hugovk I haven't considered doing it for runtime yet because to create a project on Tidelift, we need a requirements.txt file (or Pipfile) and all of our dependencies are in setup.py right? Not opposed to doing runtime … but this is a WIP.

Off the top of my head we don't have any required runtime dependencies (only optional).

@hugovk Should I put a conditional on this? Seems to be running a lot 😂 5c69dc7

What is the action actually doing? Gathering stats for the catalogue?

When do we want it to run? Is it something that should fail a PR if something needs doing? Or just need to run it occasionally, like when something is pushed to master, like a PR merge?

https://support.tidelift.com/hc/en-us/articles/4406286307220-Using-with-Continuous-Integration suggests every PR:

This action will run a check on every pull request to see if all of your dependencies are included in your catalog's approved release list.

What sort of failures can we expect, what action would I as a developer need to take if something failed?

@hugovk
Copy link
Member

hugovk commented Oct 14, 2021

Maybe @JeffStern can help ^ with my questions :)

@aclark4life Maybe let's revert the action for now and put it in the PR as well? To make sure it's running properly and is also skipped for forks (i.e. avoid this https://github.com/hugovk/Pillow/actions/runs/1343212315). I'd especially like things running smoothly for tomorrow's release :) Starting tomorrow morning, Europe time.

@aclark4life
Copy link
Member Author

@hugovk Sure go ahead

@hugovk hugovk modified the milestones: 8.4.0, 9.0.0 Oct 15, 2021
@aclark4life aclark4life pinned this issue Oct 15, 2021
@aclark4life
Copy link
Member Author

aclark4life commented Nov 30, 2021

What is the action actually doing? Gathering stats for the catalogue?

Here is some output from a Jenkins running at my house to build a catalog.


+ make scan
tidelift alignment save --debug || true
time="2021-11-30T00:09:51-05:00" level=debug msg="GET https://download.tidelift.com/cli/tidelift-cli.version"
Running alignment. Uploading files. (This may take a few minutes to complete)
time="2021-11-30T00:09:52-05:00" level=debug msg=" Reading requirements.txt"
time="2021-11-30T00:09:52-05:00" level=debug msg="Checking Pip"
time="2021-11-30T00:09:52-05:00" level=debug msg="Processing: requirements.txt"
time="2021-11-30T00:09:52-05:00" level=debug msg="Running /Users/alexclark/.jenkins/workspace/_python_python-data-science_main/bin/python3 -m venv tidelift-temp-virtualenv"
time="2021-11-30T00:09:56-05:00" level=debug msg="Running tidelift-temp-virtualenv/bin/python -m pip install --disable-pip-version-check -r requirements.txt"
time="2021-11-30T00:10:51-05:00" level=debug msg="Running tidelift-temp-virtualenv/bin/python -m pip freeze --disable-pip-version-check --local -r requirements.txt"
time="2021-11-30T00:10:53-05:00" level=debug msg="Checking Golang"
time="2021-11-30T00:10:53-05:00" level=debug msg="Is Not Golang"
time="2021-11-30T00:10:53-05:00" level=debug msg="Checking Maven"
time="2021-11-30T00:10:53-05:00" level=debug msg="Is Not Maven"
time="2021-11-30T00:10:53-05:00" level=debug msg="Checking Gradle"
time="2021-11-30T00:10:53-05:00" level=debug msg="Is Not Gradle"
time="2021-11-30T00:10:53-05:00" level=debug msg="Checking NuGet"
time="2021-11-30T00:10:53-05:00" level=debug msg="Is Not NuGet"
time="2021-11-30T00:10:53-05:00" level=debug msg="Checking Npm"
time="2021-11-30T00:10:53-05:00" level=debug msg="Is Not Npm"
time="2021-11-30T00:10:53-05:00" level=debug msg="Starting POST https://api.tidelift.com/subscriber/upload?team=team/tidelift&repository=stack-python-data-science"
time="2021-11-30T00:10:53-05:00" level=debug msg="URL: https://api.tidelift.com/subscriber/upload?team=team/tidelift&repository=stack-python-data-science"
time="2021-11-30T00:10:53-05:00" level=debug msg="Header Content-Type: multipart/form-data; boundary=c23673e21a27832a3efbf0651eaf789102e7945a73fa34a6ad8578b1e679"
time="2021-11-30T00:10:53-05:00" level=debug msg="Header Authorization: Bearer v2/repo...REDACTED...a1kgt"
time="2021-11-30T00:10:53-05:00" level=debug msg="Header Accept: */*"
time="2021-11-30T00:10:53-05:00" level=debug msg="Header X-Tidelift-Cli-Version: 1.2.5-build-4330"
time="2021-11-30T00:10:53-05:00" level=debug msg="Header User-Agent: tidelift-cli/1.2.5-build-4330"
time="2021-11-30T00:10:53-05:00" level=debug msg="Body: {--c23673e21a27832a3efbf0651eaf789102e7945a73fa34a6ad8578b1e679\r\nContent-Disposition: form-data; name=\"repository\"\r\n\r\nstack-python-data-science\r\n--c23673e21a27832a3efbf0651eaf789102e7945a73fa34a6ad8578b1e679\r\nContent-Disposition: form-data; name=\"files[]\"; filename=\"requirements.txt\"\r\nContent-Type: application/octet-stream\r\n\r\nAutomat==20.2.0\nDeprecated==1.2.13\nPillow==8.4.0\nProtego==0.1.16\nPyDispatcher==2.0.5\nPyGithub==1.55\nPyHamcrest==2.0.2\nPyJWT==2.3.0\nPyNaCl==1.4.0\nScrapy==2.5.1\nTwisted==21.7.0\nattrs==21.2.0\nbeautifulsoup4==4.10.0\ncertifi==2021.10.8\ncffi==1.15.0\nchardet==4.0.0\ncharset-normalizer==2.0.8\nconstantly==15.1.0\ncryptography==36.0.0\ncssselect==1.1.0\ncycler==0.11.0\nfonttools==4.28.2\nh2==3.2.0\nhpack==3.0.0\nhyperframe==5.2.0\nhyperlink==21.0.0\nidna==3.3\nincremental==21.3.0\nitemadapter==0.4.0\nitemloaders==1.0.4\njmespath==0.10.0\nkiwisolver==1.3.2\nlxml==4.6.4\nmatplotlib==3.5.0\nnumpy==1.21.4\npackaging==21.3\npandas==1.3.4\nparsel==1.6.0\npriority==1.3.0\npyOpenSSL==21.0.0\npyarrow==6.0.1\npyasn1-modules==0.2.8\npyasn1==0.4.8\npybaseball==2.2.1\npycparser==2.21\npyparsing==3.0.6\npython-dateutil==2.8.2\npytz==2021.3\nqueuelib==1.6.2\nrequests==2.26.0\nscipy==1.7.3\nsemantic-version==2.8.5\nservice-identity==21.1.0\nsetuptools-rust==1.0.0\nsetuptools-scm==6.3.2\nsix==1.16.0\nsoupsieve==2.3.1\ntoml==0.10.2\ntomli==1.2.2\ntqdm==4.62.3\ntyping-extensions==4.0.0\nurllib3==1.26.7\nw3lib==1.22.0\nwrapt==1.13.3\nzope.interface==5.4.0\n\r\n--c23673e21a27832a3efbf0651eaf789102e7945a73fa34a6ad8578b1e679\r\nContent-Disposition: form-data; name=\"files[]\"; filename=\"pip-resolved-dependencies.txt\"\r\nContent-Type: application/octet-stream\r\n\r\nAutomat==20.2.0\nDeprecated==1.2.13\nPillow==8.4.0\nProtego==0.1.16\nPyDispatcher==2.0.5\nPyGithub==1.55\nPyHamcrest==2.0.2\nPyJWT==2.3.0\nPyNaCl==1.4.0\nScrapy==2.5.1\nTwisted==21.7.0\nattrs==21.2.0\nbeautifulsoup4==4.10.0\ncertifi==2021.10.8\ncffi==1.15.0\nchardet==4.0.0\ncharset-normalizer==2.0.8\nconstantly==15.1.0\ncryptography==36.0.0\ncssselect==1.1.0\ncycler==0.11.0\nfonttools==4.28.2\nh2==3.2.0\nhpack==3.0.0\nhyperframe==5.2.0\nhyperlink==21.0.0\nidna==3.3\nincremental==21.3.0\nitemadapter==0.4.0\nitemloaders==1.0.4\njmespath==0.10.0\nkiwisolver==1.3.2\nlxml==4.6.4\nmatplotlib==3.5.0\nnumpy==1.21.4\npackaging==21.3\npandas==1.3.4\nparsel==1.6.0\npriority==1.3.0\npyOpenSSL==21.0.0\npyarrow==6.0.1\npyasn1-modules==0.2.8\npyasn1==0.4.8\npybaseball==2.2.1\npycparser==2.21\npyparsing==3.0.6\npython-dateutil==2.8.2\npytz==2021.3\nqueuelib==1.6.2\nrequests==2.26.0\nscipy==1.7.3\nsemantic-version==2.8.5\nservice-identity==21.1.0\nsetuptools-rust==1.0.0\nsetuptools-scm==6.3.2\nsix==1.16.0\nsoupsieve==2.3.1\ntoml==0.10.2\ntomli==1.2.2\ntqdm==4.62.3\ntyping-extensions==4.0.0\nurllib3==1.26.7\nw3lib==1.22.0\nwrapt==1.13.3\nzope.interface==5.4.0\n## The following requirements were added by pip freeze:\n\r\n--c23673e21a27832a3efbf0651eaf789102e7945a73fa34a6ad8578b1e679--\r\n}"
time="2021-11-30T00:10:54-05:00" level=debug msg="POST JSON Response: {\"revision\":\"20211130051053\"}"
time="2021-11-30T00:10:54-05:00" level=debug msg="Check for approved Status Code first"

Revision: 20211130051053 (Check alignment status by running 'tidelift status 20211130051053')
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Align)
[Pipeline] echo
Aligning..
[Pipeline] sh
+ source bin/activate
++ deactivate nondestructive
++ '[' -n '' ']'
++ '[' -n '' ']'
++ '[' -n /bin/sh -o -n '' ']'
++ hash -r
++ '[' -n '' ']'
++ unset VIRTUAL_ENV
++ '[' '!' nondestructive = nondestructive ']'
++ VIRTUAL_ENV=/Users/alexclark/.jenkins/workspace/_python_python-data-science_main
++ export VIRTUAL_ENV
++ _OLD_VIRTUAL_PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/opt/python@3.8/bin
++ PATH=/Users/alexclark/.jenkins/workspace/_python_python-data-science_main/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/opt/python@3.8/bin
++ export PATH
++ '[' -n '' ']'
++ '[' -z '' ']'
++ _OLD_VIRTUAL_PS1=
++ '[' 'x(_python_python-data-science_main) ' '!=' x ']'
++ PS1='(_python_python-data-science_main) '
++ export PS1
++ '[' -n /bin/sh -o -n '' ']'
++ hash -r
+ make align
tidelift alignment --debug || true
time="2021-11-30T00:10:55-05:00" level=debug msg="GET https://download.tidelift.com/cli/tidelift-cli.version"
time="2021-11-30T00:10:55-05:00" level=debug msg="Starting GET"
time="2021-11-30T00:10:55-05:00" level=debug msg="URL: https://api.tidelift.com/external-api/authcheck"
time="2021-11-30T00:10:55-05:00" level=debug msg="Header X-Tidelift-Cli-Version: 1.2.5-build-4330"
time="2021-11-30T00:10:55-05:00" level=debug msg="Header User-Agent: tidelift-cli/1.2.5-build-4330"
time="2021-11-30T00:10:55-05:00" level=debug msg="Header Authorization: Bearer v2/repo...REDACTED...a1kgt"
time="2021-11-30T00:10:55-05:00" level=debug msg="Header Accept: */*"
time="2021-11-30T00:10:55-05:00" level=debug msg="GET JSON: {\"message\":\"authenticated\"}"
time="2021-11-30T00:10:55-05:00" level=debug msg="Check for approved Status Code first"
Running Tidelift Alignment Check
time="2021-11-30T00:10:57-05:00" level=debug msg=" Reading requirements.txt"
time="2021-11-30T00:10:57-05:00" level=debug msg="Checking Pip"
time="2021-11-30T00:10:57-05:00" level=debug msg="Processing: requirements.txt"
time="2021-11-30T00:10:57-05:00" level=debug msg="Running /Users/alexclark/.jenkins/workspace/_python_python-data-science_main/bin/python3 -m venv tidelift-temp-virtualenv"
time="2021-11-30T00:11:03-05:00" level=debug msg="Running tidelift-temp-virtualenv/bin/python -m pip install --disable-pip-version-check -r requirements.txt"
time="2021-11-30T00:11:58-05:00" level=debug msg="Running tidelift-temp-virtualenv/bin/python -m pip freeze --disable-pip-version-check --local -r requirements.txt"
time="2021-11-30T00:12:00-05:00" level=debug msg="Checking Golang"
time="2021-11-30T00:12:00-05:00" level=debug msg="Is Not Golang"
time="2021-11-30T00:12:00-05:00" level=debug msg="Checking Maven"
time="2021-11-30T00:12:00-05:00" level=debug msg="Is Not Maven"
time="2021-11-30T00:12:00-05:00" level=debug msg="Checking Gradle"
time="2021-11-30T00:12:00-05:00" level=debug msg="Is Not Gradle"
time="2021-11-30T00:12:00-05:00" level=debug msg="Checking NuGet"
time="2021-11-30T00:12:00-05:00" level=debug msg="Is Not NuGet"
time="2021-11-30T00:12:00-05:00" level=debug msg="Checking Npm"
time="2021-11-30T00:12:00-05:00" level=debug msg="Is Not Npm"
time="2021-11-30T00:12:00-05:00" level=debug msg="Starting POST https://api.tidelift.com/external-api/catalog/team/tidelift/alignment"
time="2021-11-30T00:12:00-05:00" level=debug msg="URL: https://api.tidelift.com/external-api/catalog/team/tidelift/alignment"
time="2021-11-30T00:12:00-05:00" level=debug msg="Header Authorization: Bearer v2/repo...REDACTED...a1kgt"
time="2021-11-30T00:12:00-05:00" level=debug msg="Header Accept: */*"
time="2021-11-30T00:12:00-05:00" level=debug msg="Header X-Tidelift-Cli-Version: 1.2.5-build-4330"
time="2021-11-30T00:12:00-05:00" level=debug msg="Header User-Agent: tidelift-cli/1.2.5-build-4330"
time="2021-11-30T00:12:00-05:00" level=debug msg="Header Content-Type: multipart/form-data; boundary=20260a06b0383b6f6775ab5d58a99fb18f15e93bbf3555c31637c1747cf6"
time="2021-11-30T00:12:00-05:00" level=debug msg="Body: {--20260a06b0383b6f6775ab5d58a99fb18f15e93bbf3555c31637c1747cf6\r\nContent-Disposition: form-data; name=\"repository\"\r\n\r\nstack-python-data-science\r\n--20260a06b0383b6f6775ab5d58a99fb18f15e93bbf3555c31637c1747cf6\r\nContent-Disposition: form-data; name=\"files[]\"; filename=\"requirements.txt\"\r\nContent-Type: application/octet-stream\r\n\r\nAutomat==20.2.0\nDeprecated==1.2.13\nPillow==8.4.0\nProtego==0.1.16\nPyDispatcher==2.0.5\nPyGithub==1.55\nPyHamcrest==2.0.2\nPyJWT==2.3.0\nPyNaCl==1.4.0\nScrapy==2.5.1\nTwisted==21.7.0\nattrs==21.2.0\nbeautifulsoup4==4.10.0\ncertifi==2021.10.8\ncffi==1.15.0\nchardet==4.0.0\ncharset-normalizer==2.0.8\nconstantly==15.1.0\ncryptography==36.0.0\ncssselect==1.1.0\ncycler==0.11.0\nfonttools==4.28.2\nh2==3.2.0\nhpack==3.0.0\nhyperframe==5.2.0\nhyperlink==21.0.0\nidna==3.3\nincremental==21.3.0\nitemadapter==0.4.0\nitemloaders==1.0.4\njmespath==0.10.0\nkiwisolver==1.3.2\nlxml==4.6.4\nmatplotlib==3.5.0\nnumpy==1.21.4\npackaging==21.3\npandas==1.3.4\nparsel==1.6.0\npriority==1.3.0\npyOpenSSL==21.0.0\npyarrow==6.0.1\npyasn1-modules==0.2.8\npyasn1==0.4.8\npybaseball==2.2.1\npycparser==2.21\npyparsing==3.0.6\npython-dateutil==2.8.2\npytz==2021.3\nqueuelib==1.6.2\nrequests==2.26.0\nscipy==1.7.3\nsemantic-version==2.8.5\nservice-identity==21.1.0\nsetuptools-rust==1.0.0\nsetuptools-scm==6.3.2\nsix==1.16.0\nsoupsieve==2.3.1\ntoml==0.10.2\ntomli==1.2.2\ntqdm==4.62.3\ntyping-extensions==4.0.0\nurllib3==1.26.7\nw3lib==1.22.0\nwrapt==1.13.3\nzope.interface==5.4.0\n\r\n--20260a06b0383b6f6775ab5d58a99fb18f15e93bbf3555c31637c1747cf6\r\nContent-Disposition: form-data; name=\"files[]\"; filename=\"pip-resolved-dependencies.txt\"\r\nContent-Type: application/octet-stream\r\n\r\nAutomat==20.2.0\nDeprecated==1.2.13\nPillow==8.4.0\nProtego==0.1.16\nPyDispatcher==2.0.5\nPyGithub==1.55\nPyHamcrest==2.0.2\nPyJWT==2.3.0\nPyNaCl==1.4.0\nScrapy==2.5.1\nTwisted==21.7.0\nattrs==21.2.0\nbeautifulsoup4==4.10.0\ncertifi==2021.10.8\ncffi==1.15.0\nchardet==4.0.0\ncharset-normalizer==2.0.8\nconstantly==15.1.0\ncryptography==36.0.0\ncssselect==1.1.0\ncycler==0.11.0\nfonttools==4.28.2\nh2==3.2.0\nhpack==3.0.0\nhyperframe==5.2.0\nhyperlink==21.0.0\nidna==3.3\nincremental==21.3.0\nitemadapter==0.4.0\nitemloaders==1.0.4\njmespath==0.10.0\nkiwisolver==1.3.2\nlxml==4.6.4\nmatplotlib==3.5.0\nnumpy==1.21.4\npackaging==21.3\npandas==1.3.4\nparsel==1.6.0\npriority==1.3.0\npyOpenSSL==21.0.0\npyarrow==6.0.1\npyasn1-modules==0.2.8\npyasn1==0.4.8\npybaseball==2.2.1\npycparser==2.21\npyparsing==3.0.6\npython-dateutil==2.8.2\npytz==2021.3\nqueuelib==1.6.2\nrequests==2.26.0\nscipy==1.7.3\nsemantic-version==2.8.5\nservice-identity==21.1.0\nsetuptools-rust==1.0.0\nsetuptools-scm==6.3.2\nsix==1.16.0\nsoupsieve==2.3.1\ntoml==0.10.2\ntomli==1.2.2\ntqdm==4.62.3\ntyping-extensions==4.0.0\nurllib3==1.26.7\nw3lib==1.22.0\nwrapt==1.13.3\nzope.interface==5.4.0\n## The following requirements were added by pip freeze:\n\r\n--20260a06b0383b6f6775ab5d58a99fb18f15e93bbf3555c31637c1747cf6--\r\n}"
time="2021-11-30T00:12:00-05:00" level=debug msg="POST JSON Response: {\"_meta\":{\"version\":\"1\"},\"organization\":\"tidelift\",\"team\":\"tidelift\",\"files\":\"requirements.txt, pip-resolved-dependencies.txt\",\"alignment_pct\":100.0,\"unapproved\":[],\"alternatives\":[],\"upcoming_denials\":[],\"approved\":[{\"id\":50964,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"Automat\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"20.2.0\",\"requirement\":\"20.2.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"20.2.0\",\"status\":\"approved\"},{\"id\":101061,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"Deprecated\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"1.2.13\",\"requirement\":\"1.2.13\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"1.2.13\",\"status\":\"approved\"},{\"id\":660,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"Pillow\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"8.4.0\",\"requirement\":\"8.4.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"8.4.0\",\"status\":\"approved\"},{\"id\":132919,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"Protego\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"0.1.16\",\"requirement\":\"0.1.16\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"0.1.16\",\"status\":\"approved\"},{\"id\":34673,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"PyDispatcher\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"2.0.5\",\"requirement\":\"2.0.5\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"2.0.5\",\"status\":\"approved\"},{\"id\":32163,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"PyGithub\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"1.55\",\"requirement\":\"1.55\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"1.55\",\"status\":\"approved\"},{\"id\":38283,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"PyHamcrest\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"2.0.2\",\"requirement\":\"2.0.2\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"2.0.2\",\"status\":\"approved\"},{\"id\":30256,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"PyJWT\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"2.3.0\",\"requirement\":\"2.3.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"2.3.0\",\"status\":\"approved\"},{\"id\":31623,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"PyNaCl\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"1.4.0\",\"requirement\":\"1.4.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"1.4.0\",\"status\":\"approved\"},{\"id\":33325,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"Scrapy\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"2.5.1\",\"requirement\":\"2.5.1\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"2.5.1\",\"status\":\"approved\"},{\"id\":34676,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"Twisted\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"21.7.0\",\"requirement\":\"21.7.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"21.7.0\",\"status\":\"approved\"},{\"id\":3536,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"attrs\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"21.2.0\",\"requirement\":\"21.2.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"21.2.0\",\"status\":\"approved\"},{\"id\":105,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"beautifulsoup4\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"4.10.0\",\"requirement\":\"4.10.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"4.10.0\",\"status\":\"approved\"},{\"id\":4257,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"certifi\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"2021.10.8\",\"requirement\":\"2021.10.8\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"2021.10.8\",\"status\":\"approved\"},{\"id\":6245,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"cffi\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"1.15.0\",\"requirement\":\"1.15.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"1.15.0\",\"status\":\"approved\"},{\"id\":6246,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"chardet\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"4.0.0\",\"requirement\":\"4.0.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"4.0.0\",\"status\":\"approved\"},{\"id\":64001197,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"charset-normalizer\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"2.0.8\",\"requirement\":\"2.0.8\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"2.0.8\",\"status\":\"approved\"},{\"id\":34668,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"constantly\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"15.1.0\",\"requirement\":\"15.1.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"15.1.0\",\"status\":\"approved\"},{\"id\":6248,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"cryptography\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"36.0.0\",\"requirement\":\"36.0.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"36.0.0\",\"status\":\"approved\"},{\"id\":34669,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"cssselect\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"1.1.0\",\"requirement\":\"1.1.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"1.1.0\",\"status\":\"approved\"},{\"id\":4261,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"cycler\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"0.11.0\",\"requirement\":\"0.11.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"0.11.0\",\"status\":\"approved\"},{\"id\":35563,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"fonttools\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"4.28.2\",\"requirement\":\"4.28.2\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"4.28.2\",\"status\":\"approved\"},{\"id\":46022,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"h2\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"3.2.0\",\"requirement\":\"3.2.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"3.2.0\",\"status\":\"approved\"},{\"id\":49246,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"hpack\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"3.0.0\",\"requirement\":\"3.0.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"3.0.0\",\"status\":\"approved\"},{\"id\":49248,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"hyperframe\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"5.2.0\",\"requirement\":\"5.2.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"5.2.0\",\"status\":\"approved\"},{\"id\":50968,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"hyperlink\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"21.0.0\",\"requirement\":\"21.0.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"21.0.0\",\"status\":\"approved\"},{\"id\":6272,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"idna\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"3.3\",\"requirement\":\"3.3\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"3.3\",\"status\":\"approved\"},{\"id\":34670,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"incremental\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"21.3.0\",\"requirement\":\"21.3.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"21.3.0\",\"status\":\"approved\"},{\"id\":142633,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"itemadapter\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"0.4.0\",\"requirement\":\"0.4.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"0.4.0\",\"status\":\"approved\"},{\"id\":142634,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"itemloaders\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"1.0.4\",\"requirement\":\"1.0.4\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"1.0.4\",\"status\":\"approved\"},{\"id\":4268,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"jmespath\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"0.10.0\",\"requirement\":\"0.10.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"0.10.0\",\"status\":\"approved\"},{\"id\":4269,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"kiwisolver\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"1.3.2\",\"requirement\":\"1.3.2\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"1.3.2\",\"status\":\"approved\"},{\"id\":4270,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"lxml\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"4.6.4\",\"requirement\":\"4.6.4\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"4.6.4\",\"status\":\"approved\"},{\"id\":4272,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"matplotlib\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"3.5.0\",\"requirement\":\"3.5.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"3.5.0\",\"status\":\"approved\"},{\"id\":1488,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"numpy\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"1.21.4\",\"requirement\":\"1.21.4\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"1.21.4\",\"status\":\"approved\"},{\"id\":6802,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"packaging\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"21.3\",\"requirement\":\"21.3\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"21.3\",\"status\":\"approved\"},{\"id\":4273,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"pandas\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"1.3.4\",\"requirement\":\"1.3.4\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"1.3.4\",\"status\":\"approved\"},{\"id\":34671,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"parsel\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"1.6.0\",\"requirement\":\"1.6.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"1.6.0\",\"status\":\"approved\"},{\"id\":50970,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"priority\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"1.3.0\",\"requirement\":\"1.3.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"1.3.0\",\"status\":\"approved\"},{\"id\":29916,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"pyOpenSSL\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"21.0.0\",\"requirement\":\"21.0.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"21.0.0\",\"status\":\"approved\"},{\"id\":6312,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"pyarrow\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"6.0.1\",\"requirement\":\"6.0.1\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"6.0.1\",\"status\":\"approved\"},{\"id\":18548,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"pyasn1-modules\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"0.2.8\",\"requirement\":\"0.2.8\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"0.2.8\",\"status\":\"approved\"},{\"id\":6313,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"pyasn1\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"0.4.8\",\"requirement\":\"0.4.8\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"0.4.8\",\"status\":\"approved\"},{\"id\":63998973,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"pybaseball\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"2.2.1\",\"requirement\":\"2.2.1\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"2.2.1\",\"status\":\"approved\"},{\"id\":6314,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"pycparser\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"2.21\",\"requirement\":\"2.21\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"2.21\",\"status\":\"approved\"},{\"id\":4282,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"pyparsing\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"3.0.6\",\"requirement\":\"3.0.6\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"3.0.6\",\"status\":\"approved\"},{\"id\":4283,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"python-dateutil\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"2.8.2\",\"requirement\":\"2.8.2\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"2.8.2\",\"status\":\"approved\"},{\"id\":4285,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"pytz\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"2021.3\",\"requirement\":\"2021.3\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"2021.3\",\"status\":\"approved\"},{\"id\":34674,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"queuelib\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"1.6.2\",\"requirement\":\"1.6.2\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"1.6.2\",\"status\":\"approved\"},{\"id\":38,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"requests\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"2.26.0\",\"requirement\":\"2.26.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"2.26.0\",\"status\":\"approved\"},{\"id\":3966,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"scipy\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"1.7.3\",\"requirement\":\"1.7.3\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"1.7.3\",\"status\":\"approved\"},{\"id\":6652,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"semantic-version\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"2.8.5\",\"requirement\":\"2.8.5\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"2.8.5\",\"status\":\"approved\"},{\"id\":34675,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"service-identity\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"21.1.0\",\"requirement\":\"21.1.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"21.1.0\",\"status\":\"approved\"},{\"id\":79974,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"setuptools-rust\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"1.0.0\",\"requirement\":\"1.0.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"1.0.0\",\"status\":\"approved\"},{\"id\":1345,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"setuptools-scm\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"6.3.2\",\"requirement\":\"6.3.2\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"6.3.2\",\"status\":\"approved\"},{\"id\":4288,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"six\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"1.16.0\",\"requirement\":\"1.16.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"1.16.0\",\"status\":\"approved\"},{\"id\":442,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"soupsieve\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"2.3.1\",\"requirement\":\"2.3.1\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"2.3.1\",\"status\":\"approved\"},{\"id\":6866,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"toml\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"0.10.2\",\"requirement\":\"0.10.2\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"0.10.2\",\"status\":\"approved\"},{\"id\":64001237,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"tomli\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"1.2.2\",\"requirement\":\"1.2.2\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"1.2.2\",\"status\":\"approved\"},{\"id\":6695,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"tqdm\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"4.62.3\",\"requirement\":\"4.62.3\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"4.62.3\",\"status\":\"approved\"},{\"id\":22805,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"typing-extensions\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"4.0.0\",\"requirement\":\"4.0.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"4.0.0\",\"status\":\"approved\"},{\"id\":663,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"urllib3\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"1.26.7\",\"requirement\":\"1.26.7\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"1.26.7\",\"status\":\"approved\"},{\"id\":34677,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"w3lib\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"1.22.0\",\"requirement\":\"1.22.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"1.22.0\",\"status\":\"approved\"},{\"id\":6671,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"wrapt\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"1.13.3\",\"requirement\":\"1.13.3\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"1.13.3\",\"status\":\"approved\"},{\"id\":18011,\"latest_release_number\":null,\"license_normalized\":null,\"license_set_by_admin\":null,\"licenses\":null,\"name\":\"zope.interface\",\"platform\":\"pypi\",\"project_license\":null,\"repository_license\":null,\"sources\":null,\"type\":\"runtime\",\"unknown\":null,\"direct\":true,\"resolved_version\":\"5.4.0\",\"requirement\":\"5.4.0\",\"lockfile_requirement\":null,\"original_license\":null,\"version\":\"5.4.0\",\"status\":\"approved\"}],\"requested\":[],\"denied\":[],\"catalog\":{\"id\":\"7f3c8597-dfd5-4c78-8439-333d89706fcf\",\"name\":\"python-data-science\",\"display_name\":\"Python Data Science\"}}"
time="2021-11-30T00:12:00-05:00" level=debug msg="Check for approved Status Code first"
Checked alignment with the Python Data Science catalog.

Files analyzed: requirements.txt, pip-resolved-dependencies.txt

Alignment score: 100%

65 package releases identified       
65 package releases approved for use ■■■■■■■■■■ 


Your alignment score shows what percentage (%) of package releases in your project have been approved.




Your project is aligned with your catalog and is ready for production.
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Request)
[Pipeline] echo
Requesting..
[Pipeline] sh
+ source bin/activate
++ deactivate nondestructive
++ '[' -n '' ']'
++ '[' -n '' ']'
++ '[' -n /bin/sh -o -n '' ']'
++ hash -r
++ '[' -n '' ']'
++ unset VIRTUAL_ENV
++ '[' '!' nondestructive = nondestructive ']'
++ VIRTUAL_ENV=/Users/alexclark/.jenkins/workspace/_python_python-data-science_main
++ export VIRTUAL_ENV
++ _OLD_VIRTUAL_PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/opt/python@3.8/bin
++ PATH=/Users/alexclark/.jenkins/workspace/_python_python-data-science_main/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/opt/python@3.8/bin
++ export PATH
++ '[' -n '' ']'
++ '[' -z '' ']'
++ _OLD_VIRTUAL_PS1=
++ '[' 'x(_python_python-data-science_main) ' '!=' x ']'
++ PS1='(_python_python-data-science_main) '
++ export PS1
++ '[' -n /bin/sh -o -n '' ']'
++ hash -r
+ make request
tidelift request --all
Requesting package releases...

No package releases requested. No releases available to request.

That job runs this Makefile target on the requirements.txt file of the "data science catalog":

That target updates the requirements then calls tidelift <whatever> on the updated requirements. Tidelift uses that data to build something like (and possibly exactly) this:

When do we want it to run? Is it something that should fail a PR if something needs doing? Or just need to run it occasionally, like when something is pushed to master, like a PR merge?

https://support.tidelift.com/hc/en-us/articles/4406286307220-Using-with-Continuous-Integration suggests every PR:

This action will run a check on every pull request to see if all of your dependencies are included in your catalog's approved release list.

In building a catalog, once a day to "keep up" with ever-evolving ecosystem. In using a catalog … in our case … probably still once a day because this is experimental and the value add is TBD. I expect little benefit in Pillow's case, but I'd still like to see it running here to support Tidelift's efforts of opening up catalogs to lifters for the potential value to other projects.

What sort of failures can we expect, what action would I as a developer need to take if something failed?

Good question! In building a catalog I've seen very few failures, if any. In using a catalog, assuming the Tidelift API answers the phone, then we should just expect to occassionally see results like "Your SBOM not aligned with our catalog!"

In the case of our development requirements, we don't specify any versions so in theory, our requirements will always be 100% aligned with the security-advised PyPI catalog.

@hugovk
Copy link
Member

hugovk commented Nov 30, 2021

Thanks for checking all that!

So it sounds like we're fine running it daily and don't need to trigger it with regular PRs, at least initially.

I'd suggest something like this at the top of tidelift.yml to define how it's triggered:

on:
  schedule:
    - cron: "30 2 * * *"  # daily at 02:30 UTC
  push:
    paths:
      - ".github/workflows/tidelift.yml"
  pull_request:
    paths:
      - ".github/workflows/tidelift.yml"
  workflow_dispatch:

Also it looks like we're missing an API key:

Uploading manifests for alignment
Running alignment. Uploading files. (This may take a few minutes to complete)
Error: invalid authentication. are you using the correct API key? (ptm.tl/docs-apikeys).

https://github.com/python-pillow/Pillow/runs/4124628947?check_suite_focus=true

Please can you create one and store it in the repo secret settings as TIDELIFT_API_KEY?

@aclark4life
Copy link
Member Author

@hugovk I'll let @JeffStern answer about giving Pillow team members access to Tidelift dashboard, but in the meantime let's assume the failures are sphinx and babel not available in the catalog. Where does babel even come from? I just ran pipdeptree on requirements.txt and I don't see it.


╰─[:)] % for i in `cat requirements.txt | grep -v ^# `
do
echo -- $i --
pipdeptree -p $i
done
-- black --
black==19.3b0
  - appdirs [required: Any, installed: 1.4.4]
  - attrs [required: >=18.1.0, installed: 21.2.0]
  - click [required: >=6.5, installed: 8.0.3]
  - toml [required: >=0.9.4, installed: 0.10.2]
-- check-manifest --

-- coverage --

-- defusedxml --

-- docutils==0.16 --

-- markdown2 --

-- olefile --

-- packaging --

-- pyroma --

-- pytest --

-- pytest-cov --

-- pytest-timeout --

-- sphinx>=2.4 --

-- sphinx-copybutton --

-- sphinx-issues --

-- sphinx-removed-in --

-- sphinx-rtd-theme --

-- sphinxext-opengraph --

I'm tempted to remove all the package names and start adding them back one by one …

@aclark4life
Copy link
Member Author

Done in #5763

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants