[Packaging] Bump embedded Python version to 3.12.7 #29887
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
️✔️AzureCLI-FullTest
|
️✔️AzureCLI-BreakingChangeTest
|
|
Packaging |
bebound
changed the title
microsoft-github-policy-service
bot
requested review from
jiasli,
wangzelin007 and
yonzhan
bebound
changed the title
bebound
commented
Sep 11, 2024
|
|
||
| echo "== Testing pip install on Python 3.11 ==" | ||
| docker run \ | ||
| --rm -v $PYPI_FILES:/mnt/pypi mcr.microsoft.com/mirror/docker/library/python:3.11-slim \ |
There was a problem hiding this comment.
MAR does not provide newer Python docker image anymore, use local Python instead.
bebound
commented
Sep 11, 2024
| @@ -606,7 +597,7 @@ jobs: | |||
| -e CLI_VERSION=$CLI_VERSION \ | |||
| -e HOMEBREW_UPSTREAM_URL=$HOMEBREW_UPSTREAM_URL \ | |||
| --name azurecli \ | |||
| mcr.microsoft.com/mirror/docker/library/python:3.11-slim \ | |||
| mcr.microsoft.com/azurelinux/base/python:3 \ | |||
There was a problem hiding this comment.
Formula generation is independent of the Python version.
Use Azure Linux's 3.12 to avoid updates for the next four years.
bebound
commented
Sep 12, 2024
| shutil.rmtree(self._cache_dir, | ||
| ignore_errors=True, | ||
| onerror=lambda _, p, tr: self._logger.error('Fail to remove file %s', p)) | ||
| shutil.rmtree(self._cache_dir, ignore_errors=True) |
There was a problem hiding this comment.
onerror is only deprecated in 3.12, but there is no equivalent in earlier versions: python/cpython#112645.
onerror is meaningless when ignore_errors=True, so I remove it.
Ref: https://docs.python.org/3/library/shutil.html#shutil.rmtree
bebound
commented
Sep 14, 2024
| source env/bin/activate | ||
| cd azure-cli-extensions | ||
| python ../scripts/ci/automation_full_test.py "12" "$(Instance_idx)" "latest" "" "True" "extension" | ||
| displayName: 'Rerun tests' |
There was a problem hiding this comment.
azdev can't find extension list when running in AzureCLI@1, move it to new step.
bebound
modified the milestones:
September 2024 (2024-10-08),
October 2024 (2024-11-05)
This was referenced Oct 9, 2024
bebound
changed the title
bebound
force-pushed
the
3.12
branch
3 times, most recently
from
afd296a
to
f350858
Compare
bebound
force-pushed
the
3.12
branch
15 times, most recently
from
667728c
to
d53b7dd
Compare
bebound
commented
Oct 15, 2024
| @@ -24,7 +24,11 @@ exit_code=0 | |||
| # Disable k8s-extension temporarily: https://github.com/Azure/azure-cli-extensions/pull/6702 | |||
| # Disable alias temporarily: https://github.com/Azure/azure-cli/pull/27717 | |||
| # hybridaks is going to be deprecated: https://github.com/Azure/azure-cli/pull/29838 | |||
| ignore_list='azure-cli-ml fzf arcappliance arcdata connectedk8s k8s-extension alias hybridaks' | |||
| # db-up is going to be deprecated: https://github.com/Azure/azure-cli/pull/29887 | |||
| ignore_list='azure-cli-ml fzf arcappliance arcdata connectedk8s k8s-extension alias hybridaks db-up' | |||
There was a problem hiding this comment.
db-up's pymssql==2.2.7 does not provide wheel for 3.12. This extension is deprecated.
https://github.com/Azure/azure-cli-extensions/blob/c55f55aaf70fd5df20b5a789294bb909aab4d1ce/src/db-up/setup.py#L33
|
Our self-hosted agent provided by 1ES fails to build Windows package when using 3.12.7, but everything works fine in 3.12.6 embedded Python. When calling The microsoft-hosted agent works normally. The only difference I found is that it has Python 3.12.7. I suspect that calling commands without This issue exists for a long time, but only triggered in this PR. Here are some debug info: |
bebound
force-pushed
the
3.12
branch
8 times, most recently
from
3995731
to
ea25cd5
Compare
bebound
commented
Oct 16, 2024
| REM Install wheel to force pip install azure-cli in legacy mode | ||
| REM see https://github.com/Azure/azure-cli/pull/29887 | ||
| echo Installing setuptools wheel | ||
| %PYTHON_DIR%\python.exe -Im pip install setuptools wheel |
There was a problem hiding this comment.
For setuptools:
Although setuptools is added to azure-cli dependency list in #27196, azure-cli is installed with --no-deps. Need to install it manually.
For wheel:
In 3.11, wheel is also installed when running get-pip.py. However, wheel is not included in final package in Linux and macOS. setuptools also removed wheel dependency in pypa/setuptools#1386.
However, wheel is still a factor for pip to use pep517 or not: https://github.com/pypa/pip/blob/ec5faeac4ef6bae97df0e779566ceb2b0de89d3f/src/pip/_internal/pyproject.py#L107 (ref: pypa/packaging.python.org#1517). If wheel is not installed, pip will use pep517 and fails, see #29887 (comment). The reason is that the -I param is missing in pip's build env. (Ref: pypa/pip#9081)
So I also install wheel. (I think it's okay to remove wheel in the future)
# 3.11's get-pip.py
Successfully installed pip-24.2 setuptools-75.1.0 wheel-0.44.0
Pip set up successful
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Bump bundled Python to 3.12 and upgrade Python version in CI
The package size has also decreased, as the pyc file size is optimized in 3.12.
The windows zip package drops from 314MB to 299MB.
Related issue: #27957
History Notes
[Core] Resolve CVE-2024-6119
[Core] Resolve CVE-2024-5535
[Core] Resolve CVE-2024-4741
[Core] Resolve CVE-2024-4603
[Core] Resolve CVE-2024-2511