Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Python 3.10 support #36

Merged
merged 3 commits into from
Apr 2, 2022
Merged

Add Python 3.10 support #36

merged 3 commits into from
Apr 2, 2022

Conversation

br3ndonland
Copy link
Owner

@br3ndonland br3ndonland commented Jul 5, 2021

Description

This PR will add Python 3.10 support to inboard.

Changes

GitHub Actions

Poetry

Expand this details element for details on some of the problems seen with get-poetry.py and install-poetry.py on Python 3.10.

The previous get-poetry.py install script is not compatible with Python 3.10. Attempts to install Poetry with Python 3.10 and get-poetry.py may raise ModuleNotFoundError (python-poetry/poetry#3071, python-poetry/poetry#3345) and other errors:

 Traceback (most recent call last):
  File "/opt/poetry/bin/poetry", line 17, in <module>
    from poetry.console import main
  File "/opt/poetry/lib/poetry/console/__init__.py", line 1, in <module>
    from .application import Application
  File "/opt/poetry/lib/poetry/console/application.py", line 3, in <module>
    from cleo import Application as BaseApplication
ModuleNotFoundError: No module named 'cleo'

Poetry 1.1.11 claimed to resolve Python 3.10 issues, but it did not.

The new install-poetry.py script (added in python-poetry/poetry#3706) must therefore be used, but it was problematic when it was introduced.

install-poetry.py did not respect POETRY_VIRTUALENVS_CREATE

As of Poetry 1.1.7, there may be complications when using install-poetry.py without a virtualenv (POETRY_VIRTUALENVS_CREATE=false). When installing dependencies, the following error is frequently seen:

OSError

Could not find a suitable TLS CA certificate bundle, invalid path:
/opt/poetry/venv/lib/python3.9/site-packages/certifi/cacert.pem

at /opt/poetry/venv/lib/python3.9/site-packages/requests/adapters.py:227
in cert_verify

In some situations, Poetry uninstalls its own dependencies. The OSError above occurs because Poetry uninstalls requests and certifi, then complains that it can't find them. See:

Poetry may have also been incorrectly attempting to read from its virtualenv (instead of site-packages) if it was not respecting POETRY_VIRTUALENVS_CREATE. Downstream steps also did not appear to respect POETRY_VIRTUALENVS_CREATE, so the application did not run. See:

Poetry did not install packages correctly with install-poetry.py, POETRY_VIRTUALENVS_CREATE, and Python 3.10

Poetry errored out with a JSONDecodeError when attempting to install packages with Python 3.10. See python-poetry/poetry#4210. This appeared to be resolved as of Poetry 1.2.0a2.

There were also errors with Poetry's own virtualenv when using install-poetry.py, POETRY_VIRTUALENVS_CREATE, and multiple versions of Python, including 3.8 and 3.9. Error tracebacks typically reported a traceback sequence of CalledProcessError -> EnvCommandError -> PoetryException for each package, and appeared to involve virtualenv and pip. These errors may have been related to Poetry's "new installer", which can be disabled with the command poetry config experimental.new-installer false or the environment variable POETRY_EXPERIMENTAL_NEW_INSTALLER=false.

Example traceback from a GitHub Actions run:

  CalledProcessError

  Command '['/opt/poetry/venv/bin/python', '/opt/poetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.0.1-py3-none-any.whl/pip', 'install', '--disable-pip-version-check', '--prefix', '/opt/hostedtoolcache/Python/3.8.11/x64', '--no-deps', 'file:/home/runner/.cache/pypoetry/artifacts/94/aa/c7/f2115774653c0d0bf72ce93092eb73de70854164adf9f00749a952ba9c/websockets-9.1-cp38-cp38-manylinux2010_x86_64.whl']' returned non-zero exit status 1.

  at /opt/hostedtoolcache/Python/3.8.11/x64/lib/python3.8/subprocess.py:516 in run
       512│             # We don't call process.wait() as .__exit__ does that for us.
       513│             raise
       514│         retcode = process.poll()
       515│         if check and retcode:
    →  516│             raise CalledProcessError(retcode, process.args,
       517│                                      output=stdout, stderr=stderr)
       518│     return CompletedProcess(process.args, retcode, stdout, stderr)
       519│
       520│

The following error occurred when trying to handle this error:


  EnvCommandError

  Command ['/opt/poetry/venv/bin/python', '/opt/poetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.0.1-py3-none-any.whl/pip', 'install', '--disable-pip-version-check', '--prefix', '/opt/hostedtoolcache/Python/3.8.11/x64', '--no-deps', 'file:/home/runner/.cache/pypoetry/artifacts/94/aa/c7/f2115774653c0d0bf72ce93092eb73de70854164adf9f00749a952ba9c/websockets-9.1-cp38-cp38-manylinux2010_x86_64.whl'] errored with the following return code 1, and output:
  Traceback (most recent call last):
    File "/opt/hostedtoolcache/Python/3.8.11/x64/lib/python3.8/runpy.py", line 194, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/opt/hostedtoolcache/Python/3.8.11/x64/lib/python3.8/runpy.py", line 87, in _run_code
      exec(code, run_globals)
    File "/opt/poetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.0.1-py3-none-any.whl/pip/__main__.py", line 24, in <module>
    File "/opt/poetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.0.1-py3-none-any.whl/pip/_internal/cli/main.py", line 71, in main
    File "/opt/poetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.0.1-py3-none-any.whl/pip/_internal/commands/__init__.py", line 96, in create_command
    File "/opt/hostedtoolcache/Python/3.8.11/x64/lib/python3.8/importlib/__init__.py", line 127, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
    File "<frozen importlib._bootstrap>", line 991, in _find_and_load
    File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
    File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
    File "<frozen zipimport>", line 259, in load_module
    File "/opt/poetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.0.1-py3-none-any.whl/pip/_internal/commands/install.py", line 15, in <module>
    File "<frozen importlib._bootstrap>", line 991, in _find_and_load
    File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
    File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
    File "<frozen zipimport>", line 259, in load_module
    File "/opt/poetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.0.1-py3-none-any.whl/pip/_internal/cli/req_command.py", line 21, in <module>
    File "<frozen importlib._bootstrap>", line 991, in _find_and_load
    File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
    File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
    File "<frozen zipimport>", line 259, in load_module
    File "/opt/poetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.0.1-py3-none-any.whl/pip/_internal/req/__init__.py", line 8, in <module>
    File "<frozen importlib._bootstrap>", line 991, in _find_and_load
    File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
    File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
    File "<frozen zipimport>", line 259, in load_module
    File "/opt/poetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.0.1-py3-none-any.whl/pip/_internal/req/req_install.py", line 32, in <module>
    File "<frozen importlib._bootstrap>", line 991, in _find_and_load
    File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
    File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
    File "<frozen zipimport>", line 259, in load_module
    File "/opt/poetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.0.1-py3-none-any.whl/pip/_internal/pyproject.py", line 4, in <module>
    File "<frozen importlib._bootstrap>", line 991, in _find_and_load
    File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
    File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
    File "<frozen zipimport>", line 259, in load_module
    File "/opt/poetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.0.1-py3-none-any.whl/pip/_vendor/toml/__init__.py", line 6, in <module>
    File "<frozen importlib._bootstrap>", line 991, in _find_and_load
    File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
    File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
    File "<frozen zipimport>", line 259, in load_module
    File "/opt/poetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.0.1-py3-none-any.whl/pip/_vendor/toml/encoder.py", line 6, in <module>
    File "<frozen importlib._bootstrap>", line 991, in _find_and_load
    File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
    File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
    File "<frozen zipimport>", line 259, in load_module
    File "/opt/poetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.0.1-py3-none-any.whl/pip/_vendor/toml/decoder.py", line 7, in <module>
    File "<frozen importlib._bootstrap>", line 991, in _find_and_load
    File "<frozen importlib._bootstrap>", line 971, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 914, in _find_spec
    File "<frozen importlib._bootstrap_external>", line 1407, in find_spec
    File "<frozen importlib._bootstrap_external>", line 1381, in _get_spec
    File "<frozen importlib._bootstrap_external>", line 1362, in _legacy_get_spec
    File "<frozen importlib._bootstrap>", line 414, in spec_from_loader
    File "<frozen importlib._bootstrap_external>", line 709, in spec_from_file_location
    File "<frozen zipimport>", line 191, in get_filename
    File "<frozen zipimport>", line 709, in _get_module_code
    File "<frozen zipimport>", line 536, in _get_data
  FileNotFoundError: [Errno 2] No such file or directory: '/opt/poetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.0.1-py3-none-any.whl'


  at /opt/poetry/venv/lib/python3.8/site-packages/poetry/utils/env.py:1300 in _run
      1296│                 output = subprocess.check_output(
      1297│                     cmd, stderr=subprocess.STDOUT, env=env, **kwargs
      1298│                 )
      1299│         except CalledProcessError as e:
    → 1300│             raise EnvCommandError(e, input=input_)
      1301│
      1302│         return decode(output)
      1303│
      1304│     def execute(self, bin: str, *args: str, **kwargs: Any) -> Optional[int]:

The following error occurred when trying to handle this error:


  PoetryException

  Failed to install file:/home/runner/.cache/pypoetry/artifacts/94/aa/c7/f2115774653c0d0bf72ce93092eb73de70854164adf9f00749a952ba9c/websockets-9.1-cp38-cp38-manylinux2010_x86_64.whl

  at /opt/poetry/venv/lib/python3.8/site-packages/poetry/utils/pip.py:60 in pip_install
       56│                     *env.get_pip_command(),
       57│                     *args,
       58│                     env={**os.environ, "PYTHONPATH": str(env.purelib)},
       59│                 )
    →  60│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
       61│
       62│
       63│ def pip_editable_install(directory: Path, environment: Env) -> Union[int, str]:
       64│     return pip_install(

May be somewhat related to:

install-poetry.py was not present in stable releases

Poetry 1.1 releases did not include install-poetry.py in their source tree. This could be a problem for projects that fetch install-poetry.py from a specific Git tag, as this project does (#44).

For example:

pydantic and typing

Related

GitHub Actions

Poetry

pydantic and typing

@vercel
Copy link

vercel bot commented Jul 5, 2021

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://vercel.com/br3ndonland/inboard/7yDAzjvQKWR2tA7jWXhBuS6jFGPL
✅ Preview: https://inboard-git-python-310-br3ndonland.vercel.app

@br3ndonland br3ndonland mentioned this pull request Sep 19, 2021
1 task
br3ndonland added a commit that referenced this pull request Sep 19, 2021
#36
#44

PR #44 introduced version tests for Poetry. The GitHub Actions workflow
builds.yml was matching the version number as a substring anywhere in
the output of `poetry -V`, but the hooks.yml and tests.yml workflows
were matching the version number at the end of the `poetry -V` output.
This would have become an issue if the format of the output was changed,
and sure enough, Poetry 1.2.0 changes the format to have parentheses
around the version number.

This commit will update the hooks.yml and tests.yml workflows to match
the version number as a substring anywhere in the output of `poetry -V`.
br3ndonland added a commit to br3ndonland/dotfiles that referenced this pull request Oct 19, 2021
1d8eee0
884d475

Poetry has a new install script, install-poetry.py, which alters the
requirements for adding Poetry to `$PATH`. `$HOME/.local/bin` was
already on `$PATH` for pipx, so it seemed like a good option. Commits
1d8eee0 and 884d475 updated `.zshrc` and `script/strap-after-setup` for
install-poetry.py and `POETRY_HOME=$HOME/.local`.

This made sense initially, because Poetry installs its binaries into
`$POETRY_HOME/bin`, and because Poetry doesn't have a `$POETRY_BIN_DIR`
configuration variable like pipx does (`$PIPX_BIN_DIR`). Unfortunately,
`POETRY_HOME=$HOME/.local` ended up being problematic, because Poetry
takes over `$POETRY_HOME`, and doesn't consider other applications
installed there. For example, if the get-poetry.py or install-poetry.py
scripts were used to install Poetry, they can also be used to uninstall
Poetry. Uninstalling with `python install-poetry.py --uninstall` or
`python get-poetry.py --uninstall` deletes the entire `$POETRY_HOME`
directory, which means it deletes `$HOME/.local`, causing problems for
other applications that use `$HOME/.local` (python-poetry/poetry#4625).

There have been many other issues with the Poetry custom install scripts
get-poetry.py and install-poetry.py (br3ndonland/inboard#36), so other
installation methods are be welcome.

Poetry is now available through Homebrew, but Homebrew installation is
not supported by the Poetry maintainers. Homebrew installation also
requires its own custom install script, which creates its own issues.
python-poetry/poetry#941
python-poetry/poetry#1765
Homebrew/homebrew-core#48883
Homebrew/homebrew-core#86776

pipx (https://pypa.github.io/pipx/) can also be used to install Poetry.
The pipx installation method is suggested in the Poetry docs and GitHub,
and pipx is already in use in this repo.
python-poetry/poetry#677
python-poetry/poetry#3360

This commit will remove `export POETRY_HOME=$HOME/.local` from `.zshrc`,
and will install Poetry with pipx.
br3ndonland added a commit to br3ndonland/template-python that referenced this pull request Oct 24, 2021
br3ndonland/fastenv#7
br3ndonland/inboard#36

Python versions must now be quoted in YAML. `3.10` (without quotes) is
interpreted as a float and becomes `3.1`. `"3.10"` will be used instead.
@codecov
Copy link

codecov bot commented Oct 26, 2021

Codecov Report

Merging #36 (a1f7241) into develop (97e35b7) will not change coverage.
The diff coverage is n/a.

❗ Current head a1f7241 differs from pull request most recent head f3fd95d. Consider uploading reports for the commit f3fd95d to get more accurate results

@@            Coverage Diff            @@
##           develop       python-poetry/poetry#36   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           10        10           
  Lines          281       264   -17     
=========================================
- Hits           281       264   -17     
Flag Coverage Δ
unit 100.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
inboard/start.py 100.00% <0.00%> (ø)
inboard/logging_conf.py 100.00% <0.00%> (ø)
inboard/app/main_fastapi.py 100.00% <0.00%> (ø)
inboard/app/utilities_fastapi.py 100.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 97e35b7...f3fd95d. Read the comment docs.

Beta releases of Python 3.10 were installed with `"3.10.0-beta - 3.10"`.
Python 3.10 is now stable, so beta versions are no longer needed.

Python versions in YAML must now be quoted. `3.10` is interpreted as a
float and becomes `3.1`, so `"3.10"` must be used instead.
The Python version can be customized by using a Docker build argument,
such as `--build-arg PYTHON_VERSION=3.9`.
This commit will update type annotation syntax for Python 3.10. The
project currently also supports Python 3.8 and 3.9, so the annotations
are imported with `from __future__ import annotations`.

The Python 3.10 union operator (the pipe, like `str | None`) will not be
used on pydantic models. If running Python 3.9 or below, pydantic is not
compatible with the union operator, even if annotations are imported
with `from __future__ import annotations`.

https://peps.python.org/pep-0604/
https://docs.python.org/3/whatsnew/3.10.html
pydantic/pydantic#2597 (comment)
pydantic/pydantic#2609 (comment)
pydantic/pydantic#3300 (comment)
@sourcery-ai
Copy link
Contributor

sourcery-ai bot commented Apr 2, 2022

Sourcery Code Quality Report

Merging this PR leaves code quality unchanged.

Quality metrics Before After Change
Complexity 2.07 ⭐ 2.07 ⭐ 0.00
Method Length 54.31 ⭐ 54.27 ⭐ -0.04 👍
Working memory 6.97 🙂 6.97 🙂 0.00
Quality 77.47% 77.47% 0.00%
Other metrics Before After Change
Lines 685 693 8
Changed files Quality Before Quality After Quality Change
inboard/gunicorn_conf.py 76.44% ⭐ 76.44% ⭐ 0.00%
inboard/logging_conf.py 75.15% ⭐ 75.15% ⭐ 0.00%
inboard/start.py 73.65% 🙂 73.65% 🙂 0.00%
inboard/app/main_base.py 81.09% ⭐ 81.09% ⭐ 0.00%
inboard/app/utilities_starlette.py 69.55% 🙂 69.55% 🙂 0.00%
tests/test_gunicorn_conf.py 75.65% ⭐ 75.65% ⭐ 0.00%
tests/app/test_main.py 82.97% ⭐ 82.97% ⭐ 0.00%

Here are some functions in these files that still need a tune-up:

File Function Complexity Length Working Memory Quality Recommendation
tests/test_gunicorn_conf.py TestGunicornSettings.test_gunicorn_config_with_custom_options 0 ⭐ 162 😞 9 🙂 63.76% 🙂 Try splitting into smaller methods
inboard/start.py run_pre_start_script 5 ⭐ 86 🙂 10 😞 66.01% 🙂 Extract out complex expressions
inboard/start.py _update_uvicorn_config_options 4 ⭐ 67 🙂 10 😞 69.84% 🙂 Extract out complex expressions
inboard/start.py set_uvicorn_options 1 ⭐ 70 🙂 11 😞 70.46% 🙂 Extract out complex expressions
inboard/logging_conf.py LogFilter.set_filters 2 ⭐ 32 ⭐ 10 😞 77.42% ⭐ Extract out complex expressions

Legend and Explanation

The emojis denote the absolute quality of the code:

  • ⭐ excellent
  • 🙂 good
  • 😞 poor
  • ⛔ very poor

The 👍 and 👎 indicate whether the quality has improved or gotten worse with this pull request.


Please see our documentation here for details on how these metrics are calculated.

We are actively working on this report - lots more documentation and extra metrics to come!

Help us improve this quality report!

@br3ndonland br3ndonland marked this pull request as ready for review April 2, 2022 21:26
@br3ndonland
Copy link
Owner Author

inboard is finally ready to support Python 3.10. Thanks to everyone for their patience over the past nine months.

@br3ndonland br3ndonland merged commit 7b2fdf6 into develop Apr 2, 2022
@br3ndonland br3ndonland deleted the python-3.10 branch April 2, 2022 21:42
br3ndonland added a commit that referenced this pull request Apr 2, 2022
#36

Type annotations were updated for Python 3.10 in #36.
This `from typing import Set` was added in #49 prior to merging #36,
and was overlooked when merging the PR.

This commit will replace `typing.Set` with a parametrized generic.
https://peps.python.org/pep-0585/
https://docs.python.org/3/whatsnew/3.9.html
br3ndonland added a commit to br3ndonland/dotfiles that referenced this pull request Oct 10, 2022
After creating numerous problems for maintainers by cramming breaking
changes into patch releases, Poetry finally released a minor version,
1.2, which should probably have been a major version.

In order to upgrade, each project will need to update its pyproject.toml
(to support groups), update any references to the install script (which
has been moved twice), and make several other associated changes.

This commit will set an upper bound on the `pipx install poetry` command
to avoid upgrading to Poetry 1.2.

br3ndonland/inboard#36
br3ndonland/inboard#44
br3ndonland/inboard#47
https://python-poetry.org/blog/announcing-poetry-1.2.0/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant