Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update Python dependency to 3.9 #2691

Closed
phillxnet opened this issue Oct 10, 2023 · 10 comments
Closed

Update Python dependency to 3.9 #2691

phillxnet opened this issue Oct 10, 2023 · 10 comments
Assignees
@phillxnet
Milestone
Py3.9

Comments

@phillxnet
Copy link
Member

phillxnet commented Oct 10, 2023 •
edited
Loading

To date, in the current testing channel, we have used Py3.6. However, even thought this is our upstream OS (bar TW) default it is EOL and constraining our options. With TW no longer providing even Py3.6. Ergo move to Py3.8 Py3.9 EDIT see comment: across all our OS targets by way of the smallest step available to us across our target OSs, as we work out the ramifications of this and some of our existing older dependencies.
@phillxnet phillxnet added this to the Py3.8 milestone Oct 10, 2023
@phillxnet phillxnet self-assigned this Oct 10, 2023
@phillxnet
Copy link
Member Author

Our TW target has already been relying on Py3.8 for the last couple of releases given there was no Py3.6 available, see:
#2620
and
rockstor/rockstor-rpmbuild#26

@phillxnet phillxnet mentioned this issue Oct 10, 2023
Closed
@phillxnet
Copy link
Member Author

Note that we have a now created and linked partner issue in our rockstor-rpmbuild repo:
rockstor/rockstor-rpmbuild#48

@phillxnet
Copy link
Member Author

With the following: which would be enacted by our rpm spec in linked issue:

lbuildvm:/opt/rockstor # zypper in python38 python38-devel

we have:

Retrieving repository 'Update repository with updates from SUSE Linux Enterprise 15' metadata ..............................[done]
Building repository 'Update repository with updates from SUSE Linux Enterprise 15' cache ...................................[done]
Loading repository data...
Reading installed packages...
'python38' not found in package names. Trying capabilities.
No provider of 'python38' found.
'python38-devel' not found in package names. Trying capabilities.
No provider of 'python38-devel' found.
Resolving package dependencies...
Nothing to do.

Where-as these are available in default Tumbleweed.

@phillxnet
Copy link
Member Author

With reference to the last comment, we do however have a Py3.9 available in our default Leap 15.5 repos:

Py3.8 Leap 15.5:

zypper info python38 python38-devel
Loading repository data...
Reading installed packages...


package 'python38' not found.

package 'python38-devel' not found.

Py3.9 Leap 15.5:

lbuildvm:/opt/rockstor # zypper info python39 python39-devel
Loading repository data...
Reading installed packages...


Information for package python39:
---------------------------------
Repository     : Update repository with updates from SUSE Linux Enterprise 15
Name           : python39
Version        : 3.9.18-150300.4.33.1
Arch           : x86_64
Vendor         : SUSE LLC <https://www.suse.com/>
Installed Size : 170.4 KiB
Installed      : No
Status         : not installed
Source package : python39-3.9.18-150300.4.33.1.src
Upstream URL   : https://www.python.org/
Summary        : Python 3 Interpreter
Description    : 
    Python 3 is modern interpreted, object-oriented programming language,
    often compared to Tcl, Perl, Scheme, or Java.  You can find an overview
    of Python in the documentation and tutorials included in the python3-doc
    package.

    This package supplies rich command line features provided by readline,
    and sqlite3 support for the interpreter core, thus forming a so called
    "extended" runtime.
    Installing "python3" is sufficient for the vast majority of usecases.
    In addition, recommended packages provide UI toolkit support (python3-curses,
    python3-tk), legacy UNIX database bindings (python3-dbm), and the IDLE
    development environment (python3-idle).


Information for package python39-devel:
---------------------------------------
Repository     : Update repository with updates from SUSE Linux Enterprise 15
Name           : python39-devel
Version        : 3.9.18-150300.4.33.1
Arch           : x86_64
Vendor         : SUSE LLC <https://www.suse.com/>
Installed Size : 918.6 KiB
Installed      : No
Status         : not installed
Source package : python39-core-3.9.18-150300.4.33.1.src
Upstream URL   : https://www.python.org/
Summary        : Include Files and Libraries Mandatory for Building Python Modules
Description    : 
    The Python programming language's interpreter can be extended with
    dynamically loaded extensions and can be embedded in other programs.

    This package contains header files, a static library, and development
    tools for building Python modules, extending the Python interpreter or
    embedding Python in applications.

    This also includes the Python distutils, which were in the Python
    package up to version 2.2.2.

So given the difficulty involved (read blocker) of adding say a devel language repo, we might look to instead jump to Py3.9 across the board.

@phillxnet
Copy link
Member Author

phillxnet commented Oct 10, 2023 •
edited
Loading

Re our testing branch current Django version of 2.2.28 we have the following Python version limits:
https://docs.djangoproject.com/en/4.2/releases/2.2/#python-compatibility

Django 2.2 supports Python 3.5, 3.6, 3.7, 3.8 (as of 2.2.8), and 3.9 (as of 2.2.17). We highly recommend and only officially support the latest release of each series.

So all good on Py3.9, but no further!

Assessing our current OS targets for Py3.9.

Leap 15.4

N.B. from: https://en.opensuse.org/openSUSE:Roadmap#Schedule_for_openSUSE_Leap_15.4
EOL is in around 2 months time !!
Has Py3.9 but no Py3.8 of both packages.

TW

Has Py3.8 & Py3.9 of both packages.

@phillxnet phillxnet changed the title Update Python dependency to 3.8 Update Python dependency to 3.9 Oct 10, 2023
@phillxnet
Copy link
Member Author

Leap 15.5 dev setup.

zypper in  --no-recommends python39 python39-devel
...
The following 4 NEW packages are going to be installed:
  libpython3_9-1_0 python39 python39-base python39-devel

@phillxnet
Copy link
Member Author

Our current .venv appears to be still locked to a Py3.6 reference:

lbuildvm:/opt/rockstor # poetry update
Updating dependencies
Resolving dependencies... (0.6s)

  SolverProblemError

  The current project's Python requirement (3.6.15) is not compatible with some of the required packages Python requirement:
    - gevent requires Python >=3.8, so it will not be satisfied for Python 3.6.15
  
  Because gevent (23.9.1) requires Python >=3.8
   and no versions of gevent match !=23.9.1, gevent is forbidden.
  So, because rockstor depends on gevent (*), version solving failed.

  at ~/.local/share/pypoetry/venv/lib64/python3.6/site-packages/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│ 
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

  • Check your dependencies Python requirement: The Python requirement can be specified via the `python` or `markers` properties
    
    For gevent, a possible solution would be to set the `python` property to "<empty>"

    https://python-poetry.org/docs/dependency-specification/#python-restricted-dependencies,
    https://python-poetry.org/docs/dependency-specification/#using-environment-markers

however we saw something similar in our early testing channel and so we can assume that the rpm update will wipe .venv and reinstate it as per:
Issue: rockstor/rockstor-rpmbuild#29
PR: rockstor/rockstor-rpmbuild#31

So we can assume our RPM update will do the following via scriptlet entries:

    # Remove our prior versions .venv dir as build.sh (posttrans) will regenerate.
    rm -rf %{prefix}/%{name}/.venv

https://github.com/rockstor/rockstor-rpmbuild/blob/26c6b1f740467c6c5fc77f23a4ab0d3135668d61/rockstor.spec#L356-L357

@phillxnet
Copy link
Member Author

Proof of concept (Leap 15.5 dev base)

With to-be-linked project.py changes to date:

lbuildvm:/opt/rockstor # rm -rf .venv/
lbuildvm:/opt/rockstor # poetry update
The currently activated Python version 3.6.15 is not supported by the project (~3.9).
Trying to find and use a compatible version. 
Using python3.9 (3.9.18)
Creating virtualenv rockstor in /opt/rockstor/.venv
Updating dependencies
Resolving dependencies... (0.5s)

Package operations: 31 installs, 0 updates, 0 removals

  • Installing certifi (2023.7.22)
  • Installing charset-normalizer (2.0.12)
  • Installing greenlet (3.0.0)
  • Installing idna (3.4)
  • Installing pytz (2023.3.post1)
  • Installing six (1.16.0)
  • Installing sqlparse (0.4.4)
  • Installing urllib3 (1.26.17)
  • Installing zope.event (5.0)
  • Installing zope.interface (6.1)
  • Installing django (2.2.28)
  • Installing gevent (23.9.1)
  • Installing oauthlib (3.1.0)
  • Installing python-engineio (2.3.2)
  • Installing requests (2.27.1)
  • Installing dbus-python (1.2.18)
  • Installing distro (1.8.0)
  • Installing django-braces (1.13.0)
  • Installing django-oauth-toolkit (1.1.2)
  • Installing django-pipeline (1.7.0)
  • Installing djangorestframework (3.9.3)
  • Installing gevent-websocket (0.10.1)
  • Installing gunicorn (19.10.0)
  • Installing huey (2.3.0)
  • Installing psutil (5.9.4)
  • Installing psycogreen (1.0)
  • Installing psycopg2 (2.8.6)
  • Installing python-socketio (1.6.0)
  • Installing pyzmq (19.0.2)
  • Installing supervisor (4.2.4)
  • Installing urlobject (2.1.1)
lbuildvm:/opt/rockstor #

phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Oct 10, 2023
@phillxnet
Update Python dependency to 3.9 rockstor#2691
0826ec9 
Update from Py3.6 to Py3.9 as commonly available in
default repos across our Leap 15.4/15.5/TW
'Built on ...' OS targets.
@phillxnet
Copy link
Member Author

Emulating an rpm update for now we have, under a Py3.9:

----------------------------------------------------------------------
Ran 262 tests in 27.592s

OK

Fancy that.
I'll preset this as PR soon and begin on partner rockstor-rpmbuild issue/pr side to enable a full tests of rpm build on all platforms.

I suspect we have left our gunicorn version in the dust here so will likely have to chase that up in a follow-up issue of it's own.

lbuildvm:~ # systemctl status rockstor*
● rockstor-pre.service - Tasks required prior to starting Rockstor
     Loaded: loaded (/usr/lib/systemd/system/rockstor-pre.service; disabled; vendor preset: disabled)
     Active: active (exited) since Tue 2023-10-10 19:23:38 WEST; 9min ago
    Process: 10351 ExecStart=/root/.local/bin/poetry run initrock (code=exited, status=0/SUCCESS)
   Main PID: 10351 (code=exited, status=0/SUCCESS)

Oct 10 19:23:37 lbuildvm poetry[10351]: 2023-10-10 19:23:37,967: /etc/systemd/system/nginx.service.d/30-rockstor-nginx-override.conf up-to-date.
Oct 10 19:23:37 lbuildvm poetry[10351]: 2023-10-10 19:23:37,968: /usr/lib/systemd/system/rockstor-pre.service up-to-date.
Oct 10 19:23:37 lbuildvm poetry[10351]: 2023-10-10 19:23:37,969: /usr/lib/systemd/system/rockstor.service up-to-date.
Oct 10 19:23:37 lbuildvm poetry[10351]: 2023-10-10 19:23:37,969: /usr/lib/systemd/system/rockstor-bootstrap.service up-to-date.
Oct 10 19:23:37 lbuildvm poetry[10351]: 2023-10-10 19:23:37,969: ### BEGIN Establishing poetry path to binaries in local files...
Oct 10 19:23:37 lbuildvm poetry[10351]: 2023-10-10 19:23:37,970: samba_config already looks good.
Oct 10 19:23:37 lbuildvm poetry[10351]: 2023-10-10 19:23:37,971: rockstor_crontab already looks good.
Oct 10 19:23:37 lbuildvm poetry[10351]: 2023-10-10 19:23:37,971: The replication_crontab (/etc/cron.d/replicationtab) could not be found
Oct 10 19:23:37 lbuildvm poetry[10351]: 2023-10-10 19:23:37,972: ### DONE establishing poetry path to binaries in local files.
Oct 10 19:23:38 lbuildvm systemd[1]: Finished Tasks required prior to starting Rockstor.

● rockstor.service - Rockstor startup script
     Loaded: loaded (/usr/lib/systemd/system/rockstor.service; enabled; vendor preset: disabled)
     Active: active (running) since Tue 2023-10-10 19:23:38 WEST; 9min ago
   Main PID: 10994 (supervisord)
      Tasks: 9 (limit: 4915)
     CGroup: /system.slice/rockstor.service
             ├─ 10994 /opt/rockstor/.venv/bin/python /opt/rockstor/.venv/bin/supervisord -c /opt/rockstor/etc/supervisord.conf
             ├─ 11045 /opt/rockstor/.venv/bin/python /opt/rockstor/.venv/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/>
             ├─ 11046 /opt/rockstor/.venv/bin/python /opt/rockstor/.venv/bin/data-collector
             ├─ 11047 /opt/rockstor/.venv/bin/python /opt/rockstor/.venv/bin/django-admin run_huey --workers 2 --worker-type thread --logfile /opt/rockstor/var/log/huey.log
             ├─ 11050 /opt/rockstor/.venv/bin/python /opt/rockstor/.venv/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/>
             └─ 11051 /opt/rockstor/.venv/bin/python /opt/rockstor/.venv/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/>

Oct 10 19:23:39 lbuildvm poetry[10994]: 2023-10-10 19:23:39,302 INFO RPC interface 'supervisor' initialized
Oct 10 19:23:39 lbuildvm poetry[10994]: 2023-10-10 19:23:39,302 INFO supervisord started with pid 10994
Oct 10 19:23:40 lbuildvm poetry[10994]: 2023-10-10 19:23:40,306 INFO spawned: 'gunicorn' with pid 11045
Oct 10 19:23:40 lbuildvm poetry[10994]: 2023-10-10 19:23:40,308 INFO spawned: 'data-collector' with pid 11046
Oct 10 19:23:40 lbuildvm poetry[10994]: 2023-10-10 19:23:40,310 INFO spawned: 'ztask-daemon' with pid 11047
Oct 10 19:23:42 lbuildvm poetry[10994]: 2023-10-10 19:23:42,722 INFO success: data-collector entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
Oct 10 19:23:45 lbuildvm poetry[10994]: 2023-10-10 19:23:45,726 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
Oct 10 19:23:50 lbuildvm poetry[10994]: 2023-10-10 19:23:50,733 INFO success: ztask-daemon entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
Oct 10 19:25:34 lbuildvm useradd[12024]: new user: name=radmin, UID=1000, GID=100, home=/home/radmin, shell=/bin/bash, from=none
Oct 10 19:25:34 lbuildvm usermod[12033]: change user 'radmin' password

● rockstor-bootstrap.service - Rockstor bootstrapping tasks
     Loaded: loaded (/usr/lib/systemd/system/rockstor-bootstrap.service; disabled; vendor preset: enabled)
     Active: active (exited) since Tue 2023-10-10 19:23:39 WEST; 9min ago
    Process: 10995 ExecStart=/opt/rockstor/.venv/bin/bootstrap (code=exited, status=0/SUCCESS)
   Main PID: 10995 (code=exited, status=0/SUCCESS)

Oct 10 19:23:38 lbuildvm systemd[1]: Starting Rockstor bootstrapping tasks...
Oct 10 19:23:39 lbuildvm bootstrap[10995]: BTRFS device scan complete
Oct 10 19:23:39 lbuildvm bootstrap[10995]: Appliance is not yet setup.
Oct 10 19:23:39 lbuildvm systemd[1]: Finished Rockstor bootstrapping tasks.
lbuildvm:~ #

This was referenced Oct 10, 2023
Merged
Closed
phillxnet added a commit that referenced this issue Oct 12, 2023
@phillxnet
Merge pull request #2692 from phillxnet/2691-Update-Python-dependency…
8a4f59e 
…-to-3.9

Update Python dependency to 3.9 #2691
@phillxnet
Copy link
Member Author

Closing as:
Fixed by #2692

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@phillxnet phillxnet

Labels
None yet
Projects
None yet
Milestone
Py3.9
Development

No branches or pull requests
1 participant
@phillxnet