Update Poetry re rpmbuild & normalise on Python 3.11 #50 #52

phillxnet · 2023-11-23T18:25:02Z

Align with rockstor-core build.sh re Poetry update and install via OS provided pipx under Python 3.11.

Thanks to @FroggyFlox for assisting with development.

Includes

Path changes regarding new Poetry install location.
All targets move from Python 3.9 to Python 3.11.
Additional python311-pipx BuildRequires and Requires.
Additional update-alternatives to enforce Py3.11 pipx.

Fixes #50

See also partner:
Issue re rockstor-core: rockstor/rockstor-core#2703
Pull request re rockstor-core: rockstor/rockstor-core#2755

Align with rockstor-core `build.sh` re Poetry update and install via OS provided pipx under Python 3.11. ## Includes - Path changes regarding new Poetry install location. - All targets move from Python 3.9 to Python 3.11. - Additional python311-pipx BuildRequires and Requires. - Additional update-alternatives to enforce Py3.11 pipx.

phillxnet · 2023-11-24T11:38:45Z

Testing

We have the intended new build dependencies doing their thing:

error: Failed build dependencies:
	python311 is needed by rockstor-5.0.5-2755.x86_64
	python311-devel is needed by rockstor-5.0.5-2755.x86_64
	python311-pipx is needed by rockstor-5.0.5-2755.x86_64

phillxnet · 2023-11-24T12:16:56Z

rpmbuild - legacy Poetry in-place

%build scriptlet:

Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.N0yodl
+ umask 022
+ cd /home/buildbot/worker/Poetry-Build-on-Leap15-5/rpmbuild
+ /usr/bin/rm -rf /home/buildbot/worker/Poetry-Build-on-Leap15-5/rpmbuild/BUILDROOT/rockstor-5.0.5-2755.x86_64
++ dirname /home/buildbot/worker/Poetry-Build-on-Leap15-5/rpmbuild/BUILDROOT/rockstor-5.0.5-2755.x86_64
+ /usr/bin/mkdir -p /home/buildbot/worker/Poetry-Build-on-Leap15-5/rpmbuild/BUILDROOT
+ /usr/bin/mkdir /home/buildbot/worker/Poetry-Build-on-Leap15-5/rpmbuild/BUILDROOT/rockstor-5.0.5-2755.x86_64
+ cd rockstor-core-5.0.5-2755
+ PATH=/root/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ which poetry
/root/.local/bin/poetry
+ poetry --version
+ grep -q 1.1.15
+ echo 'Poetry version 1.1.15 found - UNINSTALLING'
Poetry version 1.1.15 found - UNINSTALLING
+ python3 - --uninstall
+ curl -sSL https://install.python-poetry.org
Removing Poetry (1.1.15)
+ export PIPX_HOME=/opt/pipx
+ PIPX_HOME=/opt/pipx
+ export PIPX_BIN_DIR=/usr/local/bin
+ PIPX_BIN_DIR=/usr/local/bin
+ python3.11 -m pipx install poetry==1.7.1
creating virtual environment...
creating shared libraries...
upgrading shared libraries...
installing poetry from spec 'poetry==1.7.1'...
done! ✨ 🌟 ✨
  installed package poetry 1.7.1, installed using Python 3.11.5
  These apps are now globally available
    - poetry
+ poetry build --format sdist
Creating virtualenv rockstor in /home/buildbot/worker/Poetry-Build-on-Leap15-5/rpmbuild/rockstor-core-5.0.5-2755/.venv
Building rockstor (5.0.5)
+ exit 0

We have a successful legacy Poety uninstall (as no update path).
We have a successful Poetry 1.7.1 install via OS pipx (3.11) under Py3.11.

%check scriptlet

Executing(%check): /bin/sh -e /var/tmp/rpm-tmp.erS4tl
+ umask 022
+ cd /home/buildbot/worker/Poetry-Build-on-Leap15-5/rpmbuild
+ cd rockstor-core-5.0.5-2755
+ export LANG=C.UTF-8
+ LANG=C.UTF-8
+ export PYTHONIOENCODING=utf8
+ PYTHONIOENCODING=utf8
+ /usr/local/bin/poetry install --no-interaction --no-ansi
+ export DJANGO_SETTINGS_MODULE=settings
+ DJANGO_SETTINGS_MODULE=settings
+ /usr/local/bin/poetry run django-admin collectstatic --no-input --verbosity 1
515 static files copied to '/home/buildbot/worker/Poetry-Build-on-Leap15-5/rpmbuild/rockstor-core-5.0.5-2755/static', 517 post-processed.
+ cd src/rockstor/
+ /usr/local/bin/poetry run django-admin test
Creating test database for alias 'default'...
Found 278 test(s).
Creating test database for alias 'smart_manager'...
System check identified no issues (0 silenced).
......................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 278 tests in 30.173s
OK
Destroying test database for alias 'default'...
Destroying test database for alias 'smart_manager'...
+ exit 0

We have a working /usr/local/bin/poetry install --no-interaction --no-ansi to create our rpmbuild test run .venv.
We then have a successful poetry run django-admin collectstatic ... to prepare our static files.
And finally we execute all test (under poetry).

phillxnet · 2023-11-24T12:46:48Z

Resulting rpm install

The resulting test rpm, installed on the build host, has the following of note in:

%posttrans scriptlet

Here we do our proper install via our rockstor-core build.sh just instantiated by the build rpm.

Executing %posttrans script 'rockstor-5.0.5-2755.x86_64.rpm' [..
/usr/local/bin/poetry
'poetry' already seems to be installed. Not modifying existing installation in
'/opt/pipx/venvs/poetry'. Pass '--force' to force installation.
Creating jslibs/js/lib & populating from rockstor-jslibs.tar.gz
rockstor-jslibs-5.0.5/DataTables-addons/
rockstor-jslibs-5.0.5/DataTables-addons/file-size.js
...
515 static files copied to '/opt/rockstor/static', 517 post-processed.
ROCKSTOR BUILD SCRIPT COMPLETED
If installing from source, from scratch, for development; i.e. NOT via RPM:
1. Run 'cd /opt/rockstor'.
2. Run 'systemctl start postgresql'.
3. Run 'export DJANGO_SETTINGS_MODULE=settings'.
4. Run 'poetry run initrock' as root (equivalent to rockstor-pre.service).
5. Run 'systemctl enable --now rockstor-bootstrap'.
No config-backups-rpmsave directory found.
..done]
program finished with exit code 0
elapsedTime=19.062524

We have our new "... i.e. NOT via RPM:" instruction and an apparent pickup of the installed during rpmbuild global poetry.

poetry-install.txt

rleap15-5:/opt/rockstor # cat poetry-install.txt
Creating virtualenv rockstor in /opt/rockstor/.venv
Installing dependencies from lock file

Package operations: 42 installs, 1 update, 0 removals

  • Installing h11 (0.14.0)
  • Installing pycparser (2.21)
  • Installing cffi (1.16.0)
  • Installing wrapt (1.16.0)
  • Installing wsproto (1.2.0)
  • Installing asgiref (3.7.2)
  • Installing certifi (2023.11.17)
  • Installing charset-normalizer (3.3.2)
  • Installing cryptography (41.0.5)
  • Installing deprecated (1.2.14)
  • Installing idna (3.4)
  • Updating setuptools (68.2.2 -> 69.0.2)
  • Installing simple-websocket (1.0.0)
  • Installing sqlparse (0.4.4)
  • Installing urllib3 (2.1.0)
  • Installing bidict (0.22.1)
  • Installing django (4.2.7)
  • Installing greenlet (3.0.1)
  • Installing jwcrypto (1.5.0)
  • Installing oauthlib (3.2.2)
  • Installing packaging (23.2)
  • Installing python-engineio (4.8.0)
  • Installing pytz (2023.3.post1)
  • Installing requests (2.31.0)
  • Installing typing-extensions (4.8.0)
  • Installing zope-event (5.0)
  • Installing zope-interface (6.1)
  • Installing dbus-python (1.3.2)
  • Installing distro (1.8.0)
  • Installing django-oauth-toolkit (2.3.0)
  • Installing django-pipeline (2.1.0)
  • Installing djangorestframework (3.14.0)
  • Installing gevent (23.9.1)
  • Installing gunicorn (21.2.0)
  • Installing huey (2.5.0)
  • Installing psutil (5.9.4)
  • Installing psycogreen (1.0)
  • Installing psycopg (3.1.13)
  • Installing python-socketio (5.9.0)
  • Installing pyzmq (25.1.1)
  • Installing six (1.16.0)
  • Installing supervisor (4.2.4)
  • Installing urlobject (2.1.1)

Installing the current project: rockstor (5.0.5)

Not sure why we have an Updating setuptools (68.2.2 -> 69.0.2) here as this is a clean rpm install so no prior .venv !! But this does not look to be a problem.

systemd services

rockstor-pre

rleap15-5:/opt/rockstor # systemctl status rockstor-pre.service 
● rockstor-pre.service - Tasks required prior to starting Rockstor
     Loaded: loaded (/usr/lib/systemd/system/rockstor-pre.service; disabled; vendor preset: disabled)
     Active: active (exited) since Fri 2023-11-24 12:10:34 WET; 32min ago
    Process: 26829 ExecStart=/usr/local/bin/poetry run initrock (code=exited, status=0/SUCCESS)
   Main PID: 26829 (code=exited, status=0/SUCCESS)

Nov 24 12:10:34 rleap15-5 poetry[26829]: 2023-11-24 12:10:34,455: /etc/systemd/system/nginx.service.d/30-rockstor-nginx-override.conf up-to-date.
Nov 24 12:10:34 rleap15-5 poetry[26829]: 2023-11-24 12:10:34,455: /usr/lib/systemd/system/rockstor-pre.service up-to-date.
Nov 24 12:10:34 rleap15-5 poetry[26829]: 2023-11-24 12:10:34,456: /usr/lib/systemd/system/rockstor.service up-to-date.
Nov 24 12:10:34 rleap15-5 poetry[26829]: 2023-11-24 12:10:34,456: /usr/lib/systemd/system/rockstor-bootstrap.service up-to-date.
Nov 24 12:10:34 rleap15-5 poetry[26829]: 2023-11-24 12:10:34,456: ### BEGIN Establishing poetry path to binaries in local files...
Nov 24 12:10:34 rleap15-5 poetry[26829]: 2023-11-24 12:10:34,457: samba_config already looks good.
Nov 24 12:10:34 rleap15-5 poetry[26829]: 2023-11-24 12:10:34,457: rockstor_crontab already looks good.
Nov 24 12:10:34 rleap15-5 poetry[26829]: 2023-11-24 12:10:34,457: The replication_crontab (/etc/cron.d/replicationtab) could not be found
Nov 24 12:10:34 rleap15-5 poetry[26829]: 2023-11-24 12:10:34,457: ### DONE establishing poetry path to binaries in local files.
Nov 24 12:10:34 rleap15-5 systemd[1]: Finished Tasks required prior to starting Rockstor.

We have our new Poetry path working in our ExecStart

rockstor

rleap15-5:/opt/rockstor # systemctl status rockstor.service 
● rockstor.service - Rockstor startup script
     Loaded: loaded (/usr/lib/systemd/system/rockstor.service; disabled; vendor preset: disabled)
     Active: active (running) since Fri 2023-11-24 12:10:34 WET; 34min ago
   Main PID: 27118 (supervisord)
      Tasks: 8 (limit: 2334)
     CGroup: /system.slice/rockstor.service
             ├─ 27118 /opt/rockstor/.venv/bin/python /opt/rockstor/.venv/bin/supervisord -c /opt/rockstor/etc/supervisord.conf
             ├─ 27128 /opt/rockstor/.venv/bin/python /opt/rockstor/.venv/bin/gunicorn --config ./conf/gunicorn.conf.py wsgi:application
             ├─ 27129 /opt/rockstor/.venv/bin/python /opt/rockstor/.venv/bin/data-collector
             ├─ 27130 /opt/rockstor/.venv/bin/python /opt/rockstor/.venv/bin/django-admin run_huey --workers 2 --worker-type thread --logfile /opt/rockstor/var/log/huey.log
             └─ 27131 /opt/rockstor/.venv/bin/python /opt/rockstor/.venv/bin/gunicorn --config ./conf/gunicorn.conf.py wsgi:application

Nov 24 12:10:34 rleap15-5 systemd[1]: Started Rockstor startup script.
Nov 24 12:10:35 rleap15-5 poetry[27118]: 2023-11-24 12:10:35,450 INFO Set uid to user 0 succeeded
Nov 24 12:10:35 rleap15-5 poetry[27118]: 2023-11-24 12:10:35,489 INFO RPC interface 'supervisor' initialized
Nov 24 12:10:35 rleap15-5 poetry[27118]: 2023-11-24 12:10:35,490 INFO supervisord started with pid 27118
Nov 24 12:10:36 rleap15-5 poetry[27118]: 2023-11-24 12:10:36,494 INFO spawned: 'gunicorn' with pid 27128
Nov 24 12:10:36 rleap15-5 poetry[27118]: 2023-11-24 12:10:36,496 INFO spawned: 'data-collector' with pid 27129
Nov 24 12:10:36 rleap15-5 poetry[27118]: 2023-11-24 12:10:36,498 INFO spawned: 'ztask-daemon' with pid 27130
Nov 24 12:10:38 rleap15-5 poetry[27118]: 2023-11-24 12:10:38,501 INFO success: data-collector entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
Nov 24 12:10:41 rleap15-5 poetry[27118]: 2023-11-24 12:10:41,506 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
Nov 24 12:10:46 rleap15-5 poetry[27118]: 2023-11-24 12:10:46,513 INFO success: ztask-daemon entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)

rockstor-bootstrap

rleap15-5:/opt/rockstor # systemctl status rockstor-bootstrap.service 
● rockstor-bootstrap.service - Rockstor bootstrapping tasks
     Loaded: loaded (/usr/lib/systemd/system/rockstor-bootstrap.service; enabled; vendor preset: enabled)
     Active: active (exited) since Fri 2023-11-24 12:10:35 WET; 35min ago
    Process: 27119 ExecStart=/opt/rockstor/.venv/bin/bootstrap (code=exited, status=0/SUCCESS)
   Main PID: 27119 (code=exited, status=0/SUCCESS)

Nov 24 12:10:34 rleap15-5 systemd[1]: Starting Rockstor bootstrapping tasks...
Nov 24 12:10:35 rleap15-5 bootstrap[27119]: BTRFS device scan complete
Nov 24 12:10:35 rleap15-5 bootstrap[27119]: Appliance is not yet setup.
Nov 24 12:10:35 rleap15-5 systemd[1]: Finished Rockstor bootstrapping tasks.

phillxnet · 2023-11-24T12:54:47Z

Poetry show

In the resulting rpm install we have a working poetry re:

rleap15-5:/opt/rockstor # poetry show --tree
dbus-python 1.3.2 Python bindings for libdbus
distro 1.8.0 Distro - an OS platform information API
django 4.2.7 A high-level Python web framework that encourages rapid development and clean, pragmatic design.
├── asgiref >=3.6.0,<4
├── sqlparse >=0.3.1
└── tzdata *
django-oauth-toolkit 2.3.0 OAuth2 Provider for Django
├── django >=2.2,<4.0.0 || >4.0.0
│   ├── asgiref >=3.6.0,<4 
│   ├── sqlparse >=0.3.1 
│   └── tzdata * 
├── jwcrypto >=0.8.0
│   ├── cryptography >=3.4 
│   │   └── cffi >=1.12 
│   │       └── pycparser * 
│   └── deprecated * 
│       └── wrapt >=1.10,<2 
├── oauthlib >=3.1.0
└── requests >=2.13.0
    ├── certifi >=2017.4.17 
    ├── charset-normalizer >=2,<4 
    ├── idna >=2.5,<4 
    └── urllib3 >=1.21.1,<3 
django-pipeline 2.1.0 Pipeline is an asset packaging library for Django.
djangorestframework 3.14.0 Web APIs for Django, made easy.
├── django >=3.0
│   ├── asgiref >=3.6.0,<4 
│   ├── sqlparse >=0.3.1 
│   └── tzdata * 
└── pytz *
gevent 23.9.1 Coroutine-based network library
├── cffi >=1.12.2
│   └── pycparser * 
├── greenlet >=3.0rc3
├── zope-event *
│   └── setuptools * 
└── zope-interface *
    └── setuptools * 
gunicorn 21.2.0 WSGI HTTP Server for UNIX
└── packaging *
huey 2.5.0 huey, a little task queue
psutil 5.9.4 Cross-platform lib for process and system monitoring in Python.
psycogreen 1.0 psycopg2 integration with coroutine libraries
psycopg 3.1.13 PostgreSQL database adapter for Python
├── typing-extensions >=4.1
└── tzdata *
python-engineio 4.8.0 Engine.IO server and client for Python
└── simple-websocket >=0.10.0
    └── wsproto * 
        └── h11 >=0.9.0,<1 
python-socketio 5.9.0 Socket.IO server and client for Python
├── bidict >=0.21.0
└── python-engineio >=4.7.0
    └── simple-websocket >=0.10.0 
        └── wsproto * 
            └── h11 >=0.9.0,<1 
pyzmq 25.1.1 Python bindings for 0MQ
└── cffi *
    └── pycparser * 
six 1.16.0 Python 2 and 3 compatibility utilities
supervisor 4.2.4 A system for controlling process state under UNIX
└── setuptools *
urlobject 2.1.1 A utility class for manipulating URLs.

&

rleap15-5:/opt/rockstor # poetry env info

Virtualenv
Python:         3.11.5
Implementation: CPython
Path:           /opt/rockstor/.venv
Executable:     /opt/rockstor/.venv/bin/python
Valid:          True

System
Platform:   linux
OS:         posix
Python:     3.11.5
Path:       /usr
Executable: /usr/bin/python3.11

&

rleap15-5:/opt/rockstor # poetry env list
.venv (Activated)

rleap15-5:/opt/rockstor # poetry --version
Poetry (version 1.7.1)

phillxnet · 2023-11-24T13:10:54Z

The rpm build against this PR and its referenced rockstor-core partner, also successfully rebooted with all services starting as expected.

phillxnet · 2023-11-24T14:44:18Z

rpmbuild - pipx installed Poetry in-place

Testing for issues with a repeat rpmbuild, where an existing instance of the new 1.7.1 pipx installer Poetry already exists.

%build scriptlet:

The exact same rpm was re-build as per comment: (#52 (comment))

Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.uc9BOi
+ umask 022
+ cd /home/buildbot/worker/Poetry-Build-on-Leap15-5/rpmbuild
+ /usr/bin/rm -rf /home/buildbot/worker/Poetry-Build-on-Leap15-5/rpmbuild/BUILDROOT/rockstor-5.0.5-2755.x86_64
++ dirname /home/buildbot/worker/Poetry-Build-on-Leap15-5/rpmbuild/BUILDROOT/rockstor-5.0.5-2755.x86_64
+ /usr/bin/mkdir -p /home/buildbot/worker/Poetry-Build-on-Leap15-5/rpmbuild/BUILDROOT
+ /usr/bin/mkdir /home/buildbot/worker/Poetry-Build-on-Leap15-5/rpmbuild/BUILDROOT/rockstor-5.0.5-2755.x86_64
+ cd rockstor-core-5.0.5-2755
+ PATH=/root/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ which poetry
/usr/local/bin/poetry
+ poetry --version
+ grep -q 1.1.15
+ export PIPX_HOME=/opt/pipx
+ PIPX_HOME=/opt/pipx
+ export PIPX_BIN_DIR=/usr/local/bin
+ PIPX_BIN_DIR=/usr/local/bin
+ python3.11 -m pipx install poetry==1.7.1
'poetry' already seems to be installed. Not modifying existing installation in
'/opt/pipx/venvs/poetry'. Pass '--force' to force installation.
+ poetry build --format sdist
Creating virtualenv rockstor in /home/buildbot/worker/Poetry-Build-on-Leap15-5/rpmbuild/rockstor-core-5.0.5-2755/.venv
Building rockstor (5.0.5)
+ exit 0

We successfully pick-up the existing non legacy poetry, with no attempt to uninstall.

phillxnet · 2023-11-24T15:08:33Z

@FroggyFlox & @Hooverdan96 Merging as-is as we clear up three outstanding issues (rockstor-core partner PR), with a hefty technical dept assistance; and we have clear-cut positive test results re the indicated issues there and here in rockstor-rpmbuild where we proven rpmbuild and repeat rpmbuild in concert with the rockstor-core PR.

Plus these multi repo partnered issues are a little cumbersome to have hanging open.

These changes may come back to haunt us be we are still mid to mid-late testing phase for some more field tests to this end - i.e. updating from prior rpms and future updates newer rpms to follow.

phillxnet mentioned this pull request Nov 23, 2023

Update Poetry build system & normalise on Python 3.11 #2703 #2754 #2693 rockstor/rockstor-core#2755

Merged

phillxnet merged commit d6b0d84 into rockstor:testing Nov 24, 2023

phillxnet deleted the 50-Update-Poetry-re-rpmbuild-&-normalise-on-Python-3.11 branch November 24, 2023 15:35

phillxnet mentioned this pull request Nov 24, 2023

Update Poetry re rpmbuild & normalise on Python 3.11 #50

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Poetry re rpmbuild & normalise on Python 3.11 #50 #52

Update Poetry re rpmbuild & normalise on Python 3.11 #50 #52

phillxnet commented Nov 23, 2023 •

edited

Loading

phillxnet commented Nov 24, 2023

phillxnet commented Nov 24, 2023 •

edited

Loading

phillxnet commented Nov 24, 2023

phillxnet commented Nov 24, 2023

phillxnet commented Nov 24, 2023 •

edited

Loading

phillxnet commented Nov 24, 2023

phillxnet commented Nov 24, 2023

Update Poetry re rpmbuild & normalise on Python 3.11 #50 #52

Update Poetry re rpmbuild & normalise on Python 3.11 #50 #52

Conversation

phillxnet commented Nov 23, 2023 • edited Loading

Includes

phillxnet commented Nov 24, 2023

Testing

phillxnet commented Nov 24, 2023 • edited Loading

rpmbuild - legacy Poetry in-place

%build scriptlet:

%check scriptlet

phillxnet commented Nov 24, 2023

Resulting rpm install

%posttrans scriptlet

poetry-install.txt

systemd services

rockstor-pre

rockstor

rockstor-bootstrap

phillxnet commented Nov 24, 2023

Poetry show

phillxnet commented Nov 24, 2023 • edited Loading

phillxnet commented Nov 24, 2023

rpmbuild - pipx installed Poetry in-place

%build scriptlet:

phillxnet commented Nov 24, 2023

phillxnet commented Nov 23, 2023 •

edited

Loading

phillxnet commented Nov 24, 2023 •

edited

Loading

phillxnet commented Nov 24, 2023 •

edited

Loading