Provide wheel for Linux and OSX [was: Install psutil without gcc?]

[guettli]

This is a question: Is it possible to install psutil without gcc?

How much work would it be to port the code to ctypes (or other way to run without gcc)?

[giampaolo]

No, it is not possible. Converting the code base to ctypes for all platforms is a huge work. The main drawback I see is that you get rid of years of tested and working code and replace it with something brand new and not tested as much. Basically all the motivations I gave in #591 (comment) apply here as well. Why installing gcc is a problem for you?

[guettli]

Thank you for your feedback.

Installing without gcc is a restriction from the one providing the environment.

I don't like this restriction and I think it is useless, too.

I can work around it:

• install it via rpm
• install it via wheel (never tried before).

If you see a different (or easier) way, please tell me. My question was answered. Thank you. Closing this.

[giampaolo]

On Linux / Unix the only way you have to install psutil right now is via source / tarball. I don't want to provide wheels for Linux (or other UNIX platforms). I would have to cover all supported python versions (7) both 32 and 64 bits, meaning 14 extra packages to compile and upload on PYPI on every release. I do that for Windows because installing VS is an order of magnitude more difficult than installing gcc on Linux/UNIX but again: not willing to do extra work on that front (sorry).
What you could do is create a wheel yourself with python setup.py build bdist_wheel by using the same python/arch version you have on the server, upload it on the server and install it with pip.

[guettli]

If the wheel folks want wheels to be successful, then they could provide a build server. Leaving building wheels to the developers is a waste of time and energy.

And non-repeatable builds will be the result if every body builds his wheels in a different build environment...

I asked on the distutils mailinglist what they think about it: https://mail.python.org/pipermail/distutils-sig/2016-May/028994.html

[NotSqrt]

With https://github.com/pypa/manylinux being quite extensively used, can this be reopened ?

[giampaolo]

In practical terms what would it take to take advantage of manylinux? Is it something which can be integrated and automatized via travis or it would require me to set up a local environment consisting of docker + manylinux + whatever?

As of now I upload the tar.gz from my linux laptop and the windows exes/wheels are hosted on appveyor (I download them locally then I upload them on PYPI). How would this kind of workflow gonna change with manylinux?

[NotSqrt]

Have a look at https://github.com/pypa/python-manylinux-demo
and https://github.com/pypa/python-manylinux-demo/blob/master/travis/build-wheels.sh

Basically, you call that script that builds the wheels for multiple python versions and use auditwheel to package .so libraries, inside a specially prepared docker container (quay.io/pypa/manylinux1_XXX).

It can be achieved locally, or you can make travis call those scripts and upload the wheels.

[NotSqrt]

You can also look at psycopg2 : they have a separate git repo with the tools for building each new release : https://github.com/psycopg/psycopg2-wheels

[giampaolo]

Not willing to do it myself locally on every release. Do you have an example for Travis?

[giampaolo]

Oh, psycopg has it, nevermind.

[bbinet]

I'm also interested in having psutil python wheels on linux: any updates on using manylinux to create the wheels?

[giampaolo]

The best way to do this would be to have Travis build the wheels on every commit and save them somewhere as "artifacts" then download them locally and finally upload them on pypi with twine. We currently use this strategy for windows via appveyor.com. The same can also be done for Linux and possibly OS X. The goal is to figure out how to do this via .travis.yml. I currently don't have time to look into this so if somebody wants to give it a try.... I am all for it.

On Wed, 6 Sep 2017 at 18:52, Bruno Binet ***@***.***> wrote: I'm also interested in having psutil python wheels on linux: any updates on using manylinux to create the wheels? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#824 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAplLC1ZrRcyBjqDBkuYxtIPkFZBrS_Hks5sfnl9gaJpZM4IlKUi> .

-- Giampaolo - http://grodola.blogspot.com

[giampaolo]

Re-opening. Goal: provide wheels for Linux and possibly OSX via travis CI.

[guettli]

@giampaolo I whish you good luck with "provide wheels for Linux and possibly OSX".

Do you already have a roadmap to reach the goal?

[giampaolo]

It's something which needs to be investigated. A quick Google search brought me here:
https://github.com/joerick/cibuildwheel
...so it appears it's definitively possible.

[giampaolo]

AFAICT whereas AppVeyor gives you space to store the wheels on their machines Travis won't, so that is the main stopper.

[guettli]

I am subscribed to the distutils mailing list. I asked the experts if what they think: https://mail.python.org/pipermail/distutils-sig/2017-September/031506.html

[agronholm]

Sure, it's doable. You will need to take advantage of Travis's new "Build stages" feature. Do your testing in one build stage and wheel building/deployment in the next. You should be able to build a suitable job matrix with relatively little work. Make sure you use the manylinux1 Docker images to minimize the necessary effort.

[westurner]

From https://www.google.com/search?q=travis+ci+build+manylinux+wheels+upload+to

From https://www.mail-archive.com/distutils-sig@python.org/msg25769.html :

https://github.com/matthew-brett/multibuild

• https://github.com/matthew-brett/multibuild/blob/master/travis_linux_steps.sh
• README.rst mentions WHEELHOUSE:
  • https://pypi.python.org/pypi/Wheelhouse

[westurner]

IIUC, storing packages in a github repo is fine for low-volume things; but artifacts uploaded as /releases are differently CDN'd?

[matthew-brett]

Guys - I'm happy to give you access to the Rackspace CDN for you to upload to. Further upload to pypi can be fairly well automated as well : https://github.com/MacPython/cython-wheels#quickstart

[agronholm]

@giampaolo Why is there need for wheel storage space with Travis? Why not just build them and upload to PyPI directly?

[giampaolo]

If that turns out to be possible then I am all for it. That should happen "on request" when we tag a new version though. I am not interested in having intermediate builds for each commit.

On Sat, 9 Sep 2017 at 19:18, Alex Grönholm ***@***.***> wrote: @giampaolo <https://github.com/giampaolo> Why is there need for wheel storage space with Travis? Why not just build them and upload to PyPI directly? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#824 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAplLPE4d352Bv-j247bm71nlPvPgzBsks5sgnQagaJpZM4IlKUi> .

-- Giampaolo - http://grodola.blogspot.com

[agronholm]

If you're fine with the wheel building happening on every build but the PyPI upload only on tags, then yes, that is totally doable.

[matthew-brett]

The way that we do it over at numpy / scipy / matplotib etc, is have a separate repo for doing the wheel builds, which we trigger at release time. This uploads to Rackspace, allowing us to check that the builds are all good, and have passed, before uploading from Rackspace to PyPI. In practice, that has been a useful extra step to avoid messing up on the PyPI release. Several projects (such as numpy, scipy, matplotlib) also do daily wheel builds on the master branch, from the same repository, to make sure the wheel build is still working.

e.g. https://github.com/MacPython/numpy-wheels

[giampaolo]

What if the build on tag fails because one of test suites fail due to a false positive? Perhaps the wheel build should happen regardless of the test results.

On Sat, 9 Sep 2017 at 19:33, Alex Grönholm ***@***.***> wrote: If you're fine with the wheel building happening on every build but the PyPI upload only on tags, then yes, that is totally doable. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#824 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAplLDYH8sNdvNCE9nyeBvFpzFCPLb18ks5sgnengaJpZM4IlKUi> .

-- Giampaolo - http://grodola.blogspot.com

[agronholm]

I don't understand what you mean by "fail due to a false positive".
@matthew-brett why is uploading to Rackspace necessary to check that the builds are fine? Can that not be done on Travis?

[gaborbernat]

@giampaolo Shouldn't manywheels project solve this, aka building via that should make this work on Linux/Mac too?

[westurner]

manylinux wheels should work on many Linux distros, but IDK about Mac? https://github.com/pypa/manylinux/blob/master/README.rst#example links to an example of building manylinux wheels with Travis CI - https://github.com/pypa/python-manylinux-demo/blob/master/.travis.yml - https://github.com/pypa/python-manylinux-demo/blob/master/travis/build-wheels.sh https://github.com/joerick/cibuildwheel :

* Builds manylinux, macOS and Windows (32 and 64bit) wheels using Travis

CI, Appveyor, and CircleCI

* Bundles shared library dependencies on Linux and macOS through

auditwheel and delocate

* Runs the library test suite against the wheel-installed version of your

library

…

On Friday, May 17, 2019, Bernát Gábor ***@***.***> wrote: @giampaolo <https://github.com/giampaolo> Shouldn't manywheels project solve this, aka building via that should make this work on Linux/Mac too? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#824?email_source=notifications&email_token=AAAMNS3XJOPKJAJVCS4HTFDPVZ43BA5CNFSM4CEUUURKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVUJHNQ#issuecomment-493392822>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAMNS55CTTNNQP5MLRMYQDPVZ43BANCNFSM4CEUUURA> .

[guettli]

For me it would be enough to support linux. I think there are just too many other OSs (solaris, aix, ....).

[gaborbernat]

We can use Azure Pipelines to build all.

[giampaolo]

How?

[matthew-brett]

The setup at https://github.com/MacPython/psutil is nearly complete - no? It builds macOS wheels too.

I think you'll find there's a pretty big demand for macOS wheels.

I personally find Azure pipelines pretty ugly to set up, compared to Travis and Appveyor - and you're nearly done, with https://github.com/MacPython/psutil - see MacPython/psutil#1 - and you'd have to solve those problems on Azure too.

[agronholm]

In my (short) experience with it, Azure Pipelines is far more elegant in design compared to Travis and Appveyor. It is, however, quite "green" as well so they may not have covered all their bases yet (and it has some minor issues too). But if I needed to run my CI on all three platforms, I would definitely use Azure Pipelines (which I have).

[matthew-brett]

I've tried Azure too - but I gave up after a while. Regardless, the barrier here is not the platform - you can use the same machinery on Travis and Azure - the barrier is getting the tests fixed over at MacPython/psutil#1

[giampaolo]

Does azure let you store the wheels somewhere so you can retrieve them via wget or something?

[gaborbernat]

Yeah, pipelines can generate artifacts that are attached to build. You can even define release stages afterwards to upload them to PyPi if the test stage succeeds (perhaps on git tag only/user button push). It's easy to get started and setup given we use tox via https://github.com/tox-dev/azure-pipelines-template (see example https://github.com/tox-dev/tox/blob/master/azure-pipelines.yml - example run https://dev.azure.com/toxdev/tox/_build/results?buildId=1766).

@matthew-brett I would definitely recommend using the tox template to set up if you're using tox. Makes the config a lot simpler/easier.

[giampaolo]

@matthew-brett

The setup at https://github.com/MacPython/psutil is nearly complete - no? It builds macOS wheels too.

If I'm not mistaken the blocker here is #1126 (comment).
Long story short: in order to use manylinux we need an old CentOS version depending on this C routine which cannot rely on CPU_ALLOC macro (only most recent Linux distros have it):

psutil/psutil/_psutil_linux.c

Line 350 in fbece8e

/*

...and that causes a segfault. So in order to proceed we should fix #1126 first.

cPython, which also implements this functionality (os.sched_getaffinity), doesn't have this problem as it relies on CPU_ALLOC:
https://github.com/python/cpython/blob/b49858b4b7b4c9d85ef6946ad020f83e4fa1caa7/Modules/posixmodule.c#L6421

So fixing #1126 is controversial. I guess we may do a runtime check instead of the pre-processor / #ifdef check, but am not sure how (there's a lot of that in _psutil_windows.c but I never did it in _psutil_linux.c).

Also, there are other #ifdef checks in _psutil_linux.c which may not compile on old CentOS, meaning the resulting wheel will have some disabled functionality also on recent Linuxes. Basically this is why this issue (producing wheels) got stuck.

I may be able to try looking at this sometime later during this year. In the meantime, for posterity, any doc/tutorial on how to setup the old CentOS distro (docker?) is appreciated.

[matthew-brett]

@giampaolo - did you try the instructions at MacPython/psutil#1 (comment) ? Did they work? Let me know, I'm happy to extend / update them.

[renefritze]

Long story short: in order to use manylinux we have to use an old CentOS version

The Manylinux2010 platform has just started rolling out a few weeks ago with an updated toolchain. That could potentially make the issue go away without intervention in this codebase, I guess.

[matthew-brett]

Manylinux2010 uses CentOS 6. Does CentOS 6 have the macro?

[nsoranzo]

@giampaolo Would you like help to automate deploy of wheels on PyPI via cibuildwheel ? If yes, with which CI service?

[huntzhan]

I've set up a GitHub workflow to build wheels for Linux/macOS:

• Workflow config: https://github.com/private-pypi/psutil/blob/master/.github/workflows/main.yml
• Example run: https://github.com/private-pypi/psutil/runs/508766809?check_suite_focus=true

Run ls -alh dist/
total 4.3M
drwxr-xr-x 2 runner docker 4.0K Mar 15 06:24 .
drwxr-xr-x 3 runner docker 4.0K Mar 15 06:24 ..
-rw-r--r-- 1 runner docker 222K Mar 15 06:24 psutil-5.7.0-cp27-cp27m-macosx_10_13_x86_64.whl
-rw-r--r-- 1 runner docker 257K Mar 15 06:24 psutil-5.7.0-cp27-cp27m-manylinux2010_i686.whl
-rw-r--r-- 1 runner docker 263K Mar 15 06:24 psutil-5.7.0-cp27-cp27m-manylinux2010_x86_64.whl
-rw-r--r-- 1 runner docker 257K Mar 15 06:24 psutil-5.7.0-cp27-cp27mu-manylinux2010_i686.whl
-rw-r--r-- 1 runner docker 263K Mar 15 06:24 psutil-5.7.0-cp27-cp27mu-manylinux2010_x86_64.whl
-rw-r--r-- 1 runner docker 222K Mar 15 06:24 psutil-5.7.0-cp35-cp35m-macosx_10_13_x86_64.whl
-rw-r--r-- 1 runner docker 261K Mar 15 06:24 psutil-5.7.0-cp35-cp35m-manylinux2010_i686.whl
-rw-r--r-- 1 runner docker 269K Mar 15 06:24 psutil-5.7.0-cp35-cp35m-manylinux2010_x86_64.whl
-rw-r--r-- 1 runner docker 222K Mar 15 06:24 psutil-5.7.0-cp36-cp36m-macosx_10_13_x86_64.whl
-rw-r--r-- 1 runner docker 261K Mar 15 06:24 psutil-5.7.0-cp36-cp36m-manylinux2010_i686.whl
-rw-r--r-- 1 runner docker 270K Mar 15 06:24 psutil-5.7.0-cp36-cp36m-manylinux2010_x86_64.whl
-rw-r--r-- 1 runner docker 222K Mar 15 06:24 psutil-5.7.0-cp37-cp37m-macosx_10_13_x86_64.whl
-rw-r--r-- 1 runner docker 266K Mar 15 06:24 psutil-5.7.0-cp37-cp37m-manylinux2010_i686.whl
-rw-r--r-- 1 runner docker 273K Mar 15 06:24 psutil-5.7.0-cp37-cp37m-manylinux2010_x86_64.whl
-rw-r--r-- 1 runner docker 222K Mar 15 06:24 psutil-5.7.0-cp38-cp38-macosx_10_13_x86_64.whl
-rw-r--r-- 1 runner docker 265K Mar 15 06:24 psutil-5.7.0-cp38-cp38-manylinux2010_i686.whl
-rw-r--r-- 1 runner docker 275K Mar 15 06:24 psutil-5.7.0-cp38-cp38-manylinux2010_x86_64.whl

[rostrovsky]

@huntzhan It would be awesome if you could upload those wheels to https://github.com/private-pypi/psutil/releases
Cheers!

[huntzhan]

@rostrovsky
You can find the wheels in https://pywharf.github.io/pywharf-pkg-repo/psutil/

[vitaly-krugl]

It would be awesome to have an official manylinux wheel for psutil on PyPi. The environment where my app runs abstracts building of the docker image and doesn't allow the app to participate in the docker image build step; the only customization to the docker build that my app is allowed to do is to provide a requirements.txt file for python packages, and the apps are not allowed to apt-get anything at all.

[giampaolo]

Finally, macOS wheels are available, see #1550 (comment).
Closing this out.