Bug fix for py-cython with python 3.10+ (installs in subdir local)

[climbfuji]

Description

Bug fix for py-cython when building with python@3.10:: With that version of python, the py-cython installation is put in subdirectory local of the usual install prefix on certain operating systems, therefore need to adjust the binary/library paths. Note that for some reason, PYTHONPATH is automatically set to the correct subdirectory {self.prefix}/local/lib/pythonX.Y/site-packages.

The py-cython bug fixes are required for installing py-shapely@1.8.x - without the bug fixes, the installation of py-shapely fails because the cython executable is not found.

Also: bug fix in py-shapely, the patch created for version 1.8.0 doesn't work for 1.8.2+ (it works for 1.8.1). We have pinned py-shapely to 1.8.0 as per JEDI requirements for now, therefore this is good enough.

Fixes #187

[climbfuji]

@travissluka I assume you didn't have this problem because you had a cython somewhere in your PATH?

[adamjstewart]

For the <prefix> vs. <prefix>/local discussion, see:

• Fix the install scheme to posix_prefix when --prefix is passed pypa/pip#10978
• Fix installations with system Python on Ubuntu/Fedora spack/spack#31939
• If using an external Python, the bin directory might not be in PATH spack/spack#32662
• Installation issue: py-docutils (symlink error) spack/spack#32769

I would not recommend the solution in this PR. It will work on some systems when using the system Python but break on most other systems or if using Spack-installed Python.

Meta comment: is there a reason for introducing hacks in this fork instead of simply using upstream? If there are specific packages you would like to maintain but don't want to put upstream, you could just maintain a package repo instead of maintaining an entire fork of Spack.

[climbfuji]

For the <prefix> vs. <prefix>/local discussion, see:

• Fix the install scheme to posix_prefix when --prefix is passed pypa/pip#10978
• Fix installations with system Python on Ubuntu/Fedora spack/spack#31939
• bin directory of python packages no longer in PATH spack/spack#32662
• Installation issue: py-docutils (symlink error) spack/spack#32769

I would not recommend the solution in this PR. It will work on some systems when using the system Python but break on most other systems or if using Spack-installed Python.

Meta comment: is there a reason for introducing hacks in this fork instead of simply using upstream? If there are specific packages you would like to maintain but don't want to put upstream, you could just maintain a package repo instead of maintaining an entire fork of Spack.

@adamjstewart This is very useful feedback, thanks. The problem is being able to follow the light-speed changes in the authoritative spack repo. If I was able to digest the 500-ish github notifications from spack every day (on top of all others), I would, but ...

Our fork is also a bit behind, will have to see if I can pull in/cherry-pick those updates or if I have to go all the way to update to spack develop. We wanted to try to stick with releases of spack and selectively pull in bug fixes as needed.

[climbfuji]

I guess a minimum-effort compromise would be to test in the override prefix property if bin exists, and if it doesn't if local/bin exists (somewhat similar to what was done in spack#31939). If so, apply the patch.

[adamjstewart]

Yep, I also don't try to digest those notifications, I just fix things when they break or complain when I don't know how to fix them. If you have any changes in your fork different from upstream, it would be great to see those upstreamed so others can benefit from them.

[climbfuji]

Yep, I also don't try to digest those notifications, I just fix things when they break or complain when I don't know how to fix them. If you have any changes in your fork different from upstream, it would be great to see those upstreamed so others can benefit from them.

Absolutely, I am trying to do this as much as my time allows. One package at a time. We have a separate "repo" (=directory under var/spack/repos/) for stuff that does not yet exist in the official spack fork. Some of it we cannot or cannot yet push to the official repo (due to access/licensing restrictions, for example). All others will be sent over one by one.

One problem is that several of them take ages. See for example spack#32857 and spack#32478.

[adamjstewart]

One problem is that several of them take ages

I usually remind people to review once per week. There's also a #pull-requests channel on Slack where you can ask for reviews.

[travissluka]

@travissluka I assume you didn't have this problem because you had a cython somewhere in your PATH?

@climbfuji I disabled all packages that I didn't need to compile/run SOCA, as part of giving up on the hdf4 problems, so I never compiled py-shapely

[adamjstewart]

This won't help if you're on an OS that uses lib64 instead of lib.

[climbfuji]

You are right, added an additional elif clause to cover this. Don't like any of these solutions, but the solutions you mentioned above aren't available in the authoritative spack repo yet.

[adamjstewart]

Note that you'll also need to do what you're doing here for every Python package in Spack...

[climbfuji]

I know ... my strategy is to have this fixed temporarily (because it's the only package in our fairly comprehensive software stack that has this problem) and then get the clean solution from spack once that is available.

[climbfuji]

@AlexanderRichert-NOAA This PR is ready, too. If you read through the conversation between Adam and I, you'll see that this is supposed to be a workaround until the problem is addressed on a spack level in the authoritative repository.