Regression: Cantera install into /usr/local/, not /usr/ on Fedora Linux #1233

mefuller · 2022-04-02T09:56:23Z

Problem description
Building current Cantera 2.6.0b1 (commit 6102cd5) with scons build and install prefix set to /usr/, packages are actually installed as follows:

/usr/local/bin/ck2cti (...)
/usr/local/lib64/python3.10/site-packages/Cantera-2.6.0b1.dist-info/INSTALLER (...)
/usr/local/lib64/python3.10/site-packages/cantera/init.pxd (...)

Output from install indicates that they should be installed where I do want them, i.e.,


    Cantera has been successfully installed.

    File locations:
      library files               /usr/lib64
      C++ headers                 /usr/include
      samples                     /usr/share/cantera/samples
      data files                  /usr/share/cantera/data
      input file converters       /usr/bin
      Python package              /usr/lib64/python3.10/site-packages
      Python examples             /usr/lib64/python3.10/site-packages/cantera/examples

Steps to reproduce
Exact reproduction of #1149

Behavior
Additional issue from #1149 is that Python files are now installed under /usr/local/lib64..., not /usr/lib64...

System information

Cantera version: 2.6.0b1
OS: Fedora Linux 35, Rawhide
Python/MATLAB/other software versions: Python 3.10

Additional context
See build log: https://download.copr.fedorainfracloud.org/results/fuller/cantera-test/fedora-rawhide-x86_64/03938964-cantera/builder-live.log.gz

#1149

The text was updated successfully, but these errors were encountered:

bryanwweber · 2022-04-02T12:47:50Z

Thanks @mefuller! So the install message shows the right place but the files don't end up in that place? Can you figure out what flags pip requires to put things in that folder?

mefuller · 2022-04-02T14:41:50Z

@bryanwweber your understanding is correct - the terminal output shows the desired directories, but the actual location is wrong.
I will try to find some time to dig into this later.
Thanks

bryanwweber · 2022-04-02T16:23:11Z

That's very strange because pip is used to find the installation location that's printed... The line in the log you linked looks correct too. Maybe I'm misunderstanding the Setuptools/pip options.

speth · 2022-04-03T22:31:29Z

I can't seem to replicate this on Fedora 35. I ran:

scons build python_package=y prefix=/usr python_prefix=/usr stage_dir=STAGE

and ended up with the Python module installed in STAGE/usr/lib64/python3.10/site-packages and the converter scripts installed in STAGE/usr/bin.

Similarly, if I do a non-staged install, the Python package is installed to /usr/lib64/python3.10/site-packages and the scripts are installed to /usr/bin, as described in the post-install message.

bryanwweber · 2022-04-04T11:01:44Z

@speth were you using the latest main?

mefuller · 2022-04-04T12:12:23Z

The problem with ck2cti et al. is present in F36 and Rawhide (future F37); the Python directory appears to be incorrect on all three (F35, F36, Rawhide).

My command is scons build prefix=/usr libdirname=lib64 system_sundials=y f90_interface=y renamed_shared_libraries=n python_package=full system_eigen=y extra_inc_dirs=/usr/include/eigen3 system_fmt=y, so perhaps I need to add python_prefix=/usr (but not stage_dir=STAGE per Ray's test?)

I haven't changed any options in my build specs, so I'll check Ray's command for F35, but something definitely changed:

Logs:
F35: https://download.copr.fedorainfracloud.org/results/fuller/cantera-test/fedora-35-x86_64/03938964-cantera/builder-live.log.gz
F36: https://download.copr.fedorainfracloud.org/results/fuller/cantera-test/fedora-36-x86_64/03938964-cantera/builder-live.log.gz
Rawhide: https://download.copr.fedorainfracloud.org/results/fuller/cantera-test/fedora-rawhide-x86_64/03938964-cantera/builder-live.log.gz

bryanwweber · 2022-04-04T13:50:43Z

@mefuller Same question to you, are you using the tagged 2.6.0b1 commit or the latest commit on main?

mefuller · 2022-04-04T13:56:03Z

I had grabbed the latest commit - I'll retest tonight or tomorrow

bryanwweber · 2022-04-04T14:35:39Z

~~I don't see b0be80346808a0efa085a4858f05d694bd721e56 anywhere in our commit history, are there patches you're applying?~~ This hash is from the Copr repo that has the build instructions for Fedora. https://copr-dist-git.fedorainfracloud.org/cgit/fuller/cantera-test/cantera.git

speth · 2022-04-04T14:39:08Z

@speth were you using the latest main?

Yes, of course. I was using a fresh clone of the repo into a Docker container, of main at b4616ef.

I tried again using @mefuller's exact build command, but am still seeing all the files end up in the right place. The logs from pip install are almost useless. Even if you ask for the maximum level of verbosity by adding -vvv to the command, it still doesn't print the path that it's installing files to. But in any case, what I get scons build and scons install can be found here: https://gist.github.com/speth/ee53b78a8e4fa3f5f384bf2129d931e4.

speth · 2022-04-04T15:49:24Z

One thing I did notice that seems incorrect here (at least, if I understand the Fedora filesystem correctly) is that the Python module is being installed (mostly) to /usr/lib/python3.10/site-packages/cantera while only the directory Cantera-2.6.0b1.dist-info is being installed to /usr/lib64/python3.10/site-packages. I think that any package that contains compiled components ought to be in /usr/lib64.

bryanwweber · 2022-04-04T17:36:43Z

Recording this here before I forget. I got (what I think is) the rpmbuild process running with Docker via:

docker run --cap-add=SYS_ADMIN -it --rm quay.io/fedora/fedora:35 /bin/bash
dnf install mock git copr-distgit-client
mkdir -p /var/lib/copr-rpmbuild/workspace/
mkdir -p /var/lib/copr-rpmbuild/results/configs
curl -o configs.tar.gz https://download.copr.fedorainfracloud.org/results/fuller/cantera-test/fedora-35-x86_64/03938964-cantera/configs.tar.gz
tar -zxf configs.tar.gz
cp configs/child.cfg /var/lib/copr-rpmbuild/results/configs/
rm -rf configs configs.tar.gz
git clone https://copr-dist-git.fedorainfracloud.org/git/fuller/cantera-test/cantera.git /var/lib/copr-rpmbuild/workspace/workdir-gg66e7eb/cantera --depth 500 --no-single-branch
git checkout b0be80346808a0efa085a4858f05d694bd721e56  # Fedora 35
cd /var/lib/copr-rpmbuild/workspace/workdir-gg66e7eb
copr-distgit-client sources
mock --buildsrpm --spec /var/lib/copr-rpmbuild/workspace/workdir-gg66e7eb/cantera/cantera.spec --sources /var/lib/copr-rpmbuild/workspace/workdir-gg66e7eb/cantera --resultdir /var/lib/copr-rpmbuild/results --uniqueext 1648892130.396279 -r /var/lib/copr-rpmbuild/results/configs/child.cfg
mock --rebuild /var/lib/copr-rpmbuild/results/cantera-2.6.0-1.fc35.src.rpm --resultdir /var/lib/copr-rpmbuild/results --uniqueext 1648892130.396279 -r /var/lib/copr-rpmbuild/results/configs/child.cfg

The important thing in the Docker line is the --cap-add=SYS_ADMIN to make sure that mock works properly.

The files after the build are located in /var/lib/mock/fedora-35-aarch64-bootstrap-<uniqueext>/root/builddir (aarch64 locally for me on my M1). JK this folder seems to be empty. It may need the -N (no cleanup) flag to the second mock command, trying that now...

mefuller · 2022-04-04T19:45:47Z

One thing I did notice that seems incorrect here (at least, if I understand the Fedora filesystem correctly) is that the Python module is being installed (mostly) to /usr/lib/python3.10/site-packages/cantera while only the directory Cantera-2.6.0b1.dist-info is being installed to /usr/lib64/python3.10/site-packages. I think that any package that contains compiled components ought to be in /usr/lib64.

Yes, this is an issue across versions (F35, F36, Rawhide). We see the directory mismatch in all three versions' buildlogs (and I just rebuilt x86_64 with the python_prefix specified and also with the latest commit, b4616ef):

...
    Cantera has been successfully installed.

    File locations:
      library files               /usr/lib64
      C++ headers                 /usr/include
      samples                     /usr/share/cantera/samples
      data files                  /usr/share/cantera/data
      input file converters       /usr/bin
      Python package              /usr/lib64/python3.10/site-packages
      Python examples             /usr/lib64/python3.10/site-packages/cantera/examples

scons: done building targets.
+ /usr/bin/find-debuginfo -j2 --strict-build-id -m -i --build-id-seed 2.6.0-1.fc35 --unique-debug-suffix -2.6.0-1.fc35.x86_64 --unique-debug-src-base cantera-2.6.0-1.fc35.x86_64 --run-dwz --dwz-low-mem-die-limit 10000000 --dwz-max-die-limit 110000000 -S debugsourcefiles.list /builddir/build/BUILD/cantera-main
extracting debug info from /builddir/build/BUILDROOT/cantera-2.6.0-1.fc35.x86_64/usr/lib/python3.10/site-packages/cantera/_cantera.cpython-310-x86_64-linux-gnu.so
...

To be extremely clear about the issue, the scripts are correctly installed on F35, as @speth observed: his builds in a F35 Docker container show the same results as mine (https://download.copr.fedorainfracloud.org/results/fuller/cantera-test/fedora-35-x86_64/03954713-cantera/builder-live.log.gz).

The regression to installing the scripts in /usr/local/bin and not /usr/bin is present on F36 and Rawhide, which has not been independently confirmed (https://download.copr.fedorainfracloud.org/results/fuller/cantera-test/fedora-36-x86_64/03954713-cantera/builder-live.log.gz, https://download.copr.fedorainfracloud.org/results/fuller/cantera-test/fedora-rawhide-x86_64/03954713-cantera/builder-live.log.gz)

bryanwweber · 2022-04-04T19:56:02Z

This is definitely pip's fault on F35 (let's solve one thing at a time 😄). According to its own internal APIs, it puts files in lib64 but then it doesn't actually put files in lib64. I'll dig into what's going on there, I have a reproducible rpmbuild environment set up.

Edit to add: pip is deciding to place files in the purelib location, rather than the platlib location. We'll need to track down why it's doing that. I wonder if it's because of how we're telling Setuptools that there's a C extension?

speth · 2022-04-04T21:32:00Z

I was able to replicate the described behavior on F37. The build commands:

scons build prefix=/usr libdirname=lib64 system_sundials=y f90_interface=y renamed_shared_libraries=n python_package=full system_eigen=y extra_inc_dirs=/usr/include/eigen3 system_fmt=y stage_dir=STAGE
scons install

eventually result in this reasonable-looking call:

"/usr/bin/python3" -m pip install --prefix=/usr --root=/cantera/STAGE --no-build-isolation --no-deps -v --force-reinstall build/python

But then both the scripts and the Python module itself end up installed in STAGE/usr/local. This definitely seems like pip's fault, but I don't know what we're supposed to be able to do about it. I guess on the plus side, it is putting the Python module in the lib64 directory, so that's something.

bryanwweber · 2022-04-04T21:34:02Z

If I manually run

/var/lib/mock/fedora-35-aarch64-1648892130.396279/root/usr/bin/python3 -m pip install --prefix='/usr' --root=../../BUILDROOT/cantera-2.6.0-1.fc35.aarch64/ . --ignore-installed --use-feature=in-tree-build --no-deps

from the built source directory, things are installed to the right place on F35.

@speth I wonder if Fedora are patching pip. Can you use pip not installed from dnf?

speth · 2022-04-04T23:46:14Z

On F37, using pip installed using python3 -m ensurepip --upgrade, I don't see any difference -- everything is still ending up in /usr/local. I'll try F35 again later. Should I also be trying a stock version of setuptools and any other packages that are involved in this (wheel? I'm not sure what else could affect this).

bryanwweber · 2022-04-05T01:35:41Z

Can you try installing the wheel that gets built by scons build instead of having SCons run pip install python/build, just replace python/build with the name of the wheel file. I wonder if pip treats things differently if it's a folder vs. a wheel file. That should take any other dependencies out of the equation.

speth · 2022-04-05T14:05:04Z

Running the pip install command directly on F37 reveals some unexpected behavior.

pip install --prefix=/usr build/dist/Cantera....whl installs to /usr/local/lib64/python3.10/site-packages/
pip install --prefix=/usr/local build/dist/Cantera....whl installs to /usr/local/local/lib64/python3.10/site-packages/
pip install --prefix=/bakedbeans build/dist/Cantera....whl installs to /bakedbeans/local/lib64/python3.10/site-packages/

Furthermore, this behavior is not particular to our package. The same thing happens installing packages like numpy and scipy, both from downloaded whl files and just directly from PyPI. It also directly contradicts the pip documentation, which describes the --prefix option as "Installation prefix where lib, bin and other top-level folders are placed".

bryanwweber · 2022-04-05T14:41:29Z

😱 This seems to be pretty clearly a bug in pip then... At least it's going in lib64? 😭

bryanwweber · 2022-04-05T23:08:33Z

Since this is a bug in pip, I think we have 2 options for the short term:

Ignore this, let pip fix it, then package for Fedora. Users can install with pip from PyPI or with conda.
Hack a patch (either upstream in Cantera/cantera or downstream in the Fedora spec) that reverts to the old setup.py install --platlib=... approach.

I have a preference for Option 1 because it's less work right now, and releasing 2.6 has already been significantly delayed by packaging issues...

decaluwe · 2022-04-06T00:36:28Z

The details of this are way out of my league, but philosophically I agree with Option 1.

ischoegl · 2022-04-06T02:27:41Z

I likewise agree that option 1 is preferable.

speth · 2022-04-06T02:31:06Z

I'd like to suggest a third option, which is to leave this as is, but add a couple lines to the Fedora packaging script to move the files to the correct location after they've been "installed" to the incorrect staging directory by pip.

bryanwweber · 2022-04-06T14:05:55Z

add a couple lines to the Fedora packaging script

That would also work, provided that Fedora allows that in their official packages. @mefuller can you weigh in here?

mefuller · 2022-04-06T14:16:51Z

It should be possible.
I'll look at it over the weekend.

mefuller · 2022-04-11T09:00:29Z

Just as an update, I can move the scripts to /usr/bin/ from /usr/local/bin/, but the Python installation is throwing "permission denied" errors when I try to move it

mefuller · 2022-04-12T20:05:40Z

ok, 2.6.0b2 is running and will be pushed to the repos - I made the required hacks

Not closing the issue since the install location is still wrong, but I leave it up to "upstream" to decide how to proceed

bryanwweber · 2022-04-13T13:22:51Z

Thanks @mefuller! Per pypa/pip#10978, can you check with Fedora upstream about how they've patched sysconfig/distutils/pip to handle this situation?

speth · 2022-04-13T17:43:48Z

I dug a bit further into this, and it seems like this is a result of RedHat/Fedora patching the builtin sysconfig module to return such paths. You can see the local subdirectory being injected when running commands like

>>> sysconfig.get_paths('posix_prefix', {'platbase': 'test'})['platlib']
'test/local/lib64/python3.10/site-packages'

which is a variant of what pip does to determine the installation location for these files. One thing I noticed is that the Fedora sysconfig module includes an additional "install scheme" (the first argument to get_paths), which doesn't include the local directory:

>>> sysconfig.get_paths('rpm_prefix', {'platbase': 'test'})['platlib']
'test/lib64/python3.10/site-packages'

However, I don't know how to specify the use of this scheme to pip. Is there somewhere we can see the scripts for packaging other Python modules for Fedora? I'm less familiar with this packaging system, so I don't know where those would be, but it might help to see any that use pip to prepare their packages.

bryanwweber · 2022-04-13T20:31:41Z

@speth I don't see local in your first example, was that copy-pasta?

speth · 2022-04-13T21:30:41Z

Yes, of course.

mefuller · 2022-04-15T10:18:18Z

Thanks @mefuller! Per pypa/pip#10978, can you check with Fedora upstream about how they've patched sysconfig/distutils/pip to handle this situation?

I will try to run down this today.

The punchline on the builds from v2.6.0b2 is that the rewrite to use wheels introduced some other errors in installation location in current versions, but it appears that two out of four total kludges will age out in a year, so I wouldn't increase the scope of the problem beyond what was originally described for Fedora Rawhide.

For those of you who are scoring at home, by the time I finished hacking in a solution, we got this:

###kludges for https://github.com/Cantera/cantera/issues/1233

# incorrect installation to /usr/local/bin on F36+
	
%if 0%{?fedora} >= 36
	
mv %{buildroot}%{_prefix}/local/bin/* %{buildroot}%{_bindir}/
	
rm -rf %{buildroot}%{_prefix}/local/bin
	
%endif
	
 
	
# incorrect installation to /usr/lib/ on 64 bit systems on F36-
	
%if 0%{?fedora} <= 36
	
if [[ -d %{buildroot}%{_prefix}/lib/python%{python3_version}/site-packages ]] && [ %{_lib} == "lib64" ]; then
	
  mkdir -p %{buildroot}%{python3_sitearch}/
	
  mv %{buildroot}%{_prefix}/lib/python%{python3_version}/site-packages/* %{buildroot}%{python3_sitearch}/
	
fi
	
if [[ -d %{buildroot}%{_prefix}/local/lib/python%{python3_version}/site-packages ]] && [ %{_lib} == "lib64" ]; then
	
  mkdir -p %{buildroot}%{python3_sitearch}/
	
  mv %{buildroot}%{_prefix}/local/lib/python%{python3_version}/site-packages/* %{buildroot}%{python3_sitearch}/
	
fi
	
%endif
	
 
	
# incorrect installation to /usr/local/lib* on F36+
	
%if 0%{?fedora} >= 36
	
mkdir -p %{buildroot}%{python3_sitearch}/
	
mv %{buildroot}%{_prefix}/local/%{_lib}/python%{python3_version}/site-packages/* %{buildroot}%{python3_sitearch}/
	
rm -rf %{buildroot}%{_prefix}/local/
	
%endif
	
###end_kludge

mefuller · 2022-04-15T11:51:57Z

Thanks @mefuller! Per pypa/pip#10978, can you check with Fedora upstream about how they've patched sysconfig/distutils/pip to handle this situation?

ok, so looking at the pip and distutils spec files (
python-distutils-extra.spec.txt
python-pip.spec.txt
), the Fedora Python build/install macros appear to be doing all the heavy listing

Those macros are defined in https://src.fedoraproject.org/rpms/python-rpm-macros/blob/rawhide/f/macros.python3

Not being so familiar with the details, I'll just leave that here for now, but perhaps it can offer some useful information

speth · 2022-04-16T20:55:34Z

I think the end result here is that there's nothing we can/should change in our build process.

mefuller mentioned this issue Apr 2, 2022

Install procedure tries to uninstall Cantera-python version installed by system pkg manager #1230

Closed

speth mentioned this issue Apr 5, 2022

Fix the install scheme to posix_prefix when --prefix is passed pypa/pip#10978

Open

1 task

speth closed this as completed Apr 16, 2022

Regression: Cantera install into /usr/local/*, not /usr/* on Fedora Linux #1233

Regression: Cantera install into /usr/local/*, not /usr/* on Fedora Linux #1233

Comments

mefuller commented Apr 2, 2022 • edited Loading

bryanwweber commented Apr 2, 2022

mefuller commented Apr 2, 2022

bryanwweber commented Apr 2, 2022

speth commented Apr 3, 2022 • edited Loading

bryanwweber commented Apr 4, 2022

mefuller commented Apr 4, 2022

bryanwweber commented Apr 4, 2022

mefuller commented Apr 4, 2022

bryanwweber commented Apr 4, 2022 • edited Loading

speth commented Apr 4, 2022

speth commented Apr 4, 2022

bryanwweber commented Apr 4, 2022 • edited Loading

mefuller commented Apr 4, 2022 • edited by bryanwweber Loading

bryanwweber commented Apr 4, 2022 • edited Loading

speth commented Apr 4, 2022 • edited Loading

bryanwweber commented Apr 4, 2022

speth commented Apr 4, 2022

bryanwweber commented Apr 5, 2022 • edited Loading

speth commented Apr 5, 2022

bryanwweber commented Apr 5, 2022

bryanwweber commented Apr 5, 2022

decaluwe commented Apr 6, 2022

ischoegl commented Apr 6, 2022

speth commented Apr 6, 2022

bryanwweber commented Apr 6, 2022

mefuller commented Apr 6, 2022

mefuller commented Apr 11, 2022

mefuller commented Apr 12, 2022

bryanwweber commented Apr 13, 2022

speth commented Apr 13, 2022 • edited Loading

bryanwweber commented Apr 13, 2022

speth commented Apr 13, 2022

mefuller commented Apr 15, 2022

mefuller commented Apr 15, 2022

speth commented Apr 16, 2022

Regression: Cantera install into /usr/local/, not /usr/ on Fedora Linux #1233

Regression: Cantera install into /usr/local/, not /usr/ on Fedora Linux #1233

mefuller commented Apr 2, 2022 •

edited

Loading

speth commented Apr 3, 2022 •

edited

Loading

bryanwweber commented Apr 4, 2022 •

edited

Loading

bryanwweber commented Apr 4, 2022 •

edited

Loading

mefuller commented Apr 4, 2022 •

edited by bryanwweber

Loading

bryanwweber commented Apr 4, 2022 •

edited

Loading

speth commented Apr 4, 2022 •

edited

Loading

bryanwweber commented Apr 5, 2022 •

edited

Loading

speth commented Apr 13, 2022 •

edited

Loading