Installing extension into project-specific directory

[voidtrance]

I searched for this in other discussions but could not find an answer:

I am writing an application that includes a C extension and a shared library internal to the project. I would like to install those in a project-specific directory: {python.platlibdir}/myproject/lib/project_shared_lib.so. To do this, I've added install_dir: get_option('python.platlibdir') / 'myproject/lib' to the library() call to build the project-internal shared library.

However, when I run the editable build with rm -rf build && python -m pip install --no-build-isolation --editable . --config-settings=setup-args=-Dbuildtype=debug I get the following error:

...

      meson-python: error: Could not map installation path to an equivalent wheel directory: '{python.platlibdir}/myproject/lib/libproject_shared_lib.so'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

First, I am not sure what it actually means and, second, what is the correct way to specify custom install paths?

[dnicolodi]

You should use

python = import('python').find_installation()
library(...
  install_dir = pythin.get_install_dir(pure: false) / 'myproject/lib'
)

However, I'm not sure doing this results in the RPATH for the Python extension module that depends on this library to be set correctly.

[voidtrance]

Hmm, so while this works for building wheels, when I try do do an editable build, the libraries don't end up in a matching directory tree. As a result, I can't use the editable build because the modules are not found.

This is what I have so far:
source tree

.
├── src
│   ├── control
│   │   └── core.c
│   └── lib
│       └── mylib.c

libmylib.so is built with install_dir: py.get_install_dir(pure: false) / 'myproject/lib' and core-*.so is installed with install_dir: py.get_install_dir(pure: false) / 'myproject/core'.

The content of the wheel seems reasonable. However, when I run python -m pip -v install --no-build-isolation --editable ., I get the following tree:

.
├── build
│   └── cp311
│       ├── build.ninja
│       ├── compile_commands.json
│       ├── meson-info
│       │   └── ...
│       ├── meson-logs
│       │   └── meson-log.txt
│       ├── meson-private
│       │   └── ...
│       ├── meson-python-native-file.ini
│       └── src
│           ├── control
│           │   ├── core.cpython-311-x86_64-linux-gnu.so
│           │   └── core.cpython-311-x86_64-linux-gnu.so.p
│           │       └── ...
│           └── lib
│               ├── libmylib.so
│               └── libmylib.so.p
│                   └── ...

So, when attempting to run from the editable build, I would have to import with import control.core instead of import myproject.core.

I know that I am missing some understanding but this seems pretty confusing to me.

[eli-schwartz]

Meson will use a build directory structure mirroring the source code (and which meson.build files the targets are defined in).

The meson-python editable install uses an import hook that ensures "myproject.core" reads the right core.cpython-311-x86_64-linux-gnu.so from disk regardless of directory structure.

If that extension has a link_with to libmylib.so then meson will set up a unix rpath for the build directory and it will "just work" during the editable install phase. The install_rpath discussion is about what you need to have a working wheel.

[voidtrance]

I am sorry, I replied to the wrong message.

However, the hook is not doing what you describe. I see the hook installed in ~/.local/lib/python3.11/site-packages. However, when I try to import myproject.core from the root directory of my project repository, I get

>>> import myproject
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'myproject'

[voidtrance]

Oh, I just noticed something - The MesonMetaFinder() is not present in sys.meta_path. I have no idea why. Even after removing ~/.local/lib/python3.11/site-packages/* and having pip remake the files, it still is not present.

[dnicolodi]

I suspect you are installing the editable wheel into a Python environment and testing from another. Unless you provide a way to reproduce exactly what you are doing, from the creation of the Python venv to the command that fails, possibly including the source code of a minimal package that reproduces the problem, it is impossible to tell what you are doing wrong. Support for editable wheels is tested and used in many projects by many developers, it is unlikely that there is such a fundamental bug in it.