Looking for recommended structure for separate C++ library and extension builds

[glennsweeney]

Hi,

I love meson, and I'm excited to start using it with Python!

I am trying to adopt mesonpy instead of my deprecated and janky setup.py build tools, and I'm looking for advice on how to structure my repository and build system.

Today, I have:

<root>
├── cpp
│   ├── include
│   │   └── *.hpp
│   ├── meson.build
│   ├── pybind
│   │   └── *.cpp
│   ├── src
│   │   └── *.cpp
│   └── subprojects
│       └── ...
├── python
│   ├── meson.build (new!)
│   ├── <package-name>
│   │   ├── __init__.py
│   │   ├── *.py
│   ├── pyproject.toml (new!)
└── README.md

root/cpp/meson.build contains standard Meson to build a C++ static library, and a few wrapper python extension modules (via pybind) around that library.

I'd like to preserve the ability to build a C++ library with Meson independent of a Python installation, but it'd also be awesome to be able to trigger a C++ build during Python installation to avoid tracking a separate system dependency. I did this previously in setup.py by invoking Meson directly and passing in an option giving a target install directory, and installing the library and extension modules into the Python package tree.

It'd be even cooler to move the pybind specific logic into the Python build.

In short; any suggestions on how to make this work in mesonpy?

[dnicolodi]

First, please note that the current organization of your repository is not compatible with distributing it as a source distribution using meson dist nor as a Python package source distribution created by meson-python. The first requires the root folder of the project to contain a meson.build file describing the project. The latter also requires the pyproject.toml file to be in the root folder of the project.

I'd like to preserve the ability to build a C++ library with Meson independent of a Python installation, but it'd also be awesome to be able to trigger a C++ build during Python installation to avoid tracking a separate system dependency

There isn't only one way in which you can achieve this. Which one is more suited to your project depends on the level of integration you want for the C++ and Python parts of the project.

If the C++ library is thought to also be used independently of the Python wrappers, what makes the most sense is probably to have the C++ library be an independent project from the Python wrappers. You can then use the C++ project as a subproject of the Python wrappers project. In this case I would move the pybind wrappers into the Python project. For a straightforward example of this, see https://github.com/dnicolodi/python-siphash24/ where I use Cython instead of pybind to create the wrappers, but the concept is the same.

If you want to keep the sources of the C++ and Python parts tightly coupled, you can keep the current project organization, but you need to have a meson.build and the pyproject.toml file in the project root. You can then make building and installing the Python wrapper conditional to passing an option to meson setup or automatic if Meson finds the Python dependency.

With a meson.options that looks like this:

option('python', type: 'feature', value: 'auto')

or

option('python', type: 'feature', value: 'disabled')

if you don't want to build the Python wrappers by default, a top level meson.build that looks something like this should be enough:

project(...)
subdir('cpp')
python = import('python').find_installation(required: get_option('python'))
if python.found()
  subdir('python')
endif

Then you can add

[tool.meson-python.args]
setup = ['-Dpython=enabled']

to pyproject.toml to ensure that the python dependency is found when building the Python wheel (although this should not be required because the Python dependency should always be found when the build is triggered by meson-python). These code snippets are strictly untested and typed from memory, thus I do not guarantee that they work as is.

[glennsweeney]

Thank you @dnicolodi for the direct, detailed, and quick answer. This is extremely helpful context. I'll work on following your proposed "tightly coupled" option.

I realize it may be obvious to many, but would you be open to a small PR adding information about the intentions of root-level meson.build and pyproject.toml files to documentation? I could imagine either adding this as a sentence in "how-to-guides/first-project", or as a new "reference/project-structure" page. Would either of these be helpful in your opinion?

[dnicolodi]

The documentation here implicitly does not say it very explicitly but hints to the fact that there must be a meson.build and pyproject.toml in the root directory of the project:

To enable pip or build to build a Python source distribution (sdist) or a binary Python package (wheel) for a Meson project, it is sufficient to add to the root of the source tree next to the top-level meson.build a pyproject.toml file specifying meson-python as the Python build backend

The tutorial is probably the place where this should be made more explicit. There are many more things that I would like to cover in the tutorial, but not much time to work on it. PR improving the documentation are welcome.