Build sdist without requiring all build dependencies

[oscarbenjamin]

Maybe I missed how to do this but is there a way to build an sdist without requiring all of the build dependencies to be present?

I would have thought that it should be possible without going through a full configure since it should be the consumer of the sdist who would do the configure when trying to build from it.

Here is what currently happens with python-flint:

$ python -m build --sdist
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - cython
  - meson-python
* Getting build dependencies for sdist...
* Building sdist...
+ meson setup /home/oscar/current/active/python-flint /home/oscar/current/active/python-flint/.mesonpy-9e_1lo1i -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/home/oscar/current/active/python-flint/.mesonpy-9e_1lo1i/meson-python-native-file.ini
The Meson build system
Version: 1.4.1
Source dir: /home/oscar/current/active/python-flint
Build dir: /home/oscar/current/active/python-flint/.mesonpy-9e_1lo1i
Build type: native build
Project name: python-flint
Project version: undefined
C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
C linker for the host machine: cc ld.bfd 2.38
Cython compiler for the host machine: cython (cython 3.0.10)
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program python found: YES (/tmp/build-env-a21dkq3d/bin/python)
Found pkg-config: YES (/usr/bin/pkg-config) 0.29.2
Run-time dependency python found: YES 3.11
Run-time dependency gmp found: YES 6.2.1
Run-time dependency mpfr found: YES 4.1.0
Found CMake: /home/oscar/.pyenv/shims/cmake (3.22.1)
WARNING: CMake Toolchain: Failed to determine CMake compilers state
Run-time dependency flint found: NO (tried pkgconfig and cmake)

../meson.build:10:12: ERROR: Dependency "flint" not found, tried pkgconfig and cmake

A full log can be found at /home/oscar/current/active/python-flint/.mesonpy-9e_1lo1i/meson-logs/meson-log.txt

ERROR Backend subprocess exited when trying to invoke build_sdist

These CI steps have to do a lot of work just to package up the files for the sdist:
https://github.com/flintlib/python-flint/blob/430b8e50f4d8e697d4f5c7666d7b75f3e440a260/.github/workflows/buildwheel.yml#L67-L73

This PR is to work around this problem in another situation by using git archive instead of asking meson-python to build the sdist: flintlib/python-flint#158

[rgommers]

From docs/how-to-guides/sdist.rst: "To generate a source distribution, meson-python must successfully configure the Meson project by running the meson setup command."

It can't be avoided, since we need to run meson setup before we can run meson dist. And that's needed to support functionality like dist scripts to customize the content of an sdist (e.g., when you want to store a generated file containing versioning info from your VCS).

I'll also link gh-567 as a related discussion, which may contain a few answers to related questions.

This PR is to work around this problem in another situation by using git archive instead of asking meson-python to build the sdist: flintlib/python-flint#158

That does not yield a valid sdist, and may or may not work for Sage depending on what you may add to meson.build files in the future or what changes in meson-python. So I recommend not doing this - there really isn't a big enough problem to solve here to think about doing this. Generating an sdist is a rare action normally reserved for maintainers; if you can build a wheel on even one platform, you are able to generate an sdist too.

CI images have compilers and other dev tools; if it's really a problem to have non-standard dependencies like libflint available in the job where Sage needs an sdist, a simple workaround would be to add a build option like -Dskip-detecting-deps with:

if get_option('skip-detecting-deps')
    # Only for more easily generating an sdist in CI without having `flint` installed
    flint_dep = []
else
    flint_dep = dependency('flint')
endif

and then you can build the sdist with:

python -m build --sdist -Csetup-args=-Dskip-detecting-deps=true

[oscarbenjamin]

Could it be that the dependencies are declared as optional but still required when actually attempting to build the extension modules but not for making an sdist?

[rgommers]

Well, that's what my snippet above does. It cannot be done without a single line of code changing, since meson setup knows nothing about how it's being called or what an sdist is. dependency() only has a required keyword which is true or false.

[oscarbenjamin]

Maybe there could be a general meson configuration option for building a source distribution like meson setup --dist-only or something. Then meson could understand that only dependencies of dist scripts are actually needed because none of the other targets are going to be built.

Currently it seems that meson dist runs a full configure, then cythonises all the files before ignoring cython's outputs and putting only the cython sources into the dist. I don't know enough about the internal architecture to understand why that happens but it seems like it should not be necessary.

[eli-schwartz]

It is technically possible for a project to run different dist scripts depending on the result of checking for cc.find_library() and find_program(), although I would personally consider it to be kind of in bad taste if a project had unreliable dist tarballs like that.

Meson also by default on a dist, will build and install the result, so having the requirements available is important in that context, but people doing --dist-only would also surely do meson dist --no-tests anyway I guess...

The cythonization definitely sounds wrong. Meson implements cythonization as a compile step, not a setup step. It will check for availability of the cython program but not do anything with it since the only target built is the dist tarball target -- which has no dependencies.

[oscarbenjamin]

It is technically possible for a project to run different dist scripts depending on the result of checking for cc.find_library() and find_program()

I consider that the purpose of the dist is that I'm packaging up the code to be distributed to other machines. It is no use checking find_library on the machine that creates the dist.

The cythonization definitely sounds wrong.

It is because I tested meson dist without passing --no-tests.

[dnicolodi]

Currently it seems that meson dist runs a full configure, then cythonises all the files before ignoring cython's outputs and putting only the cython sources into the dist. I don't know enough about the internal architecture to understand why that happens but it seems like it should not be necessary.

Unless you are doing something very unusual (and most likely very wrong) in your meson.build this should not happen.

[oscarbenjamin]

I just realised I need --no-tests to build the dist without compiling anything.