Build directory for editable installations

[dnicolodi]

The build directory for editable build is currently .mesonpy/editable/build in the source directory. I don't see a reason why the build directory is nested two levels deep into otherwise empty directories . A flatter directory structure could be used.

While doing so, attaching a Python API tag to the build directory name would allow to have multiple parallel installations of the same package for different interpreters.

Isn't there prior art for choosing the name of the build directory? I don't like something specific to meson-python much. Python uses __pycache__ for its cache. Is it allowed to add tool specific stuff in there? distutils uses a build folder, IIRC. What about other build backends?

It would be nice to have this decided before we release a version with the editable support.

EDIT: reworded for politeness

[rgommers]

Isn't there prior art for choosing the name of the build directory?

I'd say build/ in the root dir is the one more or less standard name - not just distutils/setuptools, but also Sphinx (also that also uses _build sometimes). Not sure about other backends, never had a need to actually use them.

The downside of build is that it may clash with a standard choice for using Meson directly (meson setup build, ninja -C build is already mostly muscle memory for me).

Reusing __pycache__ is a no no I'd say.

Also attaching a Python API tag to the build directory name would allow to have multiple parallel installations of the same package for different interpreters.

Hmm, that is an interesting point.

[rgommers]

I'd say that whatever we choose here is a private implementation detail though - the user has no business doing anything with it beyond adding it to .gitignore, so we should be fine tweaking it later.

[rgommers]

Thinking about this some more:

• a hidden dir isn't nice, it's already tricked me once or twice, and I kinda knew it should be there
• build is the standard name
• separate interpreters can already be supported via the builddir option (I tested it with pip install -e . --no-build-isolation --config-settings=builddir=build-py310 and it works)
• build clashing with the distutils default may actually be a good thing; having in-tree remnants from a distutils build can in some cases cause problems

tl;dr let's use build?

[dnicolodi]

I like build too. I like supporting multiple interpreters out of the box. Having a two level directory structure solves this: build/cp311 or build/cpython-3.11/ but I don't know if this is a common enough use case that is worth supporting.

[rgommers]

I'd prefer to not special-case interpreters I think. Different builds can also happen with different compilers (e.g., using GCC and Clang in parallel), different sets of build dependencies (on SciPy testing with 2 numpy versions is much more interesting than testing with 2 cpython versions), and so on.

Plus it does bring up questions like "what if we have regular and a debug 3.11 interpreters".

Matrix'ed environment testing is for tools like tox/nox or custom scripts I'd say, and they can easily pass a builddir for anything like this.

[FFY00]

Other possible option would be putting the build directory on the installation path, instead of the project source. This would also allow installations to different interpreters without any conflicts.

[rgommers]

Also attaching a Python API tag to the build directory name would allow to have multiple parallel installations of the same package for different interpreters.

+1 on this.

Okay, you both agree on this, so let's go that way.

For the other choices, how about I make an editorial decision here? Seems preferred over lots more discussion.

[dnicolodi]

I proposed tagging the build directory with the python interpreter API tag in some form because the editable wheel documentation page prominently warns (or at least I remember it this way) to the fact that being able to have editable wheels for multiple interpreters as a limitation. I thought that, if the limitation was felt so important, adding a tiny amount of complexity to allow it could have been a good compromise.

However, I'm a big fan of keeping things simple unless necessary, thus I'm happy without this feature and a paragraph in the documentation that mentions how to achieve the same using the builddir config setting.

I'm fine with @rgommers picking the build directory name that he prefers. I opened this issue only because the nested directories are not required to exist anymore and I thought we could simplify what we present to the users before releasing editable wheel support in the next release.

[dnicolodi]

I'd say that whatever we choose here is a private implementation detail though - the user has no business doing anything with it beyond adding it to .gitignore, so we should be fine tweaking it later.

We can use the same trick that Meson uses and place a .gitignore containing * in the build directory. Meson does not do that when the build directory is not empty, and we add the native file to the build directory before invoking meson setup, thus Meson does not do that for us.

[dnicolodi]

setuptools uses a build/__editable__.* directory for editable installs in strict mode https://setuptools.pypa.io/en/latest/userguide/development_mode.html#strict-editable-installs I haven't checked yet what .* should be replaced with