Suggestion: don't fallback to path backend for namespace packages when editables backend is selected

[pawamoy]

Is your feature request related to a problem? Please describe.

The src layout has only one purpose: prevent running tests against the sources.
But it seems PDM, just like Poetry, adds my src directory to PYTHONPATH:

$ python -c 'import sys; print("\n".join(sys.path))'
/home/user/.basher-packages/pyenv/pyenv/versions/3.8.11/lib/python38.zip
/home/user/.basher-packages/pyenv/pyenv/versions/3.8.11/lib/python3.8
/home/user/.basher-packages/pyenv/pyenv/versions/3.8.11/lib/python3.8/lib-dynload
/home/user/data/dev/project/__pypackages__/3.8/lib
/home/user/data/dev/project/src
/home/user/.local/lib/python3.8/site-packages
/home/user/.basher-packages/pyenv/pyenv/versions/3.8.11/lib/python3.8/site-packages

# disable PEP582
$ PYTHONPATH= python -c 'import sys; print("\n".join(sys.path))'
/home/user/.basher-packages/pyenv/pyenv/versions/3.8.11/lib/python38.zip
/home/user/.basher-packages/pyenv/pyenv/versions/3.8.11/lib/python3.8
/home/user/.basher-packages/pyenv/pyenv/versions/3.8.11/lib/python3.8/lib-dynload
/home/user/.local/lib/python3.8/site-packages
/home/user/.basher-packages/pyenv/pyenv/versions/3.8.11/lib/python3.8/site-packages

Describe the solution you'd like

I would like PDM to avoid adding my src folder to PYTHONPATH. I want the installed version of my package to be used when I run my tests, or simply run an interpreter, not the source version. Adding the sources to PYTHONPATH hides potential packaging errors.

I can't seem to find where the src folder is added to PYTHONPATH, it does not seem to be in sitecustomize.py at least, so not sure if it's really PDM that adds it.

[pawamoy]

OK, it seems my src folder is added to PYTHONPATH because that's how the path editable-backend works.
I believe this can be fixed by switching to the editables backend (https://pdm.fming.dev/pyproject/tool-pdm/#editable-build-backend), except for PEP420 namespace packages as explained in the docs, and for which it seems to fallback to the path backend.

[pawamoy]

Would it make sense to make the editable installation fail for implicit namespace packages when the editables backend is used, instead of falling back to the path backend? This would lead to earlier discovery of packaging issues.

[frostming]

pdm install --no-self will not install the project itself

[pawamoy]

But I do want to install the project itself 🙂

Let me restate things:

• local development
• editables backend selected
• missing __init__.py in my package, making it a namespace package
• when editable-installed, PDM falls back to the path backend, so everything works
• but when buliding the sdist or whell, no module is added because config in pyproject in not enough for PDM to understand it's a namespace package
• conclusion: silently works locally, fails once built/published
• instead, when editables backend is selected and package is a namespace package, editable installation should fail, making it clear the package is a namespace package and therefore is missing an __init__.py module

[frostming]

The problem is, editables backend isn't available for PEP 420 namespace packages. For such package, path backend is the only option, the fallback exists so there won't be a hard failure.

[pawamoy]

In my case the hard failure would have let me know that I was missing an __init__.py module. It would have actually helped! Explicit is better than implicit 😄 ? If you deem it's better to silently fallback, it's fine, now I know not to make the same mistake. And maybe there are cases I didn't think about where falling back is helpful as well (multi-packages projects with namespace and non-namespace packages 🤔 ?). Maybe we could just add a warning during fallback.

[frostming]

It has nothing to do with correctness. It is okay for a package without a __init__.py.editables just can't work for PEP 420 packages. PDM has no means to know whether you leave out the __init__.py intentionally or by mistake.

[pawamoy]

Of course, but if I explicitely select the editables backend and my package is a namespace package (voluntarily or not), I would like to at least see a warning that editables does not support it and that PDM falls back to the path backend instead. It is important to know it falls back because these backends have different consequences, namely the path backend will add the sources to PYTHONPATH, which means things will work locally but built sdist/wheels can be broken/empty.

The warning would serve as a hint that something might not be configured as the user intended.

[frostming]

A warning would be okay and easy to add.

[pawamoy]

Thank you so much!