Is it allowed to build packages that contain symlinks to external files?

[kalekundert]

Let me begin by briefly explaining what I'm trying to do, because it could be that I'm just approaching the whole thing wrong. I'm writing a package that provides the ability to build a database and make queries against it. I recently decided that it'd be best to split these two tasks (building and querying) into separate packages, because (i) each user will usually only be interested in one task or the other, not both, and (ii) both tasks require very different sets of dependencies, and some of those dependencies can be hard/undesirable to install in the environments where the other package would be used. However, the tasks are not fully independent; there's a lot of shared "utility" code that is used for both.

My thought was to approach this by creating a single repository with two packages, and symlinking any shared source files from one package to the other. I was hoping that the symlinks would be automatically replaced by real files in sdists/wheels, so I'd end up with two independently distributable packages that share some source code.¹

To see if this approach was viable, I made a minimal example of such a setup. The reason I'm writing this question is that, when playing around with this example, I found that some commands (pip install, flit build --format) handled symlinks in the way I was hoping for, while other commands (that I thought were equivalent—flit build, python -m build) crashed. Given the contradictory results, I'm not sure if this symlink scheme is supported or not. Is it just a fluke that it worked for pip? Or is it a bug that it didn't work for flit build? Either way, how is it even possible that it could work for pip without working for flit build?!

Minimal example:

See the above-linked repository for full details, but basically it contains two packages, one named foo and the other named bar. foo contains a module named shared.py, and bar contains a (relative) symlink to that same module. Each module (foo, bar, and shared) has an attribute called name which just contains the name of the module in question, as a string.

When I install both packages using pip, everything works:

$ python -m venv venv_pip
$ . venv_pip/bin/activate
$ pip install ./foo_pkg
...
Successfully installed foo-0.0.0
$ pip install ./bar_pkg
...
Successfully installed bar-0.0.0
$ python -c 'import foo; print(foo.name)'
foo
$ python -c 'import foo.shared; print(foo.shared.name)'
shared
$ python -c 'import bar; print(bar.name)'
bar
$ python -c 'import bar.shared; print(bar.shared.name)'
shared

When I try using python -m build to make distributions, I encounter the following error:

$ python -m venv venv_build
$ . venv_build/bin/activate
$ pip install build
$ python -m build foo_pkg
...
Successfully built foo-0.0.0.tar.gz and foo-0.0.0-py2.py3-none-any.whl
$ python -m build bar_pkg
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - flit_core >=3.2,<4
* Getting build dependencies for sdist...
* Building sdist...
* Building wheel from sdist
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - flit_core >=3.2,<4
* Getting build dependencies for wheel...
* Building wheel...
Traceback (most recent call last):
  File "/home/kale/hacking/snippets/python/multipackage_symlinks/venv_build/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 373, in <module>
    main()
  File "/home/kale/hacking/snippets/python/multipackage_symlinks/venv_build/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 357, in main
    json_out["return_val"] = hook(**hook_input["kwargs"])
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kale/hacking/snippets/python/multipackage_symlinks/venv_build/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 271, in build_wheel
    return _build_backend().build_wheel(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/build-env-4167r9eb/lib/python3.12/site-packages/flit_core/buildapi.py", line 72, in build_wheel
    info = make_wheel_in(pyproj_toml, Path(wheel_directory))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/build-env-4167r9eb/lib/python3.12/site-packages/flit_core/wheel.py", line 224, in make_wheel_in
    wb.build(editable)
  File "/tmp/build-env-4167r9eb/lib/python3.12/site-packages/flit_core/wheel.py", line 210, in build
    self.copy_module()
  File "/tmp/build-env-4167r9eb/lib/python3.12/site-packages/flit_core/wheel.py", line 164, in copy_module
    self._add_file(full_path, rel_path)
  File "/tmp/build-env-4167r9eb/lib/python3.12/site-packages/flit_core/wheel.py", line 111, in _add_file
    zinfo = zipfile.ZipInfo.from_file(full_path, rel_path)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kale/.pyenv/versions/3.12.2/lib/python3.12/zipfile/__init__.py", line 557, in from_file
    st = os.stat(filename)
         ^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'bar/shared.py'

ERROR Backend subprocess exited when trying to invoke build_wheel

When I try using flit build to make distributions, I encounter the same error, but only if I don't specify the --format option:

$ python -m venv venv_flit
$ . venv_flit/bin/activate
$ pip install flit
$ cd foo_pkg
$ python -m flit build
Found 3 files tracked in git                                        I-flit.sdist
Built sdist: dist/foo-0.0.0.tar.gz                             I-flit_core.sdist
Copying package file(s) from /tmp/tmp3232v1ow/foo-0.0.0/foo    I-flit_core.wheel
Writing metadata files                                         I-flit_core.wheel
Writing the record of files                                    I-flit_core.wheel
Built wheel: dist/foo-0.0.0-py2.py3-none-any.whl               I-flit_core.wheel
$ cd ../bar_pkg
$ python -m flit build
Found 3 files tracked in git                                        I-flit.sdist
Built sdist: dist/bar-0.0.0.tar.gz                             I-flit_core.sdist
Copying package file(s) from /tmp/tmpul817nvr/bar-0.0.0/bar    I-flit_core.wheel
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/kale/hacking/snippets/python/multipackage_symlinks/venv_flit/lib/python3.12/site-packages/flit/__main__.py", line 5, in <module>
    main()
  File "/home/kale/hacking/snippets/python/multipackage_symlinks/venv_flit/lib/python3.12/site-packages/flit/__init__.py", line 191, in main
    main(args.ini_file, formats=set(args.format or []),
  File "/home/kale/hacking/snippets/python/multipackage_symlinks/venv_flit/lib/python3.12/site-packages/flit/build.py", line 54, in main
    wheel_info = make_wheel_in(tmp_ini_file, dist_dir)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kale/hacking/snippets/python/multipackage_symlinks/venv_flit/lib/python3.12/site-packages/flit/wheel.py", line 8, in make_wheel_in
    return core_wheel.make_wheel_in(ini_path, wheel_directory, editable)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kale/hacking/snippets/python/multipackage_symlinks/venv_flit/lib/python3.12/site-packages/flit_core/wheel.py", line 224, in make_wheel_in
    wb.build(editable)
  File "/home/kale/hacking/snippets/python/multipackage_symlinks/venv_flit/lib/python3.12/site-packages/flit_core/wheel.py", line 210, in build
    self.copy_module()
  File "/home/kale/hacking/snippets/python/multipackage_symlinks/venv_flit/lib/python3.12/site-packages/flit_core/wheel.py", line 164, in copy_module
    self._add_file(full_path, rel_path)
  File "/home/kale/hacking/snippets/python/multipackage_symlinks/venv_flit/lib/python3.12/site-packages/flit_core/wheel.py", line 111, in _add_file
    zinfo = zipfile.ZipInfo.from_file(full_path, rel_path)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kale/.pyenv/versions/3.12.2/lib/python3.12/zipfile/__init__.py", line 557, in from_file
    st = os.stat(filename)
         ^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpul817nvr/bar-0.0.0/bar/shared.py'
$ python -m flit build --format sdist
Found 3 files tracked in git                                        I-flit.sdist
Built sdist: dist/bar-0.0.0.tar.gz                             I-flit_core.sdist
$ python -m flit build --format wheel
Copying package file(s) from bar                               I-flit_core.wheel
Writing metadata files                                         I-flit_core.wheel
Writing the record of files                                    I-flit_core.wheel
Built wheel: dist/bar-0.0.0-py2.py3-none-any.whl               I-flit_core.wheel

If you got this far, thanks for reading all this! I'd appreciate any insight you can give on what's happening, and what should be happening.

Footnotes

1. I know I could also move the shared code into its own package. That would definitely work, but it would lead to a more complicated release process, because then I'd have to manage three packages rather than just two. It would also become possible for the "utils" package to get out-of-sync with the "build" and "query" packages, which could cause weird bugs. So I want to avoid this solution. ↩

[takluyver]

I'd be inclined to say it's a fluke that it worked with pip. I'd guess the difference is that flit build and python -m build both build an sdist first, then unpack that and build the wheel from that, whereas pip probably builds a wheel directly from the source tree you point it to.

One difficulty here is that the packaging tools don't know what you want a symlink to mean. In your explanation, it's clear that you want the symlink to be materialised in the packaged/installed files, but someone else might put a symlink inside their source code and expect it to still be a symlink when it's installed. It could also potentially be a security risk - if I prepare a git repo or an sdist containing a symlink and get someone else to make a wheel from it, materialising the link, can I steal a secret file? I accept that that sounds far fetched, but find a few bits of unexpected behaviour like this and stick them together, and you could cause all kinds of mischief...

I'm not saying symlinks are totally out, but I'd want to think carefully about how they should work and what limitations should apply to them. One possible limitation is not following links outside the package folder, which would prevent your use case from working.

Another possible approach for your use case is to make one package with two sets of optional dependencies, so users would install mypkg[build] to create the database or mypkg[query] to query it.

[kalekundert]

Thanks for such a quick reply! I'm not surprised that there are some real issues with supporting a scheme like this, and it's good to know that this isn't something I should keep banging my head on.

Optional dependencies are a good suggestion, and probably what I'll end up doing. The downside is that I'll need to add some code to gracefully check for which dependencies are available, but that won't be too bad.

[takluyver]

Thinking about this some more, I might in the future explicitly disallow symlinks, i.e. error on trying to build sdists or wheels from source files containing a symlink. Symlinking to something outside the project directory could be (part of) a security issue, as already mentioned, and I'm not sure there's a compelling use case for symlinking one location inside the project to another. Flit also doesn't try to cover every possible use case, so even if there are rare scenarios where one wants symlinks, it may still be out of scope.

This would like be in Flit 4.0, just in case someone is doing something with symlinks that currently works, despite the lack of deliberate support.

[takluyver]

Here's a previous discussion about it: #408

[kalekundert]

Thanks for the link; I hadn't found that discussion myself. It does seem that setuptools behaves how I want in this case. I'm a little hesitant to rely on it, because I can't find any documentation saying that it should behave how I want, but that's another option for me.