Updates to PEP 660 #1978

sbidoul · 2021-06-05T14:15:38Z

A few updates to PEP 660 following ongoing discussions.

gaborbernat · 2021-06-05T14:44:05Z

pep-0660.rst

+installed layout. It is then up to the installer frontend to realize the
+editable installation by whatever mean it deems adequate for its users.
+
+In terms of capabilities, both proposals seem equivalent, at least in


If you're going with this bold statement do clarify how you're planning to support headers, data files and scripts.

Why should I? The whole point is that we don't know (in either proposal) and can let backend authors experiment.

Regarding headers, for instance, assuming there is a use case to install them in editable mode, one can use symlinks. But is there such a use case ? This PEP explicitly says only python files must be "edtiables" and changes to other file may require an additional installation step.

What about scripts? Those can be python files.

Regarding headers, for instance, assuming there is a use case to install them in editable mode, one can use symlinks. But is there such a use case?

Check astropy project that ships headers. The wheel spec allows it so there is a such use case.

Why should I? The whole point is that we don't know (in either proposal) and can let backend authors experiment.

We should at least hypothetically provide a way to achieve this, just to prove that the PEP allows that functionality.

pypa/pip#8215 doesn't seem related to this PEP.

We also rejected my prototype that uses the wheel structure but skips creating an archive. The archive is easy to produce; no one has to worry about cleaning a build directory (full of the unpacked wheel); we avoid describing exactly what an unpacked wheel is.

I think packages that link against other packages that provide headers are used to being able to find those headers in source checkouts, as well as in the installed location? I don't think the way installed headers work in Python is particularly great.

Most scripts are entry_points scripts handled by the standard wheel install.

setup.py develop is about making your code importable by other code, and providing your metadata so that your editable distribution is "installed" according to "pip list".

But old-fashioned scripts are standalone. They need to import library code, but almost never need to be "imported by" dependents. There is less pressure to copy them to the central scripts location during an editable install. If you really are editing, you may run those scripts directly from the checkout. They will work fine.

Data files are almost always relative to the source code (part of purelib/platlib) since unfortunately the data_files feature has been broken for a long time. It does not install into a predictable location given varying virtualenv / --user / global install targets.

This is a long way to say that we think the most important use case is covered without worrying about scripts, headers, and data. It's hard to manually do an in-place build with correct .dist-info metadata on the .dist-info search path. setup.py develop or a replacement editable system solves that.

Could line 218 be reworded to say something more like "both proposals could provide the core 'editable' feature".

My biggest concern with the virtual wheel proposal is that build systems do not work by generating the necessary mapping as an intermediate step. Providing it to the hook could be burdensome and restrict the design of 'editable-capable' build systems. And that while it's pretty obvious to see how the simple case would work, it would still be the installer's responsibility to make sense of whatever crazy mapping was provided to it by the backend - if the structure of the mapping did not resemble the structure of what was on disk?

In this PEP and in setup.py develop the publisher has an incentive to lay out their source code in a way that would (mostly) work with a normal .pth file.

gaborbernat · 2021-06-05T14:50:18Z

pep-0660.rst

@@ -157,11 +158,16 @@ install. This section provides examples and is not normative.
  a path importable, often including the project's own ``setup.py`` and other
  scripts that would not be part of a normal installation. The proxy strategy
  can achieve a higher level of fidelity than path-based methods.
+* Symbolic links are another useful mechanism to realize editable installs.
+  Since, at the time this writing, the ``wheel`` specification does not support
+  symbolic links, they are not directly usable. It is however possible to defer


This seems very magical, and like a hack to me. As an end-user would definitely be surprised to discover this unexpected side-effect when first running my application. Also, I think this would run into issues with resource files where the user is not using importlib.resources but pkg_resources 🤔 Because then you might need to trigger the symlink before a module alongside is actually imported, not?

Sure it's a hack. With an executable .pth it works reliably though. To be honest and blunt, all concepts we are discussing for editables sound like hacks to me (except a .pth that exposes a whole directory).

The backend generating files into the target interpreter post-installation seems like a nightmare for a frontend. All of a sudden has no idea what files it needs to uninstall. This would be a no go for me as a frontend developer. If anything I'd prefer to explicitly ban this in the PEP to provide some sanity for the frontend developers. I think the only type of operation a backend is allowed in the target interpreter is a read and update; create+delete should be prohibited.

I have added another approach to handle symbolic links.

sbidoul · 2021-06-14T20:29:35Z

@dholth have you seen this one ? Anything you'd like to change or add here ?

dholth · 2021-06-14T22:10:22Z

pep-0660.rst

+installed layout. It is then up to the installer frontend to realize the
+editable installation by whatever mean it deems adequate for its users.
+
+In terms of capabilities, both proposals seem equivalent, at least in


pypa/pip#8215 doesn't seem related to this PEP.

We also rejected my prototype that uses the wheel structure but skips creating an archive. The archive is easy to produce; no one has to worry about cleaning a build directory (full of the unpacked wheel); we avoid describing exactly what an unpacked wheel is.

I think packages that link against other packages that provide headers are used to being able to find those headers in source checkouts, as well as in the installed location? I don't think the way installed headers work in Python is particularly great.

Most scripts are entry_points scripts handled by the standard wheel install.

setup.py develop is about making your code importable by other code, and providing your metadata so that your editable distribution is "installed" according to "pip list".

But old-fashioned scripts are standalone. They need to import library code, but almost never need to be "imported by" dependents. There is less pressure to copy them to the central scripts location during an editable install. If you really are editing, you may run those scripts directly from the checkout. They will work fine.

Data files are almost always relative to the source code (part of purelib/platlib) since unfortunately the data_files feature has been broken for a long time. It does not install into a predictable location given varying virtualenv / --user / global install targets.

This is a long way to say that we think the most important use case is covered without worrying about scripts, headers, and data. It's hard to manually do an in-place build with correct .dist-info metadata on the .dist-info search path. setup.py develop or a replacement editable system solves that.

Could line 218 be reworded to say something more like "both proposals could provide the core 'editable' feature".

My biggest concern with the virtual wheel proposal is that build systems do not work by generating the necessary mapping as an intermediate step. Providing it to the hook could be burdensome and restrict the design of 'editable-capable' build systems. And that while it's pretty obvious to see how the simple case would work, it would still be the installer's responsibility to make sense of whatever crazy mapping was provided to it by the backend - if the structure of the mapping did not resemble the structure of what was on disk?

In this PEP and in setup.py develop the publisher has an incentive to lay out their source code in a way that would (mostly) work with a normal .pth file.

dholth · 2021-06-14T22:44:51Z

pep-0660.rst

+  tree such as ``.editable/$COMPATIBILITY_TAGS``, and add that directory to the
+  python path via a ``.pth`` file in the "editable" wheel. Alternatively it is
+  also possible to defer the creation of symbolic links to an import hook or
+  executable ``.pth`` that the backend adds in the "editable" wheel.


The symbolic links section is not very inspiring. We could own our lack of support for symlinks or finally add symlinks to wheel (or encourage someone to add them in the future).

I don't understand the $COMPATIBILITY_TAGS variable.

I wouldn't complain if the backend produced build/editable/ with one or two links to the top-level packages we export; add that to .pth. Solving the 'accidentally import setup.py' problem. I would be confused if a complete tree of links to all our .py was in there.

The deferred idea is best removed. It's possible but weird.

dholth · 2021-06-14T22:46:31Z

pep-0660.rst

@@ -114,11 +114,11 @@ Build-backends must produce wheels that have the same dependencies
 with the exception that they can add dependencies necessary for their editable
 mechanism to function at runtime (such as `editables`_).

-The filename for the “editable” wheel needs to be PEP 427 compliant too. It
+The filename for the "editable" wheel needs to be PEP 427 compliant too. It


Goodbye unicode quotes. I notice a lack of emoji in the PEP as well.

dholth · 2021-06-14T22:46:57Z

pep-0660.rst

+`Another approach was proposed <https://github.com/pypa/pip/pull/8215>`_, where
+the build backend returns a mapping from source files and directories to the
+installed layout. It is then up to the installer frontend to realize the
+editable installation by whatever mean it deems adequate for its users.


sbidoul · 2021-06-20T11:02:44Z

@dholth I have handled your comments, please have a look.

To the best of my knowledge all feedback received in the discussion threads has been incorporated.

sbidoul · 2021-06-20T11:28:34Z

I also removed the controversial scheme argument.

dholth · 2021-06-20T13:36:47Z

pep-0660.rst

+===========
+
+This PEP does not support making the content of the wheel .data directory
+"editable" (scripts, C header files, and other data files).


How about

This PEP focuses on making the purelib|platlib categories (installed into site-packages) "editable". It does not make special provision for the other categories headers|scripts|data. Package authors are encouraged to use console_scripts, make their scripts tiny wrappers around library functionality, or manage these from the source checkout during development.

Because:

distribution-1.0.data/(purelib|platlib|headers|scripts|data).

(purelib|platlib|headers|scripts|data) can all appear in the .data directories. These are called install paths or categories.

Thanks. Update done.

sbidoul · 2021-06-20T18:17:08Z

@pfmoore do you think it would be useful to add links to known prototype in the PEP ?

Other than that this should be ready to merge.

pfmoore · 2021-06-20T22:32:42Z

I think that would be useful, yes.

pfmoore · 2021-06-21T07:34:10Z

Would you prefer to update this PR to add the links when you have them, or do them as a separate PR? Either way works for me.

sbidoul · 2021-06-21T07:54:54Z

@pfmoore feel free to merge. I'll do another one when I have collected all the links if this one is merged.

sbidoul added 5 commits June 5, 2021 11:55

PEP 660: typo

0436c0b

PEP 660: clarify what to put in the wheel

b201e36

PEP 660: mention symbolic links

654ba54

PEP 660: add "virtual wheels" to rejected ideas

76d9186

PEP 660: consistency in referring to "editable" wheels

d606a9d

sbidoul requested a review from pfmoore as a code owner June 5, 2021 14:15

the-knights-who-say-ni added the CLA signed label Jun 5, 2021

gaborbernat reviewed Jun 5, 2021

View reviewed changes

sbidoul added 3 commits June 5, 2021 18:52

PEP 660: typos

b28203c

PEP 660: add another suggestion about symbolic links

399db67

PEP 660: add a clarification about symbolic links

53e4ebe

dholth reviewed Jun 14, 2021

View reviewed changes

sbidoul added 4 commits June 20, 2021 11:57

PEP 660: fix typo

a3b34e5

PEP 660: clarify symlink support

b3d741d

PEP 660: add unpacked wheel to rejected ideas

01b8c15

PEP 660: clarify expectations of an editable install

5630652

sbidoul force-pushed the pep660-take2-sbi branch 2 times, most recently from 6c0f870 to 15d105b Compare June 20, 2021 10:57

sbidoul added 3 commits June 20, 2021 12:58

PEP 660: mention the .data directory

ad2d7bb

PEP 660: clarify differences with PEP 662

f5c5a65

PEP 660: remove the schem argument

137fe99

sbidoul force-pushed the pep660-take2-sbi branch from 15d105b to 137fe99 Compare June 20, 2021 10:59

dholth reviewed Jun 20, 2021

View reviewed changes

PEP 660: further clarify the limitation wrt the .data dir

fc87753

pfmoore approved these changes Jun 20, 2021

View reviewed changes

PEP 660: update title

689e4eb

pfmoore merged commit f06a8c7 into python:master Jun 21, 2021

sbidoul deleted the pep660-take2-sbi branch June 21, 2021 10:07

erlend-aasland mentioned this pull request May 13, 2022

pep8/greppable exception messages erlend-aasland/peps#1

Closed

erlend-aasland mentioned this pull request Jun 27, 2022

pep 687/mark as accepted erlend-aasland/peps#2

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updates to PEP 660 #1978

Updates to PEP 660 #1978

sbidoul commented Jun 5, 2021

gaborbernat Jun 5, 2021

sbidoul Jun 5, 2021

gaborbernat Jun 5, 2021

dholth Jun 14, 2021

gaborbernat Jun 5, 2021

sbidoul Jun 5, 2021

gaborbernat Jun 5, 2021 •

edited

Loading

sbidoul Jun 14, 2021

sbidoul commented Jun 14, 2021

dholth Jun 14, 2021

dholth Jun 14, 2021

dholth Jun 14, 2021

dholth Jun 14, 2021

sbidoul commented Jun 20, 2021

sbidoul commented Jun 20, 2021

dholth Jun 20, 2021

sbidoul Jun 20, 2021

sbidoul commented Jun 20, 2021 •

edited

Loading

pfmoore commented Jun 20, 2021

pfmoore commented Jun 21, 2021

sbidoul commented Jun 21, 2021

Updates to PEP 660 #1978

Updates to PEP 660 #1978

Conversation

sbidoul commented Jun 5, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gaborbernat Jun 5, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbidoul commented Jun 14, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbidoul commented Jun 20, 2021

sbidoul commented Jun 20, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbidoul commented Jun 20, 2021 • edited Loading

pfmoore commented Jun 20, 2021

pfmoore commented Jun 21, 2021

sbidoul commented Jun 21, 2021

gaborbernat Jun 5, 2021 •

edited

Loading

sbidoul commented Jun 20, 2021 •

edited

Loading