Proposal: Use pip constraints as canonical version location

[mauritsvanrees]

Currently the canonical place where we define which versions make up a Plone release is in Buildout files, mainly here: https://github.com/plone/buildout.coredev/blob/6.0/versions.cfg
To support pip, we also publish a constraints.txt on dist.plone.org: https://dist.plone.org/release/6.0.0b3/constraints.txt
This file is created using two scripts that I call with tox -c release/tox.ini.

Pip is getting more important in the Plone world. I think we should move to making constraints.txt the canonical place, and derive the versions.cfg from there. So the other way around than it is now.
Some new possibilities then arise:

• Create nightly backend images without needing a script to turn versions.cfg into constraints.
• Use pip instead of buildout to setup the test environment for Jenkins (or gh-actions)
• Also, if we do this correctly, then versions.cfg and constraints.txt will be in sync all the time, instead of only when I am preparing a release.

So how could we get this to work? With mxdev.

requirements-plone.txt

We will start in buildout-coredev with a requirements-plone.txt, something like this:

# -c https://raw.githubusercontent.com/zopefoundation/Zope/master/constraints.txt
-c https://zopefoundation.github.io/Zope/releases/5.6/constraints.txt
# -c https://dist.plone.org/release/6.0.0b3/constraints.txt
# -c release/constraints.txt
-c constraints-extra.txt
-c constraints-ecosystem.txt
-c constraints-core.txt

It only contains references to constraints files, so perhaps call it constraints-plone.txt. We will have to see how we structure these files: several like in this example, or put everything back in one file. I voiced some ideas for the current buildout versions files on community.

mxdev.ini

Part 2 is a mxdev.ini file that uses the requirements file:

[settings]
requirements-in = requirements-plone.txt
requirements-out = requirements-mxdev.txt
constraints-out = constraints.txt
default-target = src

version-overrides =
# Zope 5.6 is not actually pinned in the Zope constraints...
    Zope == 5.6
# Here we should have versions that override Zope.
    persistent == 4.9.1

ignores =
# Let's ignore pip and friends, so we accept whatever is already installed.
# The canonical versions are in requirements.txt.
    pip
    setuptools
    wheel
    zc.buildout

# I want an auto-checkouts option, which mxdev does not have yet.
# Default can be '*', so all defined sources are checked out.
# auto-checkout =
# These packages are always checkout out:
#     Plone
#     plone.app.locales
#     plone.app.upgrade
#     Products.CMFPlone
#     Zope
#     mockup
#     docs

# Below would be the source definitions, similar to what we have in `sources.cfg`.

Then whenever you edit the source constraint files, you call mxdev -c mxdev.ini and the final constraints.txt file containing all constraints is created. The above example works for pure constraints, it does not do any checkouts/clones/editable-installs yet.

versions.cfg

How do we create a versions.cfg file? From the mxdev docs: "Functionality of mxdev can be extended by hooks. This is useful to generate additional scripts or files or automate any other setup steps related to mxdev’s domain." So we create a small package with a mxdev hook that takes the generated constraints and writes them to a versions.cfg file. (Or we put this in mxdev core, but I guess this is not wanted.)

markers / section conditional expressions

The basis of creating a Buildout versions file is of course easy: package==1.0 becomes package = 1.0. But how about this:

package==2.0
package==1.0; python_version <= "3.9"

How do we support this and all other markers in Buildout? I have created a buildout issue for this, and a PR. Basic idea: teach Buildout how to read the pip constraints markers. Then the mxdev hook can write the above as:

[versions]
package = 2.0

[versions:python_version <= "3.9"]
package = 1.0

plone.releaser

The release team uses plone.releaser as wrapper around zest.releaser. This has some nice additions like updating versions.cfg with the pinned version of the package you have released. This code would need to be adjusted to update the constraints file and run mxdev.

mr.developer

I think if mxdev fetches the editable installs, then we might not need the mr.developer extension anymore: the correct checkouts are already there in src. The mxdev hook could take the auto-extensions list and insert this in a [buildout] develop option at the top of the generated versions.cfg. If you want to develop on the coredev buildout, you would do:

git pull
mxdev -c mxdev.ini
bin/buildout

Plan

Does this sound like a plan that could work? So we have the following actions that are needed:

• Teach Buildout about PEP 508 markers: Add support for PEP 508 markers in section condition expressions. buildout/buildout#622
• Port the auto-checkout option from mr.developer to mxdev so we can have a mxdev.ini file with all source definitions, but only a few checked out in editable mode
• Write the mxdev hook.
• Update plone.releaser to update the constraints and run mxdev.

Once this is in place, then we could make the switch in the core development buildout.

Of course this is for Plone 6.0 only. And given what still needs to be done, this does not seem like something to do before Plone 6.0 final.

Feedback welcome!

[gforcada]

Great write up, as usual 😉

Wild question: if mxdev and pip can 1:1 replace buildout, do we want to keep buildout at all? 🤔 note that I'm still using buildout, but if mxdev and pip are mature enough and can handle it, do we want to keep both systems sort of official?

If this is the transition plan from one to the other, it looks fine, not sure if the buildout PR will take too much time, and somehow is the blocker of all this, but if it is only to abandon it afterwards, we might as well do the change in a lockstep, specially with good documentation, so we don't have that blocker? 🤔

[davisagli]

@gforcada I had the same thought, @gforcada. But, I think we need to think it through, maybe make a list of all the things that the buildout is used for and make sure that we have a good replacement for each one.

I'll review the rest of @mauritsvanrees' proposal more carefully soon, but in general I like the idea.

[mauritsvanrees]

There are still lots of useful buildout recipes, and a big installed base that is not quickly moved over to pip and some scripts or cookiecutter templates.

FWIW, I tried moving one part of mxdev to pip: the part the overrides the constraints. It does not look like it will ever land in pip though, so I closed the PR. See pypa/pip#11537

[mauritsvanrees]

My PR for Buildout was accepted, after I fixed some long standing failing tests with pip 22.2+. So the first hurdle is done. And zc.buildout 3.0.0 final is out! I have made a PR to let the coredev buildout use this version.

[mauritsvanrees]

Update from Alpine City Sprint 2023.
mxdev now has a general option default-use, which you can override in individual package definitions with use. Default is unchanged: if you do not specify anything, all defined packages are checked out. In coredev we would define one hundred or so Plone packages (probably not including most Zope ones). In the general mxdev settings we would set default-use = false. In packages that are now in auto-checkout in buildout, we set use = true. And then we have one part solved.
To be continued.

[tisto]

@mauritsvanrees what's the status of this? Still planned for Plone 6.1.

[mauritsvanrees]

This is in. The main change has been merged with plone/buildout.coredev#945

But it is not so much "pip constraints as canonical version location", but "pip constraints and buildout version on the same level."