Let buildout run a script to create a constraints file. [5.2]

[mauritsvanrees]

This replaces the buildout.requirements extension.

Good points for this extension:

• It is a proper buildout extension.

Bad points for this extension:

• It cannot be run standalone: it only gathers information while buildout is running.
• It ignores version pins for packages that buildout does not use in this run:
  if you run bin/buildout install part1, then version pins for other parts are ignored.
• On Jenkins node 4 with Python 3.6 the package is broken.
  Jenkins node 4 has problem with Python 3.6 and locales #642

Good points for the new script:

• It can be run as a standalone script.
  You just need any Python with any zc.buildout version.
• It is fast: it takes maybe two seconds.
• It reports all version pins, not just the ones that got installed.

Bad points for the new script:

• It is not a proper buildout extension.
  We want to run it from buildout, so we need a separate section/part for it,
  using plone.recipe.command (which is even older than buildout.requirements, though it works fine on Python 3).
  But it should be fairly easy to turn this into an extension (or recipe).
• It is hardcoded to read buildout.cfg, including its extends.
  Should be easy enough to make this configurable with argparse/optparse/whatever.

Script adapted from: https://gist.github.com/mauritsvanrees/c47e974e418c707a626200cb6561405b
See also https://community.plone.org/t/creating-constraints-and-requirements-from-buildout-config/10296

We cannot use configparser, because buildout configs contain 'duplicate' options:

$ grep -i chameleon versions.cfg
Chameleon = 3.6.2
chameleon = 3.6.2

So we use the buildout configparser.
Bonus: we automatically use the section expressions, like [versions:python27].
So the end result of running this script can be different per Python version.
That is something to remember.

[mauritsvanrees]

@jenkins-plone-org please run jobs

[mauritsvanrees]

Oops, on Jenkins the command has this result:

Installing constraints.
constraints: Running 'bin/python create-constraints.py'
/bin/sh: 1: bin/python: not found

Buildout continues, which I think is a good thing (better than the current breakage of buildout.requirements on that single node on Py 3.6.).
But I should fix this.

[jensens]

plone.versioncheck knows all versions from buildout from all sections. What about adding a dump from there? This includes picked ones.

[mauritsvanrees]

Fixed an error: bin/python was not known.

@jenkins-plone-org please run jobs

[mauritsvanrees]

plone.versioncheck knows all versions from buildout from all sections. What about adding a dump from there?

Then you must run buildout at least once to gather this information. And probably also run it after every change. Or not?

I want to be able to point to any versions file, maybe also online, and parse it to a constraints.txt. (Not possible at the moment, but that is very doable).

Actually, I see that bin/versioncheck is also fast, and seems to be able to analyse it fine, without needing to do a complete buildout run (except to install itself, but that is not the point).

Does it handle the [versions:python27] expressions correctly?
I guess that depends on what I mean by 'correctly'. But it looks like it does that part the same as my script: it just lets buildout handle it.

Do you think creating a constraints.txt file is something that would fit in plone.versioncheck? It is not what it was created for, but it may be quite doable.

[jensens]

Then you must run buildout at least once to gather this information. And probably also run it after every change. Or not?

https://github.com/plone/plone.versioncheck/blob/master/src/plone/versioncheck/parser.py does not need any buildout to be run before.

Does it handle the [versions:python27] expressions correctly?

I am not sure, probably this need to be fixed/added.

Do you think creating a constraints.txt file is something that would fit in plone.versioncheck? It is not what it was created for, but it may be quite doable.

Indeed it seems a bit out of scope. But it gathers all the data already. And writing an output to constraints.txt would be a straight forward task.

[mauritsvanrees]

Oh yeah: I compared the constraints file as output by buildout.requirements and by my script:

• My script had more, as indicated.
• A few were spelled differently, e.g. Markdown versus markdown.
• Some versions were different in a way that did not matter: zope.lifecycleevent==4.3.0 versus 4.3.
• A pip install Plone -c constraints.txt and then a pip freeze gave exactly the same results. That is what matters.

My immediate concern is that Jenkins 3.6 sometimes breaks because of buildout.requirements, and this would fix it for now.

[jensens]

LGTM

[mauritsvanrees]

Only a totally unrelated robot test failure on 2.7.
Approved, I merge.