EasyBuild v4.0

[boegel]

This issue keeps track of what changed are planned for EasyBuild v4.0, in which backwards-incompatible changes will be made compared to EasyBuild v3.x.

ETA for EasyBuild v4.0: summer 2019 (subject to change!)

---

Codebase

• drop support for using EasyBuild on top of Python 2.6 (?) (not dropping Py2.6 support for now)

  • after consulting with community via user survey?
  • only if Python 2.6 support really gets in the way (mainly w.r.t. also supporting Python 3)
  • (turns out this is not needed, we can keep supporting Python 2.6 without too much hassle)

• drop deprecated functionality (see log.deprecated statements)

  • (no action needed here, log.deprecated("...", '4.0') statements will now simply result an error

• support running EasyBuild on top of (both Python 2.7 &) Python 3 (because 2+3=4) [DONE]

  • see support Python 3.x easybuild-framework#133
  • blocker: vsc-base should be compatible with Python 3 (cfr. start looking into making vsc-base Py3 compatible hpcugent/vsc-base#231, )
    • fixed by ingesting vsc-base & vsc-install into framework, see ingest vsc-base & vsc-install easybuild-framework#2708
  • see PRs marked with python3 label: https://github.com/easybuilders/easybuild-framework/labels/python3

• avoid setuptools dependency [DONE]

  • only needed for pkg_resources.declare_namespace used to spread easybuild namespace across multiple directories
  • also relevant for --include-* support, but can be worked around? (OK)
  • central idea is to basically just replace the use of pkg_resources.declare_namespace by pkgutil.extend_path, which seems to be the one true way (tm), cfr. https://packaging.python.org/guides/packaging-namespace-packages/#pkgutil-style-namespace-packages ("This is the recommended approach for the highest level of compatibility.")
  • see drop requirement for setuptools as runtime dependency easybuild-framework#2836 + only import from pkg_resources (provided by setuptools) to check GC3Pie version easybuild-framework#2837

---

Packaging & installation

• try to avoid requiring setuptools to install EasyBuild (try to avoid requiring setuptools to install EasyBuild easybuild-framework#2941) (DONE)
• update bootstrap script to be able to install EasyBuild 4.0 (see update bootstrap script to support installing EasyBuild v4.0 easybuild-framework#3017)

---

Documentation

• overview of (breaking) changes
  • see https://easybuild.readthedocs.io/en/develop/EasyBuild4-overview-of-changes.html
• release notes

---

Features

• more flexibility w.r.t. dependency versions?
  • unclear how at this point...
• (fat) easyconfigs in YAML syntax (.yeb)?
  • unlikely, unless someone has time to work on it...
• drop exts_classmap? (will look into deprecating this in EasyBuild 4.x)
  • see exts_classmap should be removed, and replaced by an ext_option option easybuild-framework#506
• replace dummy toolchain with system toolchain
  • deprecate dummy toolchain to trigger deprecation warning on use, to be removed in EasyBuild 5.0
  • see Please rename the dummy toolchain easybuild-framework#1181
  • done, see deprecate 'dummy' toolchain, replace with 'system' toolchain easybuild-framework#2877
  • also add alias for system toolchain to EasyBuild 3.x: support using 'system' as alias for 'dummy' toolchain easybuild-framework#2960
• [ ] deprecate runtest in favor of testopts, see deprecate runtest in favor of testopts (was: Add pretestopts and testops) easybuild-framework#2868
• stable support for --containerize (needs more time)

---

Easyblocks

• PythonPackage

  • switch to installing Python packages with pip by default (rather than setup.py install, i.e. setuptools) (not switching the default, but will keep requiring use_pip = True in contributed easyconfigs)
  • cfr. https://docs.python.org/2/installing/index.html
  • needs to be re-evaluated...
  • enable download_dep_fail by default, to ensure that no dependencies are automatically downloaded by whichever Python installation tool is being used (already done in PythonBundle)

• custom easyblock for OpenMPI (done, see add custom easyblock for OpenMPI easybuild-easyblocks#1789)

---

Easyconfigs

• archiving of old easyconfig files
  • anything using ictce toolchain (already deprecated in EasyBuild 3.8.0)
    • archive easyconfigs using deprecated ictce or intel (< 2016a) toolchain easybuild-easyconfigs#8558
  • anything using goolf/1.4.10 (already deprecated in EasyBuild 3.8.0)
    • archive easyconfigs using deprecated goolf(c) toolchain easybuild-easyconfigs#8557
  • also oldest versions of intel and foss toolchains? (already deprecated in EasyBuild 3.8.0)
  • < intel/2016a: archive easyconfigs using deprecated ictce or intel (< 2016a) toolchain easybuild-easyconfigs#8558
  • < foss/2016a: archive easyconfigs using a gompi/foss toolchain older than {gompi,foss}/2016a easybuild-easyconfigs#8585 (EasyBuild v3.9.4)
  • archive easyconfig for old software versions:
  • archive ancient versions of GC3Pie/GCC/OpenMPI/ORCA easybuild-easyconfigs#8586 (EasyBuild v3.9.4)
  • try to avoid losing support for particular software packages while doing this...
• "basic" Python at GCCcore + SciPy bundle?
  • done, see {lang}[GCCcore/8.2.0] Python v3.7.2 easybuild-easyconfigs#7898 (Python 3.7.2), {lang}[GCCcore/8.2.0] Python v2.7.15 easybuild-easyconfigs#7821 (Python 2.7.15), {lang}[foss/2019a] SciPy-bundle v2019.03 w/ Python 2.7.15 + 3.7.2 easybuild-easyconfigs#7921 (SciPy-bundle)
• adopt ComputeCanada's approach to installing Python packages for multiple Python versions
  • done, see add support for multi_deps easyconfig parameter (syntactic sugar for iterative list of builddependencies) easybuild-framework#2813
• avoid warnings w.r.t. naming of local variables (all done!)
  • cfr. recommended naming scheme for local variables in warn if non-single-letter local variables not named 'local_*' or '_*' are used + enhance --fix-deprecated-easyconfigs to rename them easybuild-framework#2938
    • docs update @ add documentation on use of local variables in easyconfig files #527

[cclauss]

You folks do know that https://snarky.ca/stop-using-python-2-6 was written 3 1/2 year ago?

[boegel]

We're fully aware that Python 2.6 is obsolete, and has been for a long time, yet we opt to keep supporting it for now.
The main reason is that a small (& shrinking) part of the EasyBuild user base (~16% early 2018, ~6% early 2019 according to our user survey) is still using an OS that has Python 2.6 as the default system Python (for example CentOS 6.x). Although upgrading that to Python 2.7 is definitely doable, many people opt out of that for the sake of maintaining system stability.

I re-evaluated whether or not to keep support for Python 2.6 while working on the port to Python 3.x (in the 4.x branches), and it turned out to be very little effort to do so (since we're definitely going to keep support for Python 2.7 for a while, despite it going EOL end of 2019, for very similar reasons).

The main benefit would be being able to remove small parts of code that are required to maintain Python 2.6 support, which we haven't touched in years (so they have very little impact on development), and being able to clean up the mess (a bit) in the requirements.txt for the EasyBuild framework component (cfr. your PR easybuilders/easybuild-framework#2874).

In my view, neither of those reasons are compelling enough to drop Python 2.6 support and annoy a minority of our user base along the way, especially since it has little impact on other users or ongoing development.

[cclauss]

@hugovk Would you be willing to show us how to get the Python versions being used to pip install:

• easybuilders/easybuild
• easybuilders/easybuild-framework
• easybuilders/easybuild-easyconfigs
• easybuilders/easybuild-easyblocks

Each of these repos has a 4.x branch that will add Python 3 support. The current users should all be on legacy Python and Python 2.4 is still supported. It would be interesting to know which versions of Python are being used to install the EasyBuild tools. Thx.

[boegel]

@cclauss Are you looking for this? https://pypistats.org/packages/easybuild

The Python 2.4 is a mistake in the setup.py in https://github.com/easybuilders/easybuild, I'll get that fixed, since it's actually not supported anymore (Python 2.6 is current minimal required version).

[hugovk]

Yes, https://pypistats.org is the easiest way. And using the CLI client:

pip install pypistats
pypistats python_minor easybuild --last-month
pypistats python_minor easybuild-framework --last-month
# etc.

---

Here's the pip installs for easybuild from PyPI for April 2019:

category	percent	downloads
2.7	44.19%	1,076
null	30.39%	740
2.6	24.60%	599
3.6	0.41%	10
3.7	0.21%	5
3.4	0.16%	4
3.5	0.04%	1
Total		2,435

easybuild-framework:

category	percent	downloads
2.7	48.37%	1,305
null	29.21%	788
2.6	22.24%	600
3.6	0.11%	3
3.7	0.07%	2
Total		2,698

easybuild-easyconfigs:

category	percent	downloads
2.7	42.62%	1,123
null	34.57%	911
2.6	22.77%	600
3.6	0.04%	1
Total		2,635

easybuild-easyblocks:

category	percent	downloads
2.7	47.28%	1,193
null	28.89%	729
2.6	23.78%	600
3.6	0.04%	1
Total		2,523

[boegel]

Those stats aren't very useful though, since the easybuild-framework test suite involves re-testing our bootstrap script, which does a installation from PyPI on both Python 2.6 & 2.7. That definitely skews these stats, a lot. I'm quite sure that actual usage on Python 2.6 is a lot lower than ~23-24%...

[interrogator]

Hey easybuilders! I'm currently helping out on gc3pie, trying to get it to Python 2.7 + 3.x support. Our main motivation for maintaining 2.6 support is that you have us as a dependency. Truth be told though, the task of updating to Python 3 is made much more difficult when we have to support 2.6 (numerous backports, lack of dict comprehensions and so on).

Regarding the popularity of various Python versions, I also don't believe you can trust the above stats from pypi due to CI. Referring to https://www.jetbrains.com/research/python-developers-survey-2018/#python-3-adoption, we can see that for every 1000 Python users, 3 of them are still on 2.6. I think unless there are specific users with compelling reasons for it, supporting 2.6 is a bit of a waste of developer time at this point.

So I was wondering, if you decided not to drop 2.6 support, perhaps you would like to version lock to a stable, 2.6 supporting gc3pie for now?

[ocaisa]

@interrogator , @boegel is away on holiday so you might have to wait a bit for a response.

My 2c, I think EB should officially drop python2.6 support in v4.0 but keep it in the CI until it becomes too painful to maintain. For the purposes of the CI, locking to a gc3pie version is straightforward.

[interrogator]

OK. I'll wait to hear from @boegel . Feel free to do a version lock in any case, no harm in that I don't think!

[cclauss]

Drop support for Python 2.6.

[interrogator]

@boegel any updates on this?

[boegel]

@interrogator We're working away the last hurdles for EasyBuild 4.0, still hoping to release it by the end of this month (Aug'19)...

There's an early pre-release available via easybuilders/easybuild-easyconfigs#8568, you can use eb --from-pr 8568 to install it with the latest EasyBuild v3.9.3 (older EasyBuild versions won't work to install the pre-release).

Running on top of Python 3 should work fine with that version (which is a bit behind the current 4.x branches, but that's mostly details), but if you see any problems don't hesitate to let us know!

Ignore the warnings you'll see popping up for local variables (something like WARNING: Use of ... unknown easyconfig parameters detected), they're harmless (safe to ignore), we're still working on cleaning things up so those don't pop up anymore; see WIP docs update at https://boegel-eb.readthedocs.io/en/local_vars/Easyconfig-files-local-variables.html for more information on that aspect.

[interrogator]

In particular, I'd like to know if you're willing/able to lock to a commit/release of gc3pie for any older Python versions you are keeping around. If you did that, I think we'd be more comfortable dropping 2.6 support...

[boegel]

@interrogator We only have a minimal required version for GC3Pie now, which is 2.5.0 (see https://github.com/easybuilders/easybuild-framework/blob/master/easybuild/tools/job/gc3pie.py#L90), and I don't see a need to change that.

If people are stuck to Python 2.6, they can keep using the latest GC3Pie 2.5.x?

[riccardomurri]

We'll release GC3Pie 2.6 in a few days. It's likely the 2.6.x releases will be the last ones to support Python 2.6.

[boegel]

Update: develop branches are compatible with Python 2.6, 2.7, 3.5, 3.6 and 3.7 (barring unknown bugs, of course).
Next EasyBuild release (sometime in the coming weeks) will be 4.0.0 and compatible with Python 2.6+ and 3.5+.

[interrogator]

I don't see a need to change that.

@boegel As @riccardomurri points out, anything after the forthcoming gc3pie may not support 2.6, so if you don't add a version lock shortly, a new version of gc3pie could cause problem on your side. So by specifying a maximum version, you'll avoid that problem. Make sense?

[boegel]

EasyBuild v4.0.0 has been released (see https://pypi.org/project/easybuild/4.0.0/), so closing this...

Last remaining TODO is updating the bootstrap script, which is tackled via easybuilders/easybuild-framework#3017