{devel}[foss/2019a] PyTorch-Geometric v1.3.1 w/ Python 3.7.2, numba v0.43.1 w/ Python 2.7.15 3.7.2

[seb45tian]

(created using eb --new-pr)

PR mainly for a bundle of extension libraries for PyTorch (https://github.com/rusty1s). Don't know if the version/naming of the Extension EC file makes sense. Started with 2019.1 since it is for the 2019a toolchain. Help & feedback very welcome!

[terjekv]

sanity_pip_check is enabled in numba-0.43.1-foss-2019a.eb
sanity_pip_check is enabled in PLYfile-0.7-foss-2019a.eb

This usually implies that sanity_pip_check = True is missing from the EasyConfig. This became mandatory in EasyBuild 4.1.1. See https://easybuild.readthedocs.io/en/latest/Release_notes.html#release-notes-eb411 which under easyconfig gives us:

require that sanity_pip_check is enabled in new/changed easyconfigs (#9516, #9576)

When it comes to the multidep issues:

found 2 variants of 'PyTorch' dependency in easyconfigs using '2019a' toolchain generation

• version: 1.1.0; versionsuffix: -Python-3.7.2 as dep for set(['qpth-0.0.13-20190626-foss-2019a-Python-3.7.2.eb', 'torchvision-0.3.0-foss-2019a-Python-3.7.2.eb'])
• version: 1.3.1; versionsuffix: -Python-3.7.2 as dep for set(['PyTorch-Extensions-2019.01-foss-2019a-Python-3.7.2.eb'])

I'm wondering if this is due to the build only supporting Python-3.7.2, but I've yet to wrap my head around multideps in a good way, sorry. :-(

[seb45tian]

When it comes to the multidep issues:

found 2 variants of 'PyTorch' dependency in easyconfigs using '2019a' toolchain generation

• version: 1.1.0; versionsuffix: -Python-3.7.2 as dep for set(['qpth-0.0.13-20190626-foss-2019a-Python-3.7.2.eb', 'torchvision-0.3.0-foss-2019a-Python-3.7.2.eb'])
• version: 1.3.1; versionsuffix: -Python-3.7.2 as dep for set(['PyTorch-Extensions-2019.01-foss-2019a-Python-3.7.2.eb'])

I'm wondering if this is due to the build only supporting Python-3.7.2, but I've yet to wrap my head around multideps in a good way, sorry. :-(

So it is not allowed to have EC files in the same toolchain that depend on different versions of the same application?

i.e. for 2019a
qpth and torchvision use --> PyTorch 1.1.0
PyTorch-Extensions uses --> PyTorch 1.3.1

Should I then downgrade the dependency for PyTorch-Extensions or upgrade the dependency for qpth/torchvision?

[terjekv]

Ah, right. Yeah. Within the same toolchain EB very much wants one version of each dependency. Upgrading qpth and torchvision will impact existing installations if someone tries to reinstall or upgrade the builds. This is, obviously, bad. So for foss/2019a I don't think that's an option. However, pytorch 1.1 is from 2018, so it's not something I'd want to build upon.

If I was to be horrible, I would suggest seeing what pytorch foss/2019b offers, and seeing if going that route is viable instead?

[seb45tian]

If I was to be horrible, I would suggest seeing what pytorch foss/2019b offers, and seeing if going that route is viable instead?

Mmm I see.. however for this PR in particular I would suggest to stick to 2019a and switch to PyTorch 1.1.0 (even if outdated at this point) since every additional library/package is also meant for 2019a. This would keep the PR relatively sane and I can open a new one for 2019b with updated versions.

[terjekv]

If I was to be horrible, I would suggest seeing what pytorch foss/2019b offers, and seeing if going that route is viable instead?

Mmm I see.. however for this PR in particular I would suggest to stick to 2019a and switch to PyTorch 1.1.0 (even if outdated at this point) since every additional library/package is also meant for 2019a. This would keep the PR relatively sane and I can open a new one for 2019b with updated versions.

That's the best of both worlds. Thanks a lot for the effort in getting this done!

[seb45tian]

Should I delete the PyTorch 1.3.1 EC file from this PR? It's not used any longer for the Extension Module, but could be of use in general? Glad to help and learn btw :-)

[terjekv]

I'd zap it to prevent confusion. If people want a newer PyTorch, use a newer toolchain? Also, if I remember correctly, with --update-pr you can use ':' (colon) in front of a file to remove it from the PR.

[seb45tian]

Test report by @seb45tian
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in this PR)
r01n27.palma.wwu - Linux centos linux 7.7.1908, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz, Python 2.7.5
See https://gist.github.com/15124e77ecce60453cb63ede4b62f309 for a full test report.

[terjekv]

Test report by @terjekv
SUCCESS
Build succeeded for 11 out of 11 (4 easyconfigs in this PR)
ninhursaga.uio.no - Linux RHEL 8.1, Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz, Python 3.6.8
See https://gist.github.com/09b314ae800d2cba7b2b046bde481ce0 for a full test report.

[boegel]

Test report by @boegel
FAILED
Build succeeded for 3 out of 5 (4 easyconfigs in this PR)
node2410.golett.os - Linux centos linux 7.7.1908, Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz, Python 2.7.5
See https://gist.github.com/ace16d8fd17fd223ed1e7f2c9952b13d for a full test report.

[boegel]

Test report by @boegel
SUCCESS
Build succeeded for 7 out of 7 (4 easyconfigs in this PR)
generoso - Linux centos linux 7.6.1810, Intel(R) Xeon(R) CPU E5-2660 v4 @ 2.00GHz, Python 3.6.8
See https://gist.github.com/797668185c862ea5226ea8808131528e for a full test report.

[boegel]

Test report by @boegel
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in this PR)
node3122.skitty.os - Linux centos linux 7.7.1908, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz, Python 3.6.8
See https://gist.github.com/3661d5321a53756656f517af0356a402 for a full test report.

[boegel]

@seb45tian I looked into an updated version of this for foss/2019b in #9976, and I made a couple of changes that would make sense here too I think, see also the comments I added here.
I think my approach is slightly better/cleaner, but I'm open to feedback :)

Let me know what you think about the suggestions, and if you have any questions.

[verdurin]

@seb45tian Any update here, notably on the comments by @boegel ?

[seb45tian]

@seb45tian Any update here, notably on the comments by @boegel ?

@verdurin This should be ready to go in. I tried to resolve all the issues boegel mentioned in his comments. Should I have done anything else? Terribly sorry if I delayed this PR, I thought I had done everything I could.

[verdurin]

Thanks @seb45tian

@boegel are you happy with it now?

[surak]

Does the OP still care about this one? Or let's close it?

[seb45tian]

Does the OP still care about this one? Or let's close it?

Isn’t 2019a dropped in the upcoming 5.0 release? Then this could be closed anyway.

[boegel]

Strictly speaking the 2019a toolchain are still supported, only older toolchains are supported (cfr. https://docs.easybuild.io/deprecated-easyconfigs/#deprecated_easyconfigs_toolchains_foss).

That will likely change relatively soon though, since we're looking into being a bit more aggressive w.r.t. how far back we support things, see easybuilders/easybuild#872

With that in mind, I agree closing this makes sense, it doesn't seem worth the effort to pursue this further...