Test on all supported Python versions

[csadorf]

While reviewing our CI configuration I noticed that we are only testing on the oldest and newest supported Python versions:

aiida-core/.github/workflows/ci.yml

Line 94 in d5a706f

python-version: [3.5, 3.8]

Am I reinterpreting this correctly? And if yes, is there a specific reason for this besides saving CI time?

[sphuber]

The only reason is indeed to save CI time. When we were still on Travis running on all python versions was definitely not an option. Now that we migrated to GitHub actions the throughput is a lot higher but if we start to run a lot more builds this may become insufficient. Since for each python version we need to run two builds, one for each ORM backend, we would need to add another 4 jobs. And each of those run more than 10 minutes. I was not sure whether that added cost is worth it just to have slightly more coverage. I feel testing the lower and upper bound should give a really good idea that the middle versions should also work.

[csadorf]

@sphuber Thank's for clarifying. I'd like to recommend that we enable a test on all Python versions at the very least for release branches. The reason is that there a backwards incompatibilities between different Python minor versions, so checking the bounds is insufficient to guarantee that we actually support all versions. And furthermore, it would also be important to check whether our dependencies are available for all minor versions and function as expected.

[greschd]

Does GitHub actions support cron jobs? If so, we could add a nightly build (in addition to release branches) that checks all versions.

[csadorf]

Does GitHub actions support cron jobs? If so, we could add a nightly build (in addition to release branches) that checks all versions.

Looks like you can: https://help.github.com/en/actions/reference/events-that-trigger-workflows#scheduled-events-schedule