Limit to known-good setups from acceptance tests

[smortex]

For acceptance testing, we look into metadata.json for supported versions of Puppet and operating systems, and for each combination try to execute the acceptance test suites.

However, some combination depend on nonexistent packages, typically for a recent version of Puppet on an old Linux distribution, or with an old version of Puppet on a recent Linux distribution.

Instead of letting GitHub compute all possible matrix combinations, rely
on puppet_metadata to build the list of supported combinations.

---

For checking, I opened voxpupuli/puppet-splunk#314: compare with voxpupuli/puppet-splunk#312 which had a bogus Puppet 5 on Ubuntu 20.04 test failing because no such package is available (on a side note, everything was broken and I had to add a commit to fix CI, but this is not related to this change).

---

For checking, I opened voxpupuli/puppet-splunk#315: compare with voxpupuli/puppet-splunk#312 which had a bogus Puppet 5 on Ubuntu 20.04 test failing because no such package is available.

[ekohl]

Last time I tried this, it actually didn't work. Have you tried this? Also, perhaps we can exclude them here:
https://github.com/voxpupuli/puppet_metadata/blob/491aa993479260ef37b9a926ab4b3aa07c1e3fba/lib/puppet_metadata/github_actions.rb#L21-L30

Perhaps with a flag aio_only.

On the other hand: All of these combinations contain at least one that's EOL: either the OS (Ubuntu 14.04, Debian 8) or Puppet (5).

[smortex]

@ekohl yes, I had a hard time figuring out what to put for it to work :-) @bastelfreak also noted on IRC that at least one component was obsolete and that removing such platforms from the supported versions in metadata.json was probably better. Yet, we have some modules which are not actively maintained and can refer to old platforms, and when a contributor wants to fix a bug or send a new feature, they are facing failing test cases. I a few times I created a PR to update metadata.json and asked the maintainer to rebase on top of it, so I though that such a change would avoid this and improve contributors experience.

We also might want to support some legacy systems for some reason, it was discussed for some module not long ago but I can't recall which module 😞 (IFAICR it was about allowing smoother upgrades of legacy infrastructure to recent Puppet & OS)

On the other hand, a brutal failure on a failing combination and some code to update all metadata.json files is also adequate if situation as above are only transient and we want to make our life easier.

[bastelfreak]

I want to point out that having this code merged would make it easier to update modules that still have it in their metadata.json because it keeps the CI pipeline green.

[ekohl]

Still, this requires an update to all modules. I haven't seen a reason why we can't update the generator rather than excluding after the fact. That would instantly update modules without a code change in every module.

[bastelfreak]

Agreed, an update to the generator would be better, but I've no clue how to do that :(

[ghoneycutt]

+1 as this makes it easier for people to contribute. A regular problem we have is that someone contributes and the testing fails with something unrelated to the contribution and merging is held up. This will expedite the process and give contributors a better experience.

[smortex]

After further work, I am glad to announce that I could make puppet_metadata build the combinations of valid OS / puppet version; test it successfully in conditions it was failing; and I updated this PR accordingly.

Converting this PR to draft because it depends on this to be released first:

• Track known-good os and puppet combinations puppet_metadata#21

[smortex]

Rebased on top of #742.

voxpupuli/puppet-splunk#314 was updated accordingly (pathological cases are no more present in master, but they are covered by the unit tests in puppet_metadata).

Ready for review 🎉

[ekohl]

Thanks for all the effort!