Improve CI running time

[abikouo]

SUMMARY

Molecule default scenario will be split into multiple github jobs based on the tags.
We are a separate job for the lint. This could be merged once the following is merge into molecule repository: ansible/molecule#3166

ISSUE TYPE

• Feature Pull Request

[codecov]

Codecov Report

Merging #153 (0c091a1) into main (abcc3e8) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #153   +/-   ##
=======================================
  Coverage   24.02%   24.02%           
=======================================
  Files           1        1           
  Lines         154      154           
  Branches       29       29           
=======================================
  Hits           37       37           
- Misses        111      112    +1     
+ Partials        6        5    -1     

Impacted Files	Coverage Δ
...ections/kubernetes/core/plugins/action/k8s_info.py	24.02% <0.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update abcc3e8...0c091a1. Read the comment docs.

[gravesm]

Thanks for the fix upstream in molecule. My only suggestion would be that we might want to group some of the tags together rather than having everything run separately. That's a lot of checks. Probably quite a few of the faster k8s tags can all run together in the same time it takes the helm tests to run.

[abikouo]

Thanks for the fix upstream in molecule. My only suggestion would be that we might want to group some of the tags together rather than having everything run separately. That's a lot of checks. Probably quite a few of the faster k8s tags can all run together in the same time it takes the helm tests to run.

Yes I will do, after the first run I will make some time analysis to determine which can be grouped

[tadeboro]

I know no one asked for my opinion, but here it is anyway ;) Is there a reason why there is only one molecule scenario used to test the collection? From my experience, messing with tags is really error-prone and can quickly lead to some really hard-to-debug issues.

We use molecule a lot and we learned (the hard way, as always) that having tests split over multiple scenarios works best. For example, in the sensu.sensu_go collection I help maintain, we have around 40 scenarios: one for each module pair (module + module_info) and at least one scenario for each role we provide. This way, we can easily run only part of the test suite.

And to keep the test times (relatively) short, we then distribute the scenarios across a few executors. At the start, we statically distributed those scenarios into different groups based on some local timing information and things worked really nice even then. So maybe that would be something this collection can also use? If there is interest, I can provide some links to our setup at that time.

(And if you are really interested in some funky CI/CD automation, our current setup is probably the place to go since we use pytest to run and time our scenario executions. This timing is then used to automatically partition scenarios in future CI/CD runs. CircleCI is really awesome in this respect.)

[gravesm]

@tadeboro Thanks for the advice! I would say the reason we are using one scenario is mostly historical. @abikouo did have a PR that split things out into multiple scenarios, though using tags seems like less work. We have not seen any issues yet, but I'm sure you have more experience with molecule than I do. I'd be interested in seeing what you've done, particularly around balancing scenarios across a test run.

[abikouo]

no more relevant, will use a new strategy as we migrate to zuul