mantle: a general way of globally specifying channel or version

[dongsupark]

Since the docker 1.12 profile was removed in Edge, many kola tests started to fail to run, because of the missing docker 1.12 profile. Ideally we should be able to make a different set of profiles for each channel. Unfortunately the current Jenkins pipeline is not able to do that. It takes into account only flatcar-master branch.

For now we work around the specific issue by skipping tests for Edge. flatcar/mantle#61.

In the long run, however, in mantle, we should make a general way of globally specifying channel or version numbers, so that individual tests can selectively run based on the channel or versions.

[t-lo]

Had a quick sync with @sayanchowdhury on this - a good way to achieve the objective would be to use separate mantle branches for Alpha, Beta, Stable and Edge, respectively.

[pothos]

That won't work because we only build mantle once in Jenkins and then it is used for various channels. I don't understand why I would need to recompile kola each time I want to run a test for a different version.

[pothos]

The tests in kola are already annotated with distros and platforms they run on. Why not make another annotation for the channel? The original idea here was to have this checked at runtime by looking at the contents of the image. But we can even simplify this and introduce a command line parameter for the channel and have the test filter handle this before the tests are run.

[t-lo]

I guess it depends on the actual amount of differences between any given channel (e.g. Edge vs. Stable). Both approaches have pros and cons.

I agree that distinguishing at the test execution level has the benefit of implementing coverage for all channels in the same file / branch.

On the other hand, if we need to update tests to cover new features, using separate channels would enable us to transition updated tests through Alpha -> Beta -> Stable alongside the actual updated packages.

[pothos]

Besides the efforts to reorganize how Jenkins runs mantle, it is also error-prone when running kola manually if people need to be aware of rebuilding each time with the correct branch.

[pothos]

Here you can see how tests are annotated to run only on cl (Container Linux) and not on a certain platform: https://github.com/flatcar-linux/mantle/blob/flatcar-master/kola/tests/docker/torcx_docker_flag.go#L55 We could use the same scheme to define channels or even versions as requirements for a test.

[sayanchowdhury]

I did not mean create separate branches rather create an argument --channel which would help us achieve the same.