running upstream project tests via kola?

[miabbott]

The ostree and rpm-ostree projects have a robust set of tests that could be used to validate functionality on RHCOS/FCOS. Specifically, the vmcheck tests of rpm-ostree are such that they can be run against a VM which has an rpm-ostree daemon running.

While experimenting running these tests against RHCOS (without using kola), I ran into a few snags that made me question how the integration of the tests into kola might work.

1. The upstream tests can change with each release/commit, so kola would have to perform a checkout of the repo which matches the version of software under test. I didn't find any other examples of this kind of approach in the current codebase. Obviously, this doesn't mean we can't adapt, but I wanted to get a feel if this kind of approach would work out.

2. I found the vmcheck suite has a decent amount of dependencies that need to be installed on the host running the test. kola doesn't really have this problem, as it is just a golang binary after it is all compiled etc. Would this requirement on certain packages get pushed up to a higher layer to handle?

3. The alternative to installing packages on the host running the test would be to pull a container with the required dependencies and execute the tests via the container, via kola. I'm not sure this is a better solution than what would be required for point 2.

4. The extreme alternative is to punt on the upstream tests and re-implement some of the tests natively in 'kola`.

Maybe I'm getting ahead of myself with these kinds of questions, but I figured the folks more experienced with this codebase would be able to steer me in the right direction.

[miabbott]

Oh I forgot to add...I had short discussions with @jlebon about finding a generic way for running these kinds of upstream tests via kola. I'm not convinced about the usefulness of such an effort, since there don't seem to be a lot of projects that we work with that execute their tests the same way (i.e. Fedora STI).

[arithx]

Do we want/need these to be added to kola? I'm not familiar with vmcheck, if it's meant more to test rpm-ostree it might make sense to just add a subset of native tests that target core functionality to make sure that what we have packaged in the image is functional rather than testing all of rpm-ostree.

---

Here's some miscellaneous thoughts I've had on the subject of external tests in kola that don't necessarily directly target any of your points:

Ideally I'd like to avoid adding more vendor bloat to mantle for additional tests unless directly necessary.

Another thing for running non-native kola tests is that we don't have a great way of discovering them and providing accurate information about the test (just stdout, stderr, and exit status). In the case of vmcheck where there's a series of tests it would need it's own test runner that kola just runs.

My personal opinion (if we want to add these into kola) would learn more towards either building subsets of the tests natively (and dealing with the extra maintenance burden of duplicating existing relevant tests in golang) or some sort of hybrid where we accept that the tests will require a git clone or some form of download (could just be pulling a container) to get the relevant test bits and the kola test essentially handles creating the machine, downloading the relevant source bits it needs, and executing it in a simple way (ideally just executing a binary/script that will exit 0 or 1). In the latter case it might be worthwhile to add some sort of flag to enable/disable said tests.

Definitely open to hearing other alternatives.

cc @bgilbert for other opinions

[bgilbert]

We haven't generally had kola tests which wrap other test suites. We probably shouldn't build many tests of that kind, but it might make sense to do so in some cases. Pulling and running a container seems like the obvious way to implement them. I agree that vendoring tests directly into mantle doesn't seem great.

[cgwalters]

if it's meant more to test rpm-ostree it might make sense to just add a subset of native tests that target core functionality to make sure that what we have packaged in the image is functional rather than testing all of rpm-ostree.

Yeah, regardless we should probably at least have a basic invocation of e.g. rpm-ostree --status.

One high level thing to keep in mind here is that since we now have two streams, that makes everything at least...four times more complicated =) Since we have to think about versioning a lot more.

Additional complexity here is that rpm-ostree is used by other projects in Fedora too.

So.....

One approach would be for us to reuse kola's provisioning/teardown bits, and have it write out the ssh connection information (as a ssh-config file e.g.) and then something higher level can reuse that to execute other test suites.

[bgilbert]

One approach would be for us to reuse kola's provisioning/teardown bits, and have it write out the ssh connection information (as a ssh-config file e.g.) and then something higher level can reuse that to execute other test suites.

kola spawn could possibly be adapted for this. I like the idea of writing out an ssh config file.

[cgwalters]

I think this is fixed by #1252