implement test harness

[meganlkm]

Implement the formula-test-harness. I included a travis config if you want to set that up too.

[javierbertoli]

Hi @meganlkm, can you point me to some formulas testing done with formula-test-harness as to see the usage? I followed the link but found no examples.

I'm personally more familiar with the kitchen-ci testing harness, using the kitchen-salt provisioner and inspec, which seem to me simpler to use and inplement. They're currently being used across many fomulas and are being activelly maintained and widely used.

But, that said, I'm totally opened to new tools and ideas, so some examples will let me compare and, perhaps, adopt it 😄

[meganlkm]

Here are a few examples:

https://github.com/meganlkm/gpg-formula
https://github.com/intuitivetechnologygroup/sqlite-formula
https://github.com/saltstack-formulas/uwsgi-formula
https://github.com/intuitivetechnologygroup/credstash-formula (still in dev)
https://github.com/intuitivetechnologygroup/authconfig-formula/pull/44

[meganlkm]

it was also demoed in a recent meetup: https://github.com/intuitivetechnologygroup/saltstack-meetup-2018-03-13

[meganlkm]

There are also no tests currently implemented for this formula. It would be beneficial to have at least a test in place for this as it is a popular formula.

[blbradley]

Hello!

I would suggest setting up Travis and getting the test suite passing for each formula in this ecosystem where this has been proposed before merging.

If you didn't create the formula yourself, you'll have to ask someone on the 'Core' team to enable Travis per formula.

edit: removed 'with'

[javierbertoli]

In the examples above, do I undestand correctly that the harness just runs the formula in a container and checks that the output has no failures? I see no actual tests there (like is the service really running? and listening in the desired port?). Ie, the state modifying the config file can modify the config and produce a file that would let the service run, but would be totally useless. The harness just seems to check the states run OK, but not if they produce the correct results.

Using the tools I mention, implementing the harness here would require a .kitchen.yml file like this one and then tests can be written in files like these. And the tests look like these in travis or locally.

I'd be grateful if you can point me to tests that work in a similar fashion to the ones I point above, to see how to implement them, as that's what I'd like to do in testings.

[simplyadrian]

The test suites are dynamically recognized and run in untainted containers. You can have as many test suites as you need. The example/default test is a pytest however, the test harness is flexible enough to run most other kinds of tests. Additionally, but less important, seeing as this is a python project it seems strange to be using a ruby/chef test harness which is not simple to use nor clearly documented in the salt documentation. (yes, I have seen this https://saltstack.com/testing-your-salt-states-with-kitchen-salt/).

[blbradley]

Personally, I've had great results when using Test Kitchen to test formulae in this ecosystem. In addition, Testinfra can be used to write system level tests in Python.

[javierbertoli]

Additionally, but less important, seeing as this is a python project it seems strange to be using a ruby/chef test harness which is not simple to use nor clearly documented in the salt documentation

@simplyadrian, for what I've seen through the years, the reason they're used is that kitchen-ci is a mature testing harness which lets you use different drivers to run the same tests in different platforms, like docker, vagrant, aws, openstack, etc (a huge plus). Even SaltStack uses it for testing.

For the same reason, Inspec which is a fork of Serverspec is becoming a somewhat de facto standard for testing. Although maintained by the Chef community, is quite powerful for compliance and security testing, which lets you run the tests both locally or remotely, on already running hosts.

Or, as @blbradley says, you can use Testinfra if you want to write your tests in Python.

All these tools are plugable, through drivers, provisioners and verifiers.

a ruby/chef test harness which is not simple to use nor clearly documented

In this point I disagree with you, both are fairly easy to use, and widely documented.

The kitchen-salt provisioner's options are described here.

The test suites are dynamically recognized and run in untainted containers. You can have as many test suites as you need. The example/default test is a pytest however, the test harness is flexible enough to run most other kinds of tests.

Can I use other drivers but docker, ie to test AWS specifics? How can I plug Inspec or Testinfra tests to formula-test-harness? Can you provide me with some examples?

As I said in my original question, I'm not against new tools, nor I'm trying to disrespect or dismiss the formula-test-harness project. I just want some examples, to evaluate the benefits over existing tools and to consider if I jump to this new tool 😄

[meganlkm]

It's an opensource project. Make it so. As previously mentioned, there are currently zero tests for this project.

[aboe76]

@meganlkm I have merged your pr, could you see about the errors?

[aboe76]

@meganlkm can you take a look at why the pytests fail with an assertion error?