Add Hetzner Cloud driver

[decentral1se]

Closes #1791.

[decentral1se]

How to manual test this:

$ git clone -b add-hetzner-driver https://github.com/ansible/molecule
$ cd molecule
$ pip install -e .'[hetznercloud]'
$ molecule init -r hcloudtest -d hetznercloud && cd hcloudtest
$ export HCLOUD_TOKEN=<MYTOKEN>
$ export MOLECULE_NO_LOG=False; export MOLECULE_DEBUG=True
$ molecule test

[decentral1se]

Current status is then: there are endless issues with running the full functional test suite on the CI. SSH connection issues, machines not responding, docker networks randomly failing, incorrect passwords (really unsure how this is happening). And all the while, all the tests pass locally. I think I will abandon this for now.

[LKaemmerling]

Really sad to hear this. @decentral1se is there something we could do to help you finish this? We plan to add more modules to ansible in the near future and also use molecule internal, so we would like to push this forward :)

[decentral1se]

@LKaemmerling I'm glad Hetzner intend to use Molecule. I don't use Hetzner Cloud services as I did when I first started this work. So, I have no reason to continue to work on it since I won't use it for the forseeable future. Probably some paid staff from Hetzner can pick up this work and fight the CI issues.

[decentral1se]

Also, Hetzner could just pay me.

[LKaemmerling]

@decentral1se would it be possible that you rerun the tests? I guess I have found something. We have set the Temporary Token Service to a max token lifetime of 60min. After this 60min all resources created will be destroyed. This could be a reason because I saw the last run takes ~1h 2 min. (https://travis-ci.com/ansible/molecule/jobs/215355197)

[ssbarnea]

I propose making a fork and enabling travis with hertzner credentials and running on a daily cron, so we can see it works reliably enough.

[LKaemmerling]

I guess i have found the problem.

@decentral1se you said that the tests are working locally. I can confirm this. The tests on the CI fail because of the parallelism from travis.

You have something like the instance name (instance, instance-1 and instance-2) hardcoded. This will cause problems when running the same tests multiple times in parallel.

Because the modules work primarily with the instance name, you will have the following behavior when running the same tests in parallel. Hopefully, I can show it to you with the following sample:

Travis CI:
Job 1 (functional tests on Python 3.7): Started at 10:00
---> Will create server instance
---> Will work with this server
Job 2 (functional tests on Python 2.7): Started at 10:05
--> Will create server instance -->Attention: It will get the Server from Job1! (Because the name need to be unique and the ansible modules follow the idempotency principle)
--> Will work with this server
Job 1: At 10:20
--> All tests are done, will remove the server
--> Finished
Job 2: ERR! The server was removed, no connection (or another error)

As far as I see it, it is not a problem with our Hetzner Cloud, it is rather a problem with the CI and the errors only occur during these tests because we are the first cloud provider where the tests are done in the CI.

[decentral1se]

There's our --parallel functionality and then there is also the Travis CI job parallelization. You've probably nailed the issue here. The solution might then be: isolate all hetzner functional tests and put them in a single job that only runs once (no parallel stuff at all on any layer). A fork is no solution IMHO. I'm not starting this unless @ssbarnea is OK with the proposed direction.

[ssbarnea]

@decentral1se what kind of changes are you expecting? I am already scared about the current complexity related to sharding.

The reality is that we need to make resources unique for each travis job. Mainly the hertz driver should run even when running with "tox -p auto".

[decentral1se]

Perhaps we follow your direction, and a single cron job that runs nightly.

We do this "for now" and can at least see what is failing. Then the infrastructure will be in place when more CI resources are available to really lock it down. We keep the API keys and users to rely on real quality Q&A for the driver by keeping it available to use. Later, we get it running rock solid.

So, TLDR; an additional build in the cron job to run them all in serial execution. I will use a pytest marker to group them. At the end of the day, all the other drivers don't even run, so we'll be just as good as them.

Just thinking out loud ...

PS. Note to safe: some fix attempts over at https://github.com/LKaemmerling/molecule/commits/add-hetzner-driver to watch out for.

[decentral1se]

OK, I've added fixes from @LKaemmerling and enabled the tests on only a single job (due to lack of CI resources. All the tests pass, see https://travis-ci.com/ansible/molecule/jobs/217346476). The CI is very unreliable right now, so I re-running the other jobs which failed for unrelated issues (that's off-topic for this PR). Can I get final reviews and then let's merge this.

[decentral1se]

CI failing due to #2177 (unrelated).

[decentral1se]

OK, final approach is then to #2191 and get a clean CI build here.

[decentral1se]

@ssbarnea let's merge this, it's green 🔥 🔥 🔥