Add new .travis.yml section for beaker-docker tests

[petems]

Although we have PCCI for acceptance testing, since it's fairly easy to setup Beaker to docker tests in Travis CI, might be an idea to add that as a configurable section to the .travis.yml for modules?

matrix:
  include:
    - rvm: '2.1'
      sudo: required
      services: docker
      env: PUPPET_VERSION="3.8.4" BEAKER_set="centos-6-docker"
      script: bundle exec rspec spec/acceptance/class_spec.rb
      bundler_args: --without development
  allow_failures:
    - env: PUPPET_VERSION="3.8.4" BEAKER_set="centos-6-docker"

Add new Beaker SUT:

HOSTS:
  centos-6-x64:
    platform: el-6-x86_64
    hypervisor : docker
    image: centos:6
    docker_preserve_image: true
    docker_cmd: '["/sbin/init"]'
    docker_image_commands:
      - 'yum install -y crontabs tar wget'
CONFIG:
  type: foss
  log_level: debug

[bastelfreak]

👍 I like the idea, why not, can't be bad to have a fallback for PCCI

[rnelson0]

Should be at least 3.8.5 tho, .4 had some issues. Why not ~> 3.8.0 or ~>
3.0 to keep up?

Rob Nelson
rnelson0@gmail.com

[daenney]

I agree with @rnelson0. Our normal matrix specifies ~> 3.0 and ~> 4.0 for Puppet versions. We should stick to the same here and not bind it to specific versions. That's going to be surprising.

[nibalizer]

I think the camptocamp people do it so 👍

[jyaworski]

Agree. We have acceptance tests already, but they're not being run. As in, stuff in spec/acceptance.

[bastelfreak]

We're now the proud owner of https://github.com/voxpupuli/puppet-puppetserver (form camptocamp). They do a lot of cool acceptance testing with .travis.yml and .travis.sh. We should fiddle that out soon because I don't want to kill the tests with a modulesync, but also I want to modulesync.

[daenney]

Most of what happens in travis.sh seems to be related to getting specific versions of Augeas in place and testing with that. For modules that use Augeas this makes sense, but not by a long shot for all modules.

[bastelfreak]

We just discussed voxpupuli/puppet-corosync#276 on IRC (doing beaker-docker for corosync on travis). Acceptance tests are important, already a lot of our modules have some so we should push the usage of them further (I bet that this translation was wrong). I would prefer running the docker stuff on a own platform, not via travis. Travis gives us limited computing power and something like 4? concurrent runs for the whole organization. We are getting more and more modules with more and more pull requests, waiting for travis feedback already takes too long sometimes. I played around with my jenkins VM and tests for one module: https://ci.virtapi.org/view/acceptance%20tests/. I would prefer running beaker-docker on that platform (which I could sponsor) or via PCCI, but not on travis.

[roidelapluie]

I prefer travis as everyone has an overview of the config and understand how it works and is connected to github. Also we do not create maintenance work or unneeded efforts.

[roidelapluie]

By using travis we do not create human bottleneck as well.

[daenney]

I'm not comfortable with having people personally sponsor CI stuff for the project. As @roidelapluie said that creates potential human bottlenecks, especially when shit goes wonky.

Yes we're getting more modules but it's not like we're working on all of them at the same time so I'm not really convinced the resource constraints are that much of an issue. Looking at https://travis-ci.org/voxpupuli we do a couple of builds a day on very few modules in total, I doubt we would hit any problem anytime soon with Travis. And at that point, if people really want to sponsor I would rather they pay for capacity on Travis-CI for us to use than have members host build infrastructure. Once upon a time hosting ourselves was the only option since we couldn't run Beaker on Travis but now that we can I'd rather we spend some money on offloading that to Travis than spend money on building out / maintaining our own CI.

And yes, hours that you put in voluntarily also have a cost.

[3flex]

Thoughts on Puppet's approach? puppetlabs/modulesync_configs@9ca280f

[daenney]

That's pretty much what I would want to do.

[alexjfisher]

I'd like to add simple acceptance tests to puppet-php.

[wyardley]

I think having acceptance tests run is important. Otherwise, we run into the issues we saw with puppet-nginx where the acceptance tests were broken... a lot of folks submitting PRs won't run them locally.

As far as the slowness issues @bastelfreak mentioned, I totally understand that. On the other hand, waiting for Travis to finish has just encouraged me to be better about getting my own test setup working rather than committing to a PR and waiting to see if tests pass.

Is there any way to make Travis abort tests when a commit is superseded by a newer one? Can the acceptance test run only if all other tests have already run and passed? Or is there a way to have acceptance tests run (against master) once per-project daily or something like that?

Lastly, would it make sense to have the Travis tests post to a Slack channel? If having it in #voxpupuli would be too distracting, maybe #voxpupuli-alerts?

[3flex]

puppet-corosync now runs acceptance tests on Travis. I've rebased #230, which makes it easier for other modules to enable acceptance tests, and would enable it for puppet-nginx on the next module sync (since it's already pre-configured assuming #230 would be merged).

To @wyardley's question, you can now run tests once per-project daily thanks to Travis's Cron Jobs support. But this looks to be on top of other Travis runs so doesn't help here, but might be nice for catching issues with gem dependencies sooner after they update and drop support for old Ruby versions.

More useful would be travis_dedup, but that is deprecated - though this nugget is in the README: "Travis will ship this as a feature in Q12017, currently in beta if you ask them."

If a dedup feature is added natively to Travis I think a lot of the concerns raised in this thread would go away. Suggest merging #80 and either asking Travis about this beta feature or enabling it when it's released.