Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

acceptance tests with new(ish) Beaker version #882

Closed
wyardley opened this issue Sep 22, 2016 · 6 comments
Closed

acceptance tests with new(ish) Beaker version #882

wyardley opened this issue Sep 22, 2016 · 6 comments

Comments

@wyardley
Copy link
Collaborator

wyardley commented Sep 22, 2016 •
edited
Loading

  • Puppet: 4.6.2
  • Ruby: 2.3.1p112
  • Distribution: Mac OSX
  • Module version: [master]

How to reproduce (e.g Puppet code you use)

run bundle exec rake acceptance

What are you seeing

/usr/local/lib/ruby/gems/2.3.0/gems/beaker-2.51.0/lib/beaker/dsl/helpers/host_helpers.rb:81:inblock in on': DSL method on can only be called with a String or Beaker::Command object as the command parameter, not NilClass. (ArgumentError)`

What behaviour did you expect instead

acceptance tests to run

Output log

/usr/local/lib/ruby/gems/2.3.0/gems/beaker-2.51.0/lib/beaker/dsl/helpers/host_helpers.rb:81:in `block in on': DSL method `on` can only be called with a String or Beaker::Command object as the command parameter, not NilClass. (ArgumentError)
    from /usr/local/lib/ruby/gems/2.3.0/gems/beaker-2.51.0/lib/beaker/shared/host_manager.rb:127:in `run_block_on'
    from /usr/local/lib/ruby/gems/2.3.0/gems/beaker-2.51.0/lib/beaker/dsl/patterns.rb:37:in `block_on'
    from /usr/local/lib/ruby/gems/2.3.0/gems/beaker-2.51.0/lib/beaker/dsl/helpers/host_helpers.rb:63:in `on'
    from /Users/wby/git/puppet-nginx/spec/spec_helper_acceptance.rb:5:in `block in <top (required)>'
    from /Users/wby/git/puppet-nginx/spec/spec_helper_acceptance.rb:3:in `each'
    from /Users/wby/git/puppet-nginx/spec/spec_helper_acceptance.rb:3:in `<top (required)>'
    from /Users/wby/git/puppet-nginx/spec/acceptance/class_spec.rb:1:in `require'
    from /Users/wby/git/puppet-nginx/spec/acceptance/class_spec.rb:1:in `<top (required)>'
    from /usr/local/lib/ruby/gems/2.3.0/gems/rspec-core-3.5.3/lib/rspec/core/configuration.rb:1435:in `load'
    from /usr/local/lib/ruby/gems/2.3.0/gems/rspec-core-3.5.3/lib/rspec/core/configuration.rb:1435:in `block in load_spec_files'
    from /usr/local/lib/ruby/gems/2.3.0/gems/rspec-core-3.5.3/lib/rspec/core/configuration.rb:1433:in `each'
    from /usr/local/lib/ruby/gems/2.3.0/gems/rspec-core-3.5.3/lib/rspec/core/configuration.rb:1433:in `load_spec_files'
    from /usr/local/lib/ruby/gems/2.3.0/gems/rspec-core-3.5.3/lib/rspec/core/runner.rb:100:in `setup'
    from /usr/local/lib/ruby/gems/2.3.0/gems/rspec-core-3.5.3/lib/rspec/core/runner.rb:86:in `run'
    from /usr/local/lib/ruby/gems/2.3.0/gems/rspec-core-3.5.3/lib/rspec/core/runner.rb:71:in `run'
    from /usr/local/lib/ruby/gems/2.3.0/gems/rspec-core-3.5.3/lib/rspec/core/runner.rb:45:in `invoke'
    from /usr/local/lib/ruby/gems/2.3.0/gems/rspec-core-3.5.3/exe/rspec:4:in `<main>'
/usr/local/Cellar/ruby/2.3.1/bin/ruby -I/usr/local/lib/ruby/gems/2.3.0/gems/rspec-core-3.5.3/lib:/usr/local/lib/ruby/gems/2.3.0/gems/rspec-support-3.5.0/lib /usr/local/lib/ruby/gems/2.3.0/gems/rspec-core-3.5.3/exe/rspec spec/acceptance failed

Any additional information you'd like to impart

I set export BEAKER_VERSION=2.32.1 and ran bundle update beaker, and this seemed to fix it.
Maybe related to:
https://tickets.puppetlabs.com/browse/BKR-716
but not knowledgeable enough in this area to figure out exactly what changes or what combination of deps would allow this to work on a newer beaker version.
@wyardley wyardley changed the title acceptance tests with acceptance tests with new(ish) Beaker version Sep 22, 2016
@3flex
Copy link
Contributor

3flex commented Sep 22, 2016 •
edited
Loading

From the beaker issue, looks like you can just replace on host, shell(...) with on host, ...`` (though I haven't tried this myself)

That appears in two files, spec/spec_helper_acceptance.rb and spec/acceptance/nginx_vhost_spec.rb

I wonder how long they've been broken for... the acceptance tests need a lot of TLC because I don't think anyone's run them since I wrote them :(

@wyardley
Copy link
Collaborator Author

@3flex: Am I missing it in nginx_vhost_spec.rb? I did grep for on host before, though stupidly didn't look in spec/spec_helper_acceptance.rb itself. I will try to fix that and make a separate PR for it.

@wyardley
Copy link
Collaborator Author

wyardley commented Sep 23, 2016 •
edited
Loading

So, I made those changes (#883), but still getting this error (it's odd, because it seems to be in the install_puppet block still, but that part does seem to run as expected, and the syntax is right).

Also, presumably we should peg the Beaker version in the Gemfile, but right now it's done in this way, not sure how to simplify that, but maybe should just peg to beaker version > 2.35.0 or so (around when that change was introduced)?

group :system_tests do
  if beaker_version = ENV['BEAKER_VERSION']
    gem 'beaker', *location_for(beaker_version)
  end
  if beaker_rspec_version = ENV['BEAKER_RSPEC_VERSION']
    gem 'beaker-rspec', *location_for(beaker_rspec_version)
  else
    gem 'beaker-rspec',  :require => false
  end
  gem 'beaker-puppet_install_helper',  :require => false
end

Traceback:

ubuntu-server-1404-x64 executed in 0.46 seconds

ubuntu-server-1404-x64 10:56:54$ echo '' >> /etc/puppet/hiera.yaml

ubuntu-server-1404-x64 executed in 0.00 seconds
/usr/local/lib/ruby/gems/2.3.0/gems/beaker-2.51.0/lib/beaker/dsl/helpers/host_helpers.rb:81:in `block in on': DSL method `on` can only be called with a String or Beaker::Command object as the command parameter, not NilClass. (ArgumentError)
    from /usr/local/lib/ruby/gems/2.3.0/gems/beaker-2.51.0/lib/beaker/shared/host_manager.rb:127:in `run_block_on'
    from /usr/local/lib/ruby/gems/2.3.0/gems/beaker-2.51.0/lib/beaker/dsl/patterns.rb:37:in `block_on'
    from /usr/local/lib/ruby/gems/2.3.0/gems/beaker-2.51.0/lib/beaker/dsl/helpers/host_helpers.rb:63:in `on'
    from /Users/wby/git/puppet-nginx/spec/spec_helper_acceptance.rb:5:in `block in <top (required)>'
    from /Users/wby/git/puppet-nginx/spec/spec_helper_acceptance.rb:3:in `each'
    from /Users/wby/git/puppet-nginx/spec/spec_helper_acceptance.rb:3:in `<top (required)>'
    from /Users/wby/git/puppet-nginx/spec/acceptance/class_spec.rb:1:in `require'
    from /Users/wby/git/puppet-nginx/spec/acceptance/class_spec.rb:1:in `<top (required)>'
    from /usr/local/lib/ruby/gems/2.3.0/gems/rspec-core-3.5.3/lib/rspec/core/configuration.rb:1435:in `load'
    from /usr/local/lib/ruby/gems/2.3.0/gems/rspec-core-3.5.3/lib/rspec/core/configuration.rb:1435:in `block in load_spec_files'
    from /usr/local/lib/ruby/gems/2.3.0/gems/rspec-core-3.5.3/lib/rspec/core/configuration.rb:1433:in `each'
    from /usr/local/lib/ruby/gems/2.3.0/gems/rspec-core-3.5.3/lib/rspec/core/configuration.rb:1433:in `load_spec_files'
    from /usr/local/lib/ruby/gems/2.3.0/gems/rspec-core-3.5.3/lib/rspec/core/runner.rb:100:in `setup'
    from /usr/local/lib/ruby/gems/2.3.0/gems/rspec-core-3.5.3/lib/rspec/core/runner.rb:86:in `run'
    from /usr/local/lib/ruby/gems/2.3.0/gems/rspec-core-3.5.3/lib/rspec/core/runner.rb:71:in `run'
    from /usr/local/lib/ruby/gems/2.3.0/gems/rspec-core-3.5.3/lib/rspec/core/runner.rb:45:in `invoke'
    from /usr/local/lib/ruby/gems/2.3.0/gems/rspec-core-3.5.3/exe/rspec:4:in `<main>'
/usr/local/Cellar/ruby/2.3.1/bin/ruby -I/usr/local/lib/ruby/gems/2.3.0/gems/rspec-core-3.5.3/lib:/usr/local/lib/ruby/gems/2.3.0/gems/rspec-support-3.5.0/lib /usr/local/lib/ruby/gems/2.3.0/gems/rspec-core-3.5.3/exe/rspec spec/acceptance failed

Also, to get them running in Travis, do we need it setup to do Docker?

@3flex
Copy link
Contributor

3flex commented Sep 23, 2016

I don't think these will run as is on Travis, because they currently use Vagrant for tests.

Best would be to add Docker support in voxpupuli/modulesync_configs repo, probably based on Puppet's work (puppetlabs/modulesync_configs#58), and then leverage that for this module's acceptance tests. puppetlabs-apache module has working Docker tests in Travis running under Beaker, might be worth poking around their repo to see how it's configured.

That would be a huge win. Allows other cool stuff like possibly integrating https://github.com/ssllabs/ssllabs-scan to automatically test default configuration for broken SSL settings.

@wyardley
Copy link
Collaborator Author

It's definitely the install_puppet part that's the problem, even though it's the exact code documented in beaker-rspec's README. voxpupuli/beaker-rspec#85

#883 fixes the problem for me but, I still haven't addressed the requirement of beaker-rspec version in the Gemfile... can someone with more Ruby experience than me suggest how to change the existing code in and around:
https://github.com/wyardley/puppet-nginx/blob/master/Gemfile#L40
in a way that makes some sense?

@wyardley wyardley mentioned this issue Sep 23, 2016
Merged
@3flex
Copy link
Contributor

3flex commented Sep 24, 2016

I think the Gemfile looks good already, since as you said when you opened this issue if you set the BEAKER_VERSION environment variable you can manipulate the version that gets installed.

To your point, yes, this probably should default to a known-good beaker version so people can easily run tests on their own machines, but I think the defaults around system tests in the Gemfile are designed to back testing on Travis rather than testing on their own boxes. I also think the intention would be to have the tests successfully run on the latest beaker version rather than pinning to an older version.

I might be wrong about this, I'd suggest asking on IRC or Slack about the intentions behind it, but IMHO the current Gemfile is adequate.

@3flex 3flex closed this as completed in 48d37c3 Sep 24, 2016
darken99 pushed a commit to darken99/puppet-nginx that referenced this issue Oct 13, 2016
fix for issue voxpupuli#882, PR voxpupuli#883.
04b72d2
cegeka-jenkins pushed a commit to cegeka/puppet-nginx that referenced this issue Sep 13, 2019
fix for issue voxpupuli#882, PR voxpupuli#883.
5a10b40
Rubueno pushed a commit to Rubueno/puppet-nginx that referenced this issue Oct 19, 2020
fix for issue voxpupuli#882, PR voxpupuli#883.
d2824c1
Rubueno pushed a commit to Rubueno/puppet-nginx that referenced this issue Oct 19, 2020
@3flex
Merge pull request voxpupuli#883 from wyardley/feature_fix_acceptance…
b524393 
…_tests

Fix acceptance test failures with newer Beaker versions

Fixes voxpupuli#882
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@3flex @wyardley