Merging parallel test results

[kramerdog]

This is less of an issue and more of a request for guidance...

I have seen other threads on parallel testing and merging results etc such as

#147

but none of them seem to quite address the same situation.

I have a Rails 4.1.6 app with SImpleCov 0.9.1 which I am testing using Cucumber and Capybara

Because of the size of the test suite we have developed a parallel test runner which does the following:

• creates 4 separate 'environments' (workers) which includes a full copy of the code and test suite and its own database (worker pool)
• submits each test feature in turn to the next available worker with the 'bundle exec cucumber ' syntax

In effect therefore each feature is executed as a separate 'test run', each of which, under normal circumstances generates its own coverage report just for that feature.

I think that, with a suitable merge_timeout setting, at least all of the output for the same worker process would be 'auto-merged'?

Of course in reality I am looking for a single coverage report across the entire test suite and am looking for tips on how to do this.

The main challenges are:

1. Each test feature produces its own .resultset.json file - easy to collect these up and concatenate them together but dont know how to merge them together
2. Resultset output contains full path names to source files. Because of the different worker environments, the same real file is reported multiple times, one for each full path. Easy to strip out the worker-specific path names but again I dont know how to merge together the multiple results.

Anyway, any suggestions would be much appreciated. Thanks.

[bf4]

Have you looked at issues such as

• https://github.com/colszowka/simplecov/pull/282/files#diff-04c6e90faac2675aa89e2176d2eec7d8R373
• Create a STANDARD_RESPONSES file for common issues #340
• Question: Is it possible to manually merge multiple resultset.json's and generate output? #219

I don't know what you've tried, but I think you should be using a .simplecov file in your project root and configure the command_name and possibly set TEST_ENV_NUMBER

e.g.

  # Use multiple 'command_names' to differentiate reports being merged in together
  command_name "rails_app_#{$$}" # $$ is the processid
  merge_timeout 3600 # 1 hour

otherwise you might want to write your own formatter and post-process the output.

[kramerdog]

Hi Ben

Thanks very much for your response.

I don't think that I am fully 'getting' the command_name thing. After
reading various posts regarding this I had concluded that this was a means
of obtaining a single report split into different sections for each
'command' eg 'rspec' and 'cucumber'. I couldnt see how to apply this to
'parallel tests' which are essentially the same 'command' running over the
same code, just divided into different processes. Am I missing something
here?

However, #219 looks to be exactly what I am looking for and I hadn't
stumbled upon this to date (wrong keyword obviously). Is this April 2013
approach still the best way to go or is there now a more elegant 'inbuilt'
solution?

Trying this the #219 approach will be my next step anyway....

Cheers

Craig

On Sat, Nov 15, 2014 at 8:12 PM, Benjamin Fleischer <
notifications@github.com> wrote:

Have you looked at issues such as

https://github.com/colszowka/simplecov/pull/282/files#diff-04c6e90faac2675aa89e2176d2eec7d8R373

• Create a STANDARD_RESPONSES file for common issues #340 Create a STANDARD_RESPONSES file for common issues #340
• Question: Is it possible to manually merge multiple resultset.json's and generate output? #219 Question: Is it possible to manually merge multiple resultset.json's and generate output? #219

I don't know what you've tried, but I think you should be using a
.simplecov file in your project root and configure the command_name and
possibly set TEST_ENV_NUMBER
https://github.com/colszowka/simplecov/blob/master/lib/simplecov/command_guesser.rb#L19

e.g.

Use multiple 'command_names' to differentiate reports being merged in together

command_name "rails_app_#{$$}" # $$ is the processid
merge_timeout 3600 # 1 hour

otherwise you might want to write your own formatter and post-process the
output.

—
Reply to this email directly or view it on GitHub
#350 (comment).

[bf4]

command_name is a way of identifying distinct runs. Hence command_name "rails_app_#{$$}" Will always be different for each process.

[josephks]

If you use "rails_app_#{$$}" remember to blow away your coverage directory between runs.

[kramerdog]

Thanks for the info.

If you use the command_name approach therefore, am I right in thinking that
the distinct runs, while appearing in the same report, do not actually have
their results merged together and appear as distinct runs? Havent tried it
yet obviously!

On Sun, Nov 16, 2014 at 3:08 PM, Joseph Shraibman notifications@github.com
wrote:

If you use "rails_app_#{$$}" remember to blow away your coverage
directory between runs.

—
Reply to this email directly or view it on GitHub
#350 (comment).

[bf4]

@kramerdog No, the whole point of giving each process a different name is to reconcile them for one report. Checks the docs for 'merge_timeout'

[bf4]

@kramerdog is this still an issue?

[bf4]

Closing due to lack of activity. Feel free to reopen with new info

[chrishough]

I am running into this same issue with parallel tests now. How was this resolved @josephks ?

[colszowka]

Please give some more context as to what exactly the issue you are running into is, and how the answers suggested in this thread do not solve the problem. Thanks!

[chrishough]

Sure thing, here is an example of running the specs without parallel.

...............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
....................................................................................................................................................................................................
............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... ...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................


If you need more of the backtrace for any of these deprecations to
identify where to make the necessary changes, you can configure
`config.raise_errors_for_deprecations!`, and it will turn the
deprecation warnings into errors, giving you the full backtrace.

1 deprecation warning total

Finished in 70 minutes 58 seconds (files took 27.15 seconds to load)
2394 examples, 0 failures

Coverage report generated for RSpec to /Users/chrishough/Code/WeaveUp/weaveup-test-runner/tmp/coverage. 9085 / 11763 LOC (77.23%) covered.

Here is an example of the numbers after a parallel run

1 deprecation warning total

Finished in 14 minutes 42 seconds (files took 22.24 seconds to load)
217 examples, 0 failures

..............Coverage report generated for (2/10), (3/10), (4/10), (5/10), (7/10), (9/10) to /Users/chrishough/Code/WeaveUp/weaveup-test-runner/tmp/coverage. 7101 / 9945 LOC (71.4%) covered.
..............

Finished in 10 minutes 46 seconds (files took 4 minutes 32.8 seconds to load)
240 examples, 0 failures

Coverage (72.81%) is below the expected minimum coverage (77.00%).
.....Coverage report generated for (1/10), (2/10), (3/10), (4/10), (5/10), (7/10), (9/10) to /Users/chrishough/Code/WeaveUp/weaveup-test-runner/tmp/coverage. 7840 / 10660 LOC (73.55%) covered.
.........................

Finished in 11 minutes 59 seconds (files took 3 minutes 33.5 seconds to load)
245 examples, 0 failures

.Coverage (73.55%) is below the expected minimum coverage (77.00%).
..Coverage report generated for (1/10), (2/10), (3/10), (4/10), (5/10), (7/10), (8/10), (9/10) to /Users/chrishough/Code/WeaveUp/weaveup-test-runner/tmp/coverage. 8366 / 11173 LOC (74.88%) covered.
...Coverage (74.88%) is below the expected minimum coverage (77.00%).
....................................Coverage report generated for (2/10), (3/10), (4/10), (5/10), (7/10), (9/10) to /Users/chrishough/Code/WeaveUp/weaveup-test-runner/tmp/coverage. 7515 / 10322 LOC (72.81%) covered.
..........Coverage (74.88%) is below the expected minimum coverage (77.00%).
............................................


If you need more of the backtrace for any of these deprecations to
identify where to make the necessary changes, you can configure
`config.raise_errors_for_deprecations!`, and it will turn the
deprecation warnings into errors, giving you the full backtrace.

1 deprecation warning total

Finished in 13 minutes 26 seconds (files took 2 minutes 33.4 seconds to load)
254 examples, 0 failures

Coverage report generated for (1/10), (2/10), (3/10), (4/10), (5/10), (6/10), (7/10), (8/10), (9/10) to /Users/chrishough/Code/WeaveUp/weaveup-test-runner/tmp/coverage. 8871 / 11602 LOC (76.46%) covered.
.Coverage (76.46%) is below the expected minimum coverage (77.00%).
....................

Finished in 15 minutes 34 seconds (files took 32.64 seconds to load)
250 examples, 0 failures

Coverage report generated for (1/10), (10/10), (2/10), (3/10), (4/10), (5/10), (6/10), (7/10), (8/10), (9/10) to /Users/chrishough/Code/WeaveUp/weaveup-test-runner/tmp/coverage. 9086 / 11763 LOC (77.24%) covered.

2394 examples, 0 failures

Took 1160 seconds (19:20)
Tests Failed
Coverage report generated for (1/10), (10/10), (2/10), (3/10), (4/10), (5/10), (6/10), (7/10), (8/10), (9/10), RSpec to /Users/chrishough/Code/WeaveUp/weaveup-test-runner/tmp/coverage. 9086 / 11763 LOC (77.24%) covered.

[chrishough]

I removed a good portion of the warnings, however, you can see the coverage is ~ 77% on the first run, and averages much lower on the second. Thoughts on how to make this report a cumulative average?

[colszowka]

The section in the readme about command_name should have you covered, you will need to identify the distinct test suite runs by number: https://github.com/colszowka/simplecov/blob/master/README.md#test-suite-names

[chrishough]

If anyone else comes across this post, this is how we fixed it...

# NOTE: Example Build from research below...
# https://github.com/colszowka/simplecov/issues/64
# https://github.com/increments/simplecov-parallel
# https://github.com/increments/simplecov-parallel/blob/master/spec/spec_helper.rb

require 'simplecov/parallel'

SimpleCov::Parallel.activate

SimpleCov.merge_timeout 3600

SimpleCov.minimum_coverage 85

SimpleCov.formatters = SimpleCov::Formatter::MultiFormatter.new([
  SimpleCov::Formatter::HTMLFormatter
])

SimpleCov.start do
  coverage_dir 'tmp/coverage'

  add_filter '/.bundle/'
  add_filter '/spec/'
  add_filter '/config/initializers'
  add_filter '/docs/'

  add_group 'Config', 'config'
  add_group 'Controllers', 'app/controllers'
  add_group 'Helpers', 'app/helpers'
  add_group 'Ignored Code' do |src_file|
    File.readlines(src_file.filename).grep(/:nocov:/).any?
  end
  add_group 'Integrations', 'lib/integration'
  add_group 'Lib', 'lib'
  add_group "Long Files" do |src_file|
    src_file.lines.count > 300
  end
  add_group 'Models', 'app/models'
  add_group 'Modules', 'lib/modules'
  add_group 'Providers', 'lib/provider'
  add_group 'Serializers', 'app/serializers'
end

if ENV['CODECLIMATE_REPO_TOKEN']
  require 'codeclimate-test-reporter'
  CodeClimate::TestReporter.start
end

We used https://github.com/increments/simplecov-parallel gem.

[bf4]

@chrishough sounds like the problem was that you're on CircleCI which doesn't have a shared file system between nodes. Thanks for the gem reference. I haven't seen it before. Implementation looks nice.

[chrishough]

We are using Jenkins actually. Do you want me to setup a PR for this for the docs here for other people too?

[bf4]

a PR would be a good place to discuss, if nothing else see discussion in #558 (comment)

[PragTob]

@chrishough this is weird. Is the build running within a single JenkinsCI worker/server? We are doing just this at work and have no problem with the coverage whatsoever as each one has their own work space and they are on the same file system writing to the JSON file usually works just fine :)

[chrishough]

@PragTob what is your config setup? I tried a number of the options posted in the readme. Only the gem I posted worked correctly. Thoughts?

[PragTob]

@chrishough well we have a fairly standard setup. Jenkins wise it's just a server that ahs workers which will eventually execute our build script which calls rake parallel:spec with 4 processes.

.simplecov is roughly this:

require "simplecov-rcov"

SimpleCov.formatters = [SimpleCov::Formatter::HTMLFormatter,
                        SimpleCov::Formatter::RcovFormatter]

SimpleCov.merge_timeout 3600

SimpleCov.start :rails do
  add_filter ".gems"
  add_filter "spec"
  add_filter "config/initializers"
  add_filter "features"
  add_filter "stuff"
end

We then define the correct command names in spec_helper/env.rb like: `

if Object.const_defined? "SimpleCov"
  SimpleCov.command_name "specs:" + (ENV["TEST_ENV_NUMBER"] || "")
end

if Object.const_defined? "SimpleCov"
  SimpleCov.command_name "features:" + (ENV["TEST_ENV_NUMBER"] || "")
end

Now that I see it we should probably use fetch but hey sometimes old habits die hard :)

[chrishough]

@PragTob interesting, we are not using those other commands. As a followup question have you ever encountered missing files in the simplecov report after setting up parallel execution?

[PragTob]

@chrishough the dommand names are pretty important for making parallel tests work correctly. Sure We have encountered missing files, due to various reasons, some of them being dependencies that hardcore load stuff, missing test results and for a time when our merge_timeout was set to too little. + when we introduced bootsnap and didn't configure it correctly :D Ever since then, no - just checked and coverage was consistent for all the builds we keep around (~60)

[chrishough]

@PragTob are you or is anyone here working with Code Climate? Have you gotten your test results to show up accurately there ?