-
Notifications
You must be signed in to change notification settings - Fork 552
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Need to wait until all tests suite complete before failing because of minimum_coverage #232
Comments
Do something like this: running_in_parallel = ENV['PARALLEL_TEST_GROUPS'] && ENV['TEST_ENV_NUMBER']
SimpleCov.at_exit do
result = SimpleCov.result
# Count the number of commas in the command name to figure out how many result groups were combined into this
# result. We only want to do the processing below when the last parallel worker finishes.
if !running_in_parallel || ENV['PARALLEL_TEST_GROUPS'].to_i <= result.command_name.scan(/,/).size + 1
result.format! # Print out coverage result
exit(1) if result.covered_percent < 100.0
end
end |
I'm not using parallel_tests across suites (yet), but your suggestion is definitely giving me ideas on how to implement a workaround for my issue. Thanks! |
It would be cool to have some standardized solution to this kind of thing built into simplecov, without having to resort to defining custom at_exit behaviour. @ronwsmith What did you end up doing? |
Here's what I did: SimpleCov.at_exit do
result = SimpleCov.result
result.format!
simplecov_test_suites = ['minitest', 'rspec']
parallel_offset = ENV['PARALLEL_TEST_GROUPS'] ? ENV['PARALLEL_TEST_GROUPS'].to_i - 1 : 0
minimum_coverage = 97.25
# Count the number of commas in the command name to figure out how many result groups were combined into this result
if result.command_name.scan(/,/).size + 1 >= simplecov_test_suites.size + (parallel_offset * 2) # two parallel suites
# We only want to enforce minimum coverage after all test suites finish
if result.covered_percent.round(2) < minimum_coverage
puts "Coverage percentage has dropped below the minimum threshold of #{minimum_coverage}%"
exit(1)
end
end
end I also have these defined in the # Merge coverage reports for multiple test types
use_merging true
# Merge results within the last 60 minutes. If the tests take longer, this must be extended
merge_timeout 3600 Edit: Note, this assumes I'm always running in parallel, so for an official SimpleCov feature, we should add in what @jshraibman-mdsol recommended above. |
@colszowka Would you mind enabling the wiki so people can share solutions such as these? |
I have three test suites and use the use_merging flag to combine coverage, but the report is generated after each test suite completes as well.
As my test suites are running, the coverage goes up like this:
Unit tests: 84%
Functional tests: 91%
Features: 95%
Currently, I have to have minimum_coverage set to 84%, but I'd like to be able to set it to 95% since that's our total combined coverage.
Is there any way to have simplecov wait to make a run as failed based on minimum_coverage until after the final test suite completes?
The text was updated successfully, but these errors were encountered: