Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mutation testing: Automated github action to generate issue #2114

Closed
3 tasks
ValarDragon opened this issue Jul 18, 2022 · 3 comments · Fixed by #2204
Closed
3 tasks

mutation testing: Automated github action to generate issue #2114

ValarDragon opened this issue Jul 18, 2022 · 3 comments · Fixed by #2204

Comments

@ValarDragon
Copy link
Member

ValarDragon commented Jul 18, 2022

Background

We should automate the routine mutation testing issue creation (cref #2069 #2105 ).

This should be pretty straightforward:

  • Make github action that runs once per week
  • It runs make test-mutation
  • Takes mutation_test_result.txt, and creates a github issue. (with some formatting. Perhaps only parse out failures, and split them up with a number and separate code block)

The mutation test issue should also include next steps. (Going from the workflow noted before):

  • We then manually "group" these issues and sort into buckets:
    {errors that are legit}
    {errors that are noise}

Then for legit bucket, further sort errors legitimate errors by testing strategy that will hit it, and make new issues for that testing strategy into follow-up issues. These should be solved within the week, or otherwise be looped into a larger plan.

For the noise bucket, make a new github issue detailing the noise error type. We should aim to get a fork of gomuttest / try disabling noise / finding a strategy to mitigate noise.

@stackman27
Copy link
Contributor

stackman27 commented Jul 19, 2022

@ValarDragon should i create individual issues or put all the failed cases in one issue like you did here?

Also how would i parse through .txt file in .yml file? have we done this elsewhere?

@p0mvn p0mvn added this to the Mutation Testing milestone Jul 19, 2022
@stackman27
Copy link
Contributor

So i've thought about two approaches for formatting mutation_test_result.txt to generate github issues

  1. modify mutation-test.sh to generate a mutation_test_result_fails.tx and create github issue
  2. Run a python script within mutation-test.yml file that formats the mutation_test_result_fails.tx (probably with regex) and generate issues

@ValarDragon
Copy link
Member Author

I'd probably just write a python script for formatting it (And do something more structured than a regex)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

3 participants