-
Notifications
You must be signed in to change notification settings - Fork 131
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Automate the QC tests #166
Comments
See also #109 |
@eclare108213 @dabail10 @duvivier @JFLemieux73 We are looking for input about what other features might be useful to improve the qc validation tool. At this point, it's relatively easy to create qc cases from two sandboxes using the documentation provided here, https://cice-consortium-cice.readthedocs.io/en/latest/user_guide/ug_testing.html#end-to-end-testing-procedure. We also have a script that tests the qc test procedure. What else might be useful? Would it be useful to provide a script that a user could edit that provides the steps to run the qc test? Would it be useful to have an option on cice.setup that would automatically generate a baseline for a qc test from the master? Is the qc test straight-forward enough that what we have is fine? Is the documentation clear? Whatever we do should take into account the fact that the qc test could be used to compare
We can develop some tools/feature that address some or all of these use cases as needed. What's missing or what's unclear? What would be useful? How is the documentation? |
I agree that creating the QC tests manually is intuitive and easy to do, given the instructions that @apcraig linked above. I think my issue with it was not having the instructions at my fingertips. It would be nice to be able to push a "go" button and have it generate the results, but considering the many variations, that's probably too complex to be worth the effort (apologies to @mattdturner who already spent time on it). My recommendation at this point is that we check the user documentation (@duvivier) to make sure that it's clear to users that when they submit a PR, they (and we!) need to check for BFB, and if the changes are not BFB, then do the QC tests and provide documentation of the results. We might already be making that clear, but let's check. Sound okay? |
@apcraig I think the documentation is pretty clear since I just tested it recently. The only thing that is a bit (intentionally?) murky is the following line in the first paragraph: "Exactly what to test is a separate question and depends on the kinds of code changes being made." I know that we're basically saying the tests we need for b4b is different from non-b4b. However, it might be nice to have a very simple and clear description of the procedure for the b4b tests. We already do this for the non-b4b (https://cice-consortium-cice.readthedocs.io/en/master/user_guide/ug_testing.html#code-compliance-test-non-bit-for-bit-validation AND https://cice-consortium-cice.readthedocs.io/en/master/user_guide/ug_testing.html#end-to-end-testing-procedure) I am thinking just another subsection on the test suites section (https://cice-consortium-cice.readthedocs.io/en/master/user_guide/ug_testing.html#test-suites) saying "If you are adding code that is b4b, you should do the following procedure:
IF changes are made, I would prefer a script that the user edits rather than an option in cice.setup. I like scripts because it eliminates some of the errors I can make typing or copying/pasting stuff in at the command line and makes reproducibility of testing a bit easier. This might also address @eclare108213 's issue of not having the instructions at your fingertips since it would be there in the script. But I don't think this is essential. |
@apcraig is this still something we want to do before release? Or is it on hold? |
I think the only thing we might do before the release is review documentation. |
I agree. Let's just make sure the documentation is clear. We can come back to this later if users struggle with it. |
The qc documentation has been updated over several PRs. There has been some review and I believe things are working OK and the documentation is consistent with the implementation. I will close this for now. We can open a new PR if further work is needed. |
There needs to be an automated process to validate the QC script when updates are made. My current thoughts are to have 2 scripts. 1 of the scripts would build and submit 4 cases (
qc
,qc_bfb
,qc_nonbfb
, andqc_fail
; see #161 for more information on the 4 cases), and the other script would then run the QC script in the combinations of cases. This could be done as a single script, but due to the runtime (and queue time) of the QC simulations a script might sit there idle for too long.Additionally, there needs to be some additional thought that goes into the QC script process in order to make it easier for the user to use.
The text was updated successfully, but these errors were encountered: