Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deprecated test plans remain in test queue and in-progress results are lost when a newer test plan version deprecates an older test plan version #877

Closed
mcking65 opened this issue Dec 14, 2023 · 4 comments
Closed

Deprecated test plans remain in test queue and in-progress results are lost when a newer test plan version deprecates an older test plan version #877

mcking65 opened this issue Dec 14, 2023 · 4 comments
Labels
bug Something isn't working

Comments

@mcking65
Copy link

mcking65 commented Dec 14, 2023 •
edited
Loading

Advancing a new version of a test plan to draft review caused the test queue to contain test runs for a deprecated test plan version and cause the loss of previously completed work on manual test runs.

The procedure that manifest this bug was:

  1. Command button V23.12.06 was in draft review and had three test runs in the test queue.
  2. Results were present for all the runs.
  3. We discovered that a change was needed to the settings specified for two VoiceOver commands.
  4. We made V23.12.13 of the plan with the changes to the VoiceOver commands and merged it to master so it showed on the data management page as R&D Complete.
  5. We advanced V23.12.13 of the command button test plan to draft review.

Expected results:

  1. Since V23.12.06 is deprecated, its runs no longer show in the test queue.
  2. To avoid unnecessary loss of manual test work, equivalent runs of V23.12.13 are added to the test queue, preserving as many test results as possible.

Actual results:

  1. Runs of V23.12.06, which became deprecated, remain in the test queue.
  2. Runs of V23.12.13 that we subsequently added to the test queue were empty, so testers have to start over even for tests that were not changed.

Requirements for data preservation during test plan deprecation

To prevent unnecessary loss or repetition of manual test run work, the system needs to determine which test results can be preserved when a version of a test plan is deprecated as the result of a test plan advancing into draft review, into candidate review, or into recommended.

the system should:

  1. Identify AT/browser preservation candidates: For each AT/browser combination supported by the test plan, identify the AT/browser combinations for which the deprecated plan has results (either draft or final) and the new plan version does not have results. If the new version has results for a given combination, it is not necessary to attempt to copy any results from the prior test plan version to the new test plan version for that AT/browser combination. This comparison can ignore AT and browser versions, e.g., if the new plan version and the deprecated plan version both have results for JAWS/Chrome, the version of JAWS and Chrome are un important. There is no need to copy results for JAWS/Chrome in that scenario.
  2. Identify preservable results: For each AT/browser preservation candidate, identify the test/assertion/command/settings that are equivalent in the new and deprecated plans.
  3. If an AT/Browser preservation candidate has a final report generated from the deprecated test plan and if all results are preservable, copy the final report from the deprecated test plan to the new test plan.
  4. If an AT/Browser preservation candidate has a final report generated from the deprecated test plan and if some but not all results are preservable, then:
    1. copy results for the equivalent tests/assertions/commands into new incomplete test plan runs.
    2. Add the incomplete runs to the test queue; assign to the same testers who previously generated the results.
  5. If an AT/Browser preservation candidate has a run in the test queue and any of its results are preservable, then:
    1. copy results for the equivalent tests/assertions/commands into new incomplete test plan runs.
    2. Add the incomplete runs to the test queue; assign to the same testers who previously generated the results.
    3. Remove the test plan runs for the deprecated test plan version from the test queue.
@mcking65 mcking65 added the bug Something isn't working label Dec 14, 2023
@css-meeting-bot
Copy link
Member

The ARIA-AT Community Group just discussed App gbug impacting manual test running.

The full IRC log of that discussion <jugglinmike> Topic: App gbug impacting manual test running
<jugglinmike> github: https://github.com//issues/877
<jugglinmike> Matt_King: We have the December 6th verison of a test plan in "draft review". We have results for some of those tests. While we were running the test, we found a problem. So we made a new version, and that new version shows up as an "r&d" versino
<jugglinmike> s/versino/version/
<jugglinmike> Matt_King: So we have three active versions of this test plan. We advanced the r&d version to draft review.
<jugglinmike> Matt_King: The december 13th version is now in "Draft" and the december 6th version becomes "deprecated"
<jugglinmike> Matt_King: But the december 6th version is still in the test queue and it has results
<jugglinmike> Matt_King: The December 13th version is not in the test queue at all
<jugglinmike> Matt_King: This is not the situation we want, because if people continue testing, we'll have to throw their work away
<jugglinmike> Matt_King: The bug is that whenever a test plan is in the test queue and we advance a new version of that plan to the same phase as the version that is in the test queue--well, what should happen?
<jugglinmike> Matt_King: In this bug report, I proposed what should happen
<jugglinmike> Matt_King: The first thing is that if a plan becomes deprecated, any runs that exist in the test queue for the deprecated plan should be remoeved from the test queue (not thrown away, just removed)
<jugglinmike> Matt_King: Then, since there were runs, the runs should still be there--we should add a new test run for the new version of the test plan, retaining the metadata from the prior version (e.g. AT version, browser version). Then, for any test for which the results would still be valid, copy the results into the new test plan run
<jugglinmike> Matt_King: Is this what people would expect?
<jugglinmike> IsaDC: That's what I already expected would happen, especially because we didn't change anything for VoiceOver
<jugglinmike> howard-e: I do agree with the scenario that you've outlined. That's what should have happened
<jugglinmike> howard-e: The issue is that I think there was a failed comparison. After these new changes, there's a part of that test which should be excluded during the comparison, but it isn't being excluded
<jugglinmike> Matt_King: Let's say all the tests changes and none of the data is transferrable, should we still do everything else? The deprecated version should definitely be removed from the queue, right?
<jugglinmike> Matt_King: It seems like there are two different kinds of failure here
<jugglinmike> howard-e: If every test changed, then if you transitioned a test plan version to that phase, then every tester would have zero results
<jugglinmike> Matt_King: But the deprecated version should also be replaced by the newer version, right?
<jugglinmike> howard-e: Yes, that's right
<jugglinmike> Matt_King: The new V2 format separates one version of a screen reader with another version very cleanly
<jugglinmike> Matt_King: We have an open issue related to updating the comparison algorithm...
<jugglinmike> Matt_King: There's another scenario, here. Let's say we had marked "final" the JAWS test for this plan.
<jugglinmike> Matt_King: So it's not in the test queue. But now that test plan is deprecated, and we advance a test plan from one phase to the next phase.
<jugglinmike> Matt_King: Should we automatically create a new draft report for any report that was previously marked "final" for that plan?
<jugglinmike> Matt_King: This is a harder question, especially in the world of automation
<jugglinmike> Matt_King: If it's in draft review, maybe this is an easier question to answer
<jugglinmike> IsaDC: I think the data should be copied into the new version, if there are no other changes.
<jugglinmike> Matt_King: If we had made a change to something that affected JAWS, should we not just keep it and also automatically add it to the test queue...?
<jugglinmike> howard-e: Only if it's in draft
<jugglinmike> [the group discusses further edge cases and potential remediations]
<jugglinmike> Matt_King: I'm getting a sense for what we need to do, here. I'll follow up with some more information on the bug report itself
<jugglinmike> Zakim, end the meeting

@howard-e howard-e mentioned this issue Dec 18, 2023
Merged
@lolaodelola
Copy link

@mcking65 This is currently being worked on and I know you wanted to add more information here, please let us know if there's any more info or context you want to include.

@mcking65
Copy link
Author

mcking65 commented Jan 2, 2024

@lolaodelola @howard-e

I finally got the requirements clearly specified in the top comment. Please use those requirements for testing resolution of this bug.

howard-e added a commit that referenced this issue Jan 24, 2024
@howard-e
Update updatePhaseResolver to better align with the deprecation of te…
fb28a2e 
…st results process described in #877; update dataManagement.test.js; update sample populate script to support testing v2 test format tests workflow
@ccanash ccanash mentioned this issue Jan 24, 2024
Closed
alflennik pushed a commit that referenced this issue Jan 29, 2024
@howard-e
Update hashTests function to support the migration of test results …
eaf9109 
…across test plan versions using the v2 test format (#880)

* Update aria.js to account for properties to be ignored during a v2 test format version update so applicable results can be transferred over during phase update

* Recalculate the hashes for already existing v2 items

* Check testPlanVersions.length before updating related test plan reports and runs

* * Remove stray eslint-disable
* Rename testWithNoIds -> testWithOmittedAttributes

* Created util for regenerating the hashedTests and test results; additionally, now deleting found duplicates as there is an error when tested against prod data where unnecessary duplicates were created

* Update migration testFormatVersion migration to use util following rebase

* Avoid showing reports for DEPRECATED test plan versions on Test Queue page

* Update workflow to separate commits for v1-related and v2-related test format tests to be imported

* Update updatePhaseResolver to better align with the deprecation of test results process described in #877; update dataManagement.test.js; update sample populate script to support testing v2 test format tests workflow

* Update comments on test

* Update dataManagement.test.js
@mcking65
Copy link
Author

Tested by advancing link test plan v23.12.13 from draft to candidate. The prior version (v23.12.06) had multiple runs in the test queue and some runs complete. The advancement of the new version went perfectly.

Caveat: I didn't look at all the assertions in the test plan before and after advancing the new version to make 100% sure all equivalent results were kept and 0 verdicts that were associated with changed assertions were not kept. I am counting on the eng team to have tests in place that cover results migration. I primarily focused on the state of the queue and random checking of test plan results migration.

@howard-e howard-e mentioned this issue Feb 26, 2024
Closed
5 tasks
@ccanash ccanash moved this from In staging/sandbox to In production / Completed in MVP support of ARIA-AT Working Mode by ARIA-AT App - 2023 H1 Project Mar 11, 2024
@ccanash ccanash closed this as completed Mar 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
No open projects
MVP support of ARIA-AT Working Mode by ARIA-AT App - 2023 H1 Project
Status: In production / Completed
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mcking65 @lolaodelola @css-meeting-bot @ccanash