LP1954865: The sequence of job execution is different between first run and rerun

[beliaev-maksim]

This issue was migrated from https://bugs.launchpad.net/checkbox-ng/+bug/1954865

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2021-12-15 03:23:18	6	High	False

Description

[Summary]
When user reruns the failed jobs, the execution order will lose the original sequence.

[Steps to reproduce]

1. Run checkbox-cli
2. Choose "USB3 test (Manual)" plan and execute it
3. Plug USB 2.0 to make job "usb3/insert" be failed
4. Rerun all cases (usb3/insert, usb3/storage-automated and usb3/remove)

[Expected result]
The order should be

1. usb3/insert
2. usb3/storage-automated
3. usb3/remove

[Actual result]
Rerun sequence will be

1. usb3/insert
2. usb3/remove
3. usb3/storage-automated

[Failure rate]
3/3

Attachments

No attachments

Tags:
[]

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CHECKBOX-1120.

This message was autogenerated

[fernando79513]

I repeated the test and the issue seems to be present still. The order is not kept when a test fails.

[fernando79513]

Initially, we discussed renaming the output file of the insertion test to avoid ordering issues. However, I’m considering a different approach: joining the three separate jobs (insertion, storage, and removal) into one. This would eliminate the ordering problem and also resolve the inconsistent job execution sequences between runs.

Currently, our test plans run all three tests sequentially. We could also create a version that only performs insertion and removal without storage if needed.

The idea is to have these tests:

Insertion + Storage + Removal
Insertion + Removal
(Insertion maybe?)
That will force the insertion test to always be run before the storage and the removal, but that is currently the case, as both of them depend on the insertion test. This would only force them to be run all of them at the same time.