DAOS-16027 test: Adding daos_test REBUILD31-34 subtests

[phender]

Adding rebuild tests currently not executed by doas_test/rebuild.py.

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium-md-on-ssd: false
Test-tag: test_rebuild_31 test_rebuild_32 test_rebuild_33 test_rebuild_34

Required-githooks: true

Before requesting gatekeeper:

• Two review approvals and any prior change requests have been resolved.
• Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
• Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
• Commit messages follows the guidelines outlined here.
• Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

• You are the appropriate gatekeeper to be landing the patch.
• The PR has 2 reviews by people familiar with the code, including appropriate owners.
• Githooks were used. If not, request that user install them and check copyright dates.
• Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
• All builds have passed. Check non-required builds for any new compiler warnings.
• Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
• If applicable, the PR has addressed any potential version compatibility issues.
• Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
• Extra checks if forced landing is requested
  • Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
  • No new NLT or valgrind warnings. Check the classic view.
  • Quick-build or Quick-functional is not used.
• Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

[github-actions]

Ticket title is 'daos_test -r cases REBUILD31-REBUILD35 are not tested in functional testing'
Status is 'In Review'
Labels: 'scrubbed_2.8,triaged,testp2'
https://daosio.atlassian.net/browse/DAOS-16027

[daltonbohning]

Do we really want to add another 4 * 800s timeout (Almost an hour) to pr?

[daltonbohning]

FYI Test-tag wrapped

Test-tag: test_rebuild_31 test_rebuild_32 test_rebuild_33
test_rebuild_34

[daosbuild1]

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14584/1/execution/node/916/log

[daosbuild1]

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14584/1/execution/node/962/log

[kccain]

Unfortunately test_rebuild_31 currently hangs. I discussed with @liuxuezhao and that failure is related to some overall rebuild error case handling that needs refinement, https://daosio.atlassian.net/browse/DAOS-15847.

He gave me an idea to try with test_rebuild_31 using a different type of fault injection. I can check that out separately. In the meantime, maybe we could (if the tests are able to pass) enable tests 32-35.

[phender]

I've added skip list entries for test_rebuild_31 and test_rebuild_32.

[daosbuild1]

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14584/2/execution/node/916/log

[daosbuild1]

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14584/2/execution/node/1014/log

[daosbuild1]

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14584/3/execution/node/916/log

[daosbuild1]

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14584/3/execution/node/1014/log

[phender]

Skip list entries added for the new tests:

[['DAOS-16575', 'test_method_name', 'test_rebuild_31']]
[['DAOS-16576', 'test_method_name', 'test_rebuild_32']]

[daltonbohning]

Can we put these in daily instead of pr? this adds 4 tests * 800s each. Which is almost 1 hour for every pr

[phender]

Can we put these in daily instead of pr? this adds 4 tests * 800s each. Which is almost 1 hour for every pr

@kccain thoughts?

[phender]

Rebuild 31-32 skipped and Rebuild 33-34 passed in https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14584/4/testReport/FTEST_daos_test/DaosCoreTestRebuild/ taking ~5.5 minutes per stage:

2024-09-16 20:32:28,369 test             L0530 INFO | START 1-./daos_test/rebuild.py:DaosCoreTestRebuild.test_rebuild_31;run-agent_config-transport_config-daos_tests-args-daos_test-pools_created-stopped_ranks-test_name-dmg-hosts-pool-server_config-engines-0-storage-0-1-timeouts-4b1a
2024-09-16 20:32:28,378 test             L0471 INFO | ==> Step 1: tearDown(): Called after test completion (test timeout: 1590s, elapsed: 0.01s, remaining: 1589.99s) [elapsed since last step: 0.01s]
2024-09-16 20:32:28,754 test             L0530 INFO | START 2-./daos_test/rebuild.py:DaosCoreTestRebuild.test_rebuild_32;run-agent_config-transport_config-daos_tests-args-daos_test-pools_created-stopped_ranks-test_name-dmg-hosts-pool-server_config-engines-0-storage-0-1-timeouts-4b1a
2024-09-16 20:32:28,762 test             L0471 INFO | ==> Step 1: tearDown(): Called after test completion (test timeout: 1590s, elapsed: 0.01s, remaining: 1589.99s) [elapsed since last step: 0.01s]
2024-09-16 20:32:29,139 test             L0530 INFO | START 3-./daos_test/rebuild.py:DaosCoreTestRebuild.test_rebuild_33;run-agent_config-transport_config-daos_tests-args-daos_test-pools_created-stopped_ranks-test_name-dmg-hosts-pool-server_config-engines-0-storage-0-1-timeouts-4b1a
2024-09-16 20:32:29,563 test             L0471 INFO | ==> Step 1: setUp(): Starting servers [elapsed since last step: 0.43s]
2024-09-16 20:33:20,315 test             L0471 INFO | ==> Step 2: setUp(): Starting agents [elapsed since last step: 50.75s]
2024-09-16 20:33:25,183 test             L0471 INFO | ==> Step 3: setUp(): Setup complete [elapsed since last step: 4.87s]
2024-09-16 20:34:54,204 test             L0471 INFO | ==> Step 4: tearDown(): Called after test completion (test timeout: 290s, elapsed: 145.06s, remaining: 144.94s) [elapsed since last step: 89.02s]
2024-09-16 20:34:57,237 test             L0939 INFO | PASS 3-./daos_test/rebuild.py:DaosCoreTestRebuild.test_rebuild_33;run-agent_config-transport_config-daos_tests-args-daos_test-pools_created-stopped_ranks-test_name-dmg-hosts-pool-server_config-engines-0-storage-0-1-timeouts-4b1a
2024-09-16 20:34:57,406 test             L0530 INFO | START 4-./daos_test/rebuild.py:DaosCoreTestRebuild.test_rebuild_34;run-agent_config-transport_config-daos_tests-args-daos_test-pools_created-stopped_ranks-test_name-dmg-hosts-pool-server_config-engines-0-storage-0-1-timeouts-4b1a
2024-09-16 20:34:57,841 test             L0471 INFO | ==> Step 1: setUp(): Starting servers [elapsed since last step: 0.44s]
2024-09-16 20:35:01,320 test             L0471 INFO | ==> Step 2: setUp(): Starting agents [elapsed since last step: 3.48s]
2024-09-16 20:35:01,806 test             L0471 INFO | ==> Step 3: setUp(): Destroying any existing pools before the test [elapsed since last step: 0.49s]
2024-09-16 20:35:03,002 test             L0471 INFO | ==> Step 4: setUp(): Setup complete [elapsed since last step: 1.20s]
2024-09-16 20:37:48,141 test             L0471 INFO | ==> Step 5: tearDown(): Called after test completion (test timeout: 1590s, elapsed: 170.73s, remaining: 1419.27s) [elapsed since last step: 165.14s]
2024-09-16 20:37:51,175 test             L0939 INFO | PASS 4-./daos_test/rebuild.py:DaosCoreTestRebuild.test_rebuild_34;run-agent_config-transport_config-daos_tests-args-daos_test-pools_created-stopped_ranks-test_name-dmg-hosts-pool-server_config-engines-0-storage-0-1-timeouts-4b1a

[phender]

Tests passing/skipping as expected in https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14584/5/testReport/FTEST_daos_test/DaosCoreTestRebuild/