DAOS-16265 test: Add debug for common test dir between variants

[phender]

Help determine contents of common test dir between test variants.

Skip-unit-tests: true
Skip-fault-injection-test: true
Allow-unstable-test: true
Test-tag: EcodFioRebuild

Required-githooks: true

Before requesting gatekeeper:

• Two review approvals and any prior change requests have been resolved.
• Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
• Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
• Commit messages follows the guidelines outlined here.
• Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

• You are the appropriate gatekeeper to be landing the patch.
• The PR has 2 reviews by people familiar with the code, including appropriate owners.
• Githooks were used. If not, request that user install them and check copyright dates.
• Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
• All builds have passed. Check non-required builds for any new compiler warnings.
• Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
• If applicable, the PR has addressed any potential version compatibility issues.
• Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
• Extra checks if forced landing is requested
  • Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
  • No new NLT or valgrind warnings. Check the classic view.
  • Quick-build or Quick-functional is not used.
• Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

[github-actions]

Ticket title is '[12-24]-./erasurecode/rebuild_fio.py:EcodFioRebuild.test_ec_online_rebuild_fio tests fail due to daos_server startup problem.'
Status is 'Open'
Labels: 'ci_master_weekly,md_on_ssd,weekly_test'
https://daosio.atlassian.net/browse/DAOS-16265

[daltonbohning]

test.py has conflicts

[daosbuild1]

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14818/4/execution/node/1008/log

[phender]

Tests passed in

Updated output:

• Initially and until the directory is +90% full it will only display the df output:

2024-08-17 06:30:10,553 test             L0768 INFO | ----------------------------------------------------------------------------------------------------
2024-08-17 06:30:10,553 test             L0769 DEBUG| Common test directory (/var/tmp/daos_testing) contents:
2024-08-17 06:30:10,554 run_utils        L0462 DEBUG| Running on wolf-[51,110-116] with a 120 second timeout: df -h /var/tmp/daos_testing
2024-08-17 06:30:10,780 run_utils        L0336 DEBUG|   wolf-51 (rc=0):
2024-08-17 06:30:10,780 run_utils        L0341 DEBUG|     Filesystem      Size  Used Avail Use% Mounted on
2024-08-17 06:30:10,780 run_utils        L0341 DEBUG|     /dev/sda7        28G   26M   26G   1% /var/tmp
2024-08-17 06:30:10,780 run_utils        L0336 DEBUG|   wolf-110 (rc=0):
2024-08-17 06:30:10,780 run_utils        L0341 DEBUG|     Filesystem      Size  Used Avail Use% Mounted on
2024-08-17 06:30:10,780 run_utils        L0341 DEBUG|     /dev/sda7        28G  1.1G   25G   5% /var/tmp
2024-08-17 06:30:10,780 run_utils        L0336 DEBUG|   wolf-113 (rc=0):
2024-08-17 06:30:10,780 run_utils        L0341 DEBUG|     Filesystem      Size  Used Avail Use% Mounted on
2024-08-17 06:30:10,780 run_utils        L0341 DEBUG|     /dev/sda7        28G  527M   26G   2% /var/tmp
2024-08-17 06:30:10,781 run_utils        L0336 DEBUG|   wolf-114 (rc=0):
2024-08-17 06:30:10,781 run_utils        L0341 DEBUG|     Filesystem      Size  Used Avail Use% Mounted on
2024-08-17 06:30:10,781 run_utils        L0341 DEBUG|     /dev/sda7        28G  341M   26G   2% /var/tmp
2024-08-17 06:30:10,781 run_utils        L0336 DEBUG|   wolf-116 (rc=0):
2024-08-17 06:30:10,781 run_utils        L0341 DEBUG|     Filesystem      Size  Used Avail Use% Mounted on
2024-08-17 06:30:10,781 run_utils        L0341 DEBUG|     /dev/sda7        28G  1.1M   26G   1% /var/tmp
2024-08-17 06:30:10,781 run_utils        L0336 DEBUG|   wolf-111 (rc=0):
2024-08-17 06:30:10,781 run_utils        L0341 DEBUG|     Filesystem      Size  Used Avail Use% Mounted on
2024-08-17 06:30:10,781 run_utils        L0341 DEBUG|     /dev/sda7        28G  378M   26G   2% /var/tmp
2024-08-17 06:30:10,781 run_utils        L0336 DEBUG|   wolf-115 (rc=0):
2024-08-17 06:30:10,781 run_utils        L0341 DEBUG|     Filesystem      Size  Used Avail Use% Mounted on
2024-08-17 06:30:10,781 run_utils        L0341 DEBUG|     /dev/sda7        28G  236M   26G   1% /var/tmp
2024-08-17 06:30:10,781 run_utils        L0336 DEBUG|   wolf-112 (rc=0):
2024-08-17 06:30:10,781 run_utils        L0341 DEBUG|     Filesystem      Size  Used Avail Use% Mounted on
2024-08-17 06:30:10,781 run_utils        L0341 DEBUG|     /dev/sda7        28G  312M   26G   2% /var/tmp
2024-08-17 06:30:10,782 test             L0775 INFO | ----------------------------------------------------------------------------------------------------

[daosbuild1]

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14818/6/execution/node/922/log