Support executing one test multiple times #1614

kkaarreell · 2022-10-19T18:22:27Z

When a test case is run multiple times in a test plan, the resulting junit will contain just a single <testcase> record with a consolidated log in <system-out> element.
Instead, separate <testcase> record should be created for each test case run.

The following warning should be added to the documentation as part of the fixing pull request:

Using cwd for storing test files is not a good idea, especially for multihost testing, use TMT_TEST_DATA instead.

The text was updated successfully, but these errors were encountered:

lukaszachy · 2022-10-21T10:54:18Z

Could you paste reproducer please? I've tried

 /plan:
  execute:
    how: tmt
  discover:
    how: fmf
    test:
    - /test
    - /test
  provision:
    how: local

/test:
  test: echo

but resulting junit.xml contains two testcases (tried with 1.18.0)

kkaarreell · 2022-10-21T11:09:21Z

@lukaszachy
I am observing it in Packit CI on GitHub.
Plan is https://github.com/RedHat-SP-Security/keylime-tests/blob/main/plans/upstream-keylime-tests-github-ci.fmf
and the log is
https://artifacts.dev.testing-farm.io/87e413da-578a-47b9-bc06-a13ba2b0aace/
/functional/basic-attestation-on-localhost is the test run multiple times.

lukaszachy · 2022-10-21T11:44:05Z

Pardon my ignorance but where is the produced junit? https://artifacts.dev.testing-farm.io/7f36ff62-a883-4f2f-ba6b-215c840a47bb/work-rust-keylime-testsEeTL_a/ shows that 'report -h display' is run, not junit.

Either way, tmt has problem to create correct multiple results for same test. But for some reason display (and junit) report correct number as far as I can tell.

kkaarreell · 2022-10-21T12:16:59Z

I am sorry, I have accidentally pasted wrong link for logs.. fixing now.
There is results-junit.xml which is AFAIK junit from tmt post-processed by Testing Farm. I don't know if original junit is available. Of course, it is also possible that it is that post-processing introducing this issue.

lukaszachy · 2022-10-21T12:29:15Z

I can't find dedicated issue for tmt's incomplete results.yaml so I'd use this one for it.
See https://artifacts.dev.testing-farm.io/87e413da-578a-47b9-bc06-a13ba2b0aace/work-upstream-keylime-tests-github-ciXsKbgX/plans/upstream-keylime-tests-github-ci/execute/results.yaml - unfortunately it's a dictionary.

We have to change that.

lukaszachy · 2022-10-21T12:31:36Z

This isn't problem for junit (and other report plugins). The issue is in what they are served (even though it is correct number of results in most cases it certainly isn't correct results)

lukaszachy · 2022-10-21T12:36:29Z

Modified reproducer:

/plan:
  execute:
    how: tmt
  discover:
    how: fmf
    test:
    - /test
    - /test
  provision:
    how: container

/test:
  test: |
    test -e this && exit 0 ;
    touch this && exit 1

This is produced junit.xml (from report -h junit):

<?xml version="1.0" ?>
<testsuites disabled="0" errors="0" failures="1" tests="2" time="0.0">
	<testsuite disabled="0" errors="0" failures="1" name="/plan" skipped="0" tests="2" time="0">
		<testcase name="/test">
			<failure type="failure" message="fail"/>
		</testcase>
		<testcase name="/test"/>
	</testsuite>
</testsuites>

But problem is when one reads results.yaml, only one results is there:

/test:
    result: pass
    log:
      - data/test/output.txt
    duration: 00:00:00

So if one regenerates report from saved rundir (tmt run --last report):

/plan
    report
        status: done
        summary: 1 test passed

psss · 2023-01-27T10:35:08Z

Summary from the hacking session and multihost discussion:

Assign a unique identifier for each test execution during the discover phase
Use test name with a suffix to allow easy log investigation (tab completion in workdir)
Convert the tests.yaml and results.yaml files format from dict to a list of dictionaries

Example tests.yaml file created by the discover step:

- identifier: /tests/area/feature/001
  name: /tests/area/feature
  summary: This is a test for this area
  test: ./test.sh
  path: /default-0/tests/tests/area/feature
  id: af994876-1c68-49a7-90e8-c8d2b189189d
  ...

- identifier: /tests/area/feature/002
  name: /tests/area/feature
  summary: This is a test for this area
  test: ./test.sh
  path: /default-0/tests/tests/area/feature
  id: af994876-1c68-49a7-90e8-c8d2b189189d
  ...

- identifier: /tests/area/feature/003
  name: /tests/area/feature
  summary: This is a test for this area
  test: ./test.sh
  path: /default-0/tests/tests/area/feature
  id: af994876-1c68-49a7-90e8-c8d2b189189d
  ...

Example results.yaml file created by the execute step:

- identifier: /tests/area/feature/01
  name: /tests/area/feature
  result: pass
  duration: 00:00:00
  log:
    - data/test/output.txt

- identifier: /tests/area/feature/02
  name: /tests/area/feature
  result: pass
  duration: 00:00:00
  log:
    - data/test/output.txt

- identifier: /tests/area/feature/03
  name: /tests/area/feature
  result: pass
  duration: 00:00:00
  log:
    - data/test/output.txt

Not sure about the name of the new identifier key. It is an identifier, but id is already used for the core attribute. It is also a test-log-dir, or test-data-dir or something like test-case-run-id. Don't know. Any better ideas?

happz · 2023-01-27T11:05:18Z

results.yaml lacks the guest-the-test-ran-on info, e.g.

- identifier: /tests/area/feature/01
  name: /tests/area/feature
  result: pass
  duration: 00:00:00
  log:
    - data/test/output.txt
  guest:  # as an object? I like namespaces, guest_name and guest_role would look weird to me...
    name: foo
    role: client
    # Any more fields? `guest` with the IP/hostname?

Not sure about the name of the new identifier key. It is an identifier, but id is already used for the core attribute. It is also a test-log-dir, or test-data-dir or something like test-case-run-id. Don't know. Any better ideas?

Right, identifier does not look right to me, too. It's how the test instance is identified among its peers within the context of what, one phase? phase-id or step-id smell like they are id of a phase/step rather than the test within a phase... Focusing on the data directory nature of this key makes sense to me, data-dir wouldn't be bad, as it's sort of exactly this, but there might be other things we might derive from this id, so it's probably a wrong direction and "id" should be preferred. serial-number? :)

kkaarreell · 2023-01-27T11:19:31Z

Yes, serial-number wiki definition matches precisely.

* drop `ResultData`, it's no longer needed and `Result` can handle all the work; * add `guest` key as proposed in [1]; * add schema for validation of (custom) `results.yaml`; * `execute` step saves `results.yaml` as a list of results rather than a mapping, to make allow one test being executed multiple times. See [2]. [1] #1614 (comment) [2] #1614

happz · 2023-02-15T16:54:27Z

Submitted #1854 which turns tests.yaml into a list - this seems to be trivial enough for me to do, there are no serial-number or path changes involved. Those can be added on top of this change independently.

* drop `ResultData`, it's no longer needed and `Result` can handle all the work; * add `guest` key as proposed in [1]; * add schema for validation of (custom) `results.yaml`; * `execute` step saves `results.yaml` as a list of results rather than a mapping, to make allow one test being executed multiple times. See [2]. [1] #1614 (comment) [2] #1614

* drop `ResultData`, it's no longer needed and `Result` can handle all the work; * add `guest` key as proposed in [1]; * add schema for validation of (custom) `results.yaml` (the schema is not applied yet); * `execute` step saves `results.yaml` as a list of results rather than a mapping, to make allow one test being executed multiple times. See [2]. [1] #1614 (comment) [2] #1614

happz · 2023-04-02T21:14:26Z

I believe this issue can be closed now, as both results.yaml and tests.yaml support tests as a list rather than a dictionary.

psss · 2023-04-04T07:30:06Z

Agreed, but let's add at least a basic test coverage checking that: #1965

kkaarreell mentioned this issue Oct 21, 2022

Test phases of a beakerlib test not counted properly when scheduling a single test multiple times #1615

Closed

lukaszachy changed the title ~~Having test case multiple times in a plan results in a single testcase record in junit~~ results.yaml cannot store repeated test execution Oct 21, 2022

lukaszachy added step | execute Stuff related to the execute step breakage Change will change current behaviour labels Oct 21, 2022

psss added this to the 1.22 milestone Jan 25, 2023

psss changed the title ~~results.yaml cannot store repeated test execution~~ Support executing one test multiple times Jan 27, 2023

psss added the step | discover Stuff related to the discover step label Jan 27, 2023

happz mentioned this issue Feb 10, 2023

Simplify the Result class implementation #1845

Merged

happz mentioned this issue Feb 27, 2023

Assign a data path and serial number to each test in discover #1876

Merged

psss closed this as completed Apr 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support executing one test multiple times #1614

Support executing one test multiple times #1614

kkaarreell commented Oct 19, 2022 •

edited by psss

Loading

lukaszachy commented Oct 21, 2022

kkaarreell commented Oct 21, 2022 •

edited

Loading

lukaszachy commented Oct 21, 2022

kkaarreell commented Oct 21, 2022 •

edited

Loading

lukaszachy commented Oct 21, 2022

lukaszachy commented Oct 21, 2022

lukaszachy commented Oct 21, 2022

psss commented Jan 27, 2023

happz commented Jan 27, 2023

kkaarreell commented Jan 27, 2023

happz commented Feb 15, 2023

happz commented Apr 2, 2023

psss commented Apr 4, 2023

Support executing one test multiple times #1614

Support executing one test multiple times #1614

Comments

kkaarreell commented Oct 19, 2022 • edited by psss Loading

lukaszachy commented Oct 21, 2022

kkaarreell commented Oct 21, 2022 • edited Loading

lukaszachy commented Oct 21, 2022

kkaarreell commented Oct 21, 2022 • edited Loading

lukaszachy commented Oct 21, 2022

lukaszachy commented Oct 21, 2022

lukaszachy commented Oct 21, 2022

psss commented Jan 27, 2023

happz commented Jan 27, 2023

kkaarreell commented Jan 27, 2023

happz commented Feb 15, 2023

happz commented Apr 2, 2023

psss commented Apr 4, 2023

kkaarreell commented Oct 19, 2022 •

edited by psss

Loading

kkaarreell commented Oct 21, 2022 •

edited

Loading

kkaarreell commented Oct 21, 2022 •

edited

Loading