unittest "test run" reports wrong test case count

[jayaranigarg]

Steps to reproduce

Write a unittest function with one test case data input

[DataTestMethod]
[DataRow(1)]
public void YouTest(int value)
{...}

When you run the test using

dotnet test

the output will be:
...
Starting test execution, please wait...
Total tests: 2. Passed: 2. Failed: 0. Skipped: 0. Test Run Successful.

Expected behavior

Actually it's only one test here, [DataTestMethod] is not a test.

Actual behavior

2 tests passed

Environment

MSTest 1.3.2

Note- This happened after we added support for hierarchical results #417
Fix - It would perhaps be best to say both in the CLI output - x tests and x+n test results.

[parrainc]

Also happening when marked the unit test function with [TestMethod] attribute.
Somehow the issue seems to be related with the DataRow attribute.

dotnet test
...

Starting test execution, please wait...

Total tests: 2. Passed: 2. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 1.4402 Seconds

Checking to see if I can figured out what's going on to provide a fix for this.

[jayaranigarg]

@parrainc : In case of data-driven tests, there is a parent result for all the aggregated test results. Currently, this parent result is also counted as test and hence a mismatch. So it would be better if we could clearly display that there are 'x tests and x+n test results' in our output.

[parrainc]

Totally agree @jayaranigarg . It's way better if in the output it says what you point out instead of the actual count.
I'm looking at the code to provide some changes for this. Will keep commenting here if questions comes up or when have something to show.

[parrainc]

@jayaranigarg did a change in UpdateResultsWithParentInfo method, removing the line 423, so the parent result is not added anymore in the results list. Even though this broke some of the tests, it's now behaving as it should in the example provided:

dotnet test
...
Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 1.4402 Seconds

Also, planning to add a property to the TestResult (something like ParentResult) for either holding the parent result that was being passed as a test or the count of parent test results (every inner result will hold its parent result).

My question is, where should I send those values in order to display them in the CLI output (kind of lost here)?

[jayaranigarg]

@parrainc : We will come back with an update on this. Making this change has other implications on our system and we will have to investigate that. Thank you for your patience.

[cltshivash]

This change is needed to ensure that the data driven tests work fine in different CI scenarios like distribution and rerun and is intentional.