VsTest Code Coverage is broken by design

[NKnusperer]

Environment

• Azure DevOps Server 2019 Update 1 Patch 1 (Dev17.M153.3)
• Agent v2.153.2 inside Docker Container based on mcr.microsoft.com/dotnet/framework/runtime:4.8-20190709-windowsservercore-1903
• Microsoft.TestPlatform v16.3.0 installed via VsTestPlatformToolInstaller

Issue Description

With codeCoverageEnabled: true the VsTest task generates a proprietary .coverage file which cannot be consumed by Azure DevOps / PublishCodeCoverageResults nor is it published as an artifact.
The only supported formats are Cobertura and JaCoCo.
Reading the docs for PublishCodeCoverageResults it says :

Tasks such as Visual Studio Test, .NET Core, Ant, Maven, Gulp, Grunt also provide the option to publish code coverage data to the pipeline. If you are using these tasks, you do not need a separate Publish Code Coverage Results task in the pipeline.

https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-code-coverage-results

This is wrong because the VsTest task neither converts the resulting .coverage to a supported file format nor uploads the raw .coverage file as an artifact.

But the story continues...

So if the task itself does not produce a valid output why not simply convert the .coverage file to Cobertura using danielpalme/ReportGenerator and then use PublishCodeCoverageResults ?

Unfortunately this is also not possible because of #6279.
The .coverage file does not exist anymore after a test run so there is nothing we could do with it.
This effectively renders the complete code coverage feature of the VsTest task useless.

Obviously we could use a simple script to execute our tests with enabled code coverage and then do the file conversion, but then we would also loos all the nice features of the VsTest task like distributed batching and flaky test detection.

Proposal

It would be great to get some generic infrastructure to use any code coverage tool which would resolve the issue in the first place.
Because most of the the time these tools works the same (you just run them with an argument telling them what other executable to run) this should be quite simple?

Degenerated example using dotCover:
dotCover.exe cover --TargetExecutable=$(vsTestExecutable) --TargetArguments=$(vsTestArguments) --ReturnTargetExitCode

[anshii03]

@NKnusperer - VsTest Task do upload .coverage file as an artifact. Can you please share logs by setting system.debug = true ?

Alternatively , For a first class view , we are soon going to roll out this the following:

If you are interested in this , we will enable this on your account in couple of days.
You can try this and give us further feedback before this feature becomes GA.

[andrebriggs]

@anshii03 I'm interested in this view

[anshii03]

@andrebriggs - Please share your account details. We will enable it on your account and let you know.

[anshii03]

@NKnusperer - we would love to share this view with you. Please share your account details so that we can enable it on your account as well.

[NKnusperer]

@anshii03 we are using an on-premises installation so guess this is not possible?

[anshii03]

@anshii03 we are using an on-premises installation so guess this is not possible?

Yes , not possible on on-premises.

[anshii03]

@NKnusperer - VsTest Task do upload .coverage file as an artifact. Can you please share logs by setting system.debug = true ?

Alternatively , For a first class view , we are soon going to roll out this the following:

If you are interested in this , we will enable this on your account in couple of days.
You can try this and give us further feedback before this feature becomes GA.

@NKnusperer - You can share the logs by setting system.debug = true . We do upload .coverage file as an artifact.

[NKnusperer]

@anshii03 I cannot do this at the moment, however one more thing to mention is that we are running Unit Tests in parallel using multiple build agents.
Maybe the issue is somehow related to this? Haven't tested it using just a single agent.

[anshii03]

@NKnusperer Publishing multiple coverage reports through publish code coverage task won't work. This is by design as merging of code coverage data is not supported: - https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-code-coverage-results?view=azure-devops#q--a

[anshii03]

VsTest Task do upload .coverage files as an artifact.

And Publish Code Coverage Task don't publish multiple code coverage reports as mentioned in doc https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-code-coverage-results?view=azure-devops#q--a

Alternatively , Private preview is ready for the new first class view :

If anyone is interested , please mail your account details on devops_tools@microsoft.com.
We would love to have your feedback before it becomes GA.

Hence , closing this ticket.

[matiasdd]

We would love to have your feedback before it becomes GA.

@anshii03 Any idea when this becomes generally available?

[MartyIce]

I'm interested and emailed devops_tools@microsoft.com, but received this in response:

[ian-g-holm-intel]

Is this code coverage view going to be made available soon for Azure DevOps Server 2019? It appears to only be available currently for microsoft hsoted Azure DevOps services.

[p00j4]

@anshii03 I'm seeking to enable this in my org too. how can we connect in DM for the details you would need.

[p00j4]

I have sent mail to devops_tools@microsoft.com and still waiting it to be assigned to someone right. can someone please help?

[p00j4]

This issue needs to be Re - Opened because the proposed feature enable to display the html by default never came to light. And every new user of VsTest task integrating on the ADO is facing the same race condition.

• Can't move to powershell to manage whole test lifecycle by self
• And can't use VsTest task because the .coverage is not displayable.

Proposal: why can't codecoverage (from VsTest task) also output the .xml by default? It's upto the use to consume it or not.

[simonachmueller]

@ShreyasRmsft @anshii03 this issue needs to be definitely re-opened, we are facing a similar problem (very well described here microsoft/vstest#981 (comment))

[sathishreddy48]

Yes, we definitely recommend rewriting the code for this tool. However, the issue still persists as it doesn't generate a coverage report, which is essential for checking the code coverage in .NET 6. In my case, I'm using two different unit test tasks: one for differential code coverage and another one for regular code coverage.

Unfortunately, we do not have the option to verify code coverage with the previous build (last committed/merged code coverage) if we select coverageFailOption as the build. This means it's literally comparing with the previous build in the same PR, and if the code coverage is the same as the previous build, it doesn't allow us to merge the code :(