Not able to render Code Coverage results in the "Code Coverage" tab in Azure DevOps

[pradudev]

I have a classic build pipeline in Azure DevOps for that builds and run tests for a .Net Core 3.1 app
I am using self-hosted build agents running Windows Server 2019 OS

Below is the pipeline screenshot

My tasks:

1. Restore Nuget
2. Build the solution in release mode
3. Install latest vstest.console.exe tool
4. Run tests using above tool and generate .coverage file
5. Install CodeCoverage.exe from Nuget
6. PowerShell script to convert .coverage to .xml
7. Using ReportGenerator task to generate HTML and Cobertura (HTMLInline;HTMLChart;Cobertura)
8. Publish the code coverage results to the server using Cobertura.xml from step 7 and also uploading HTML report folder

Build runs successfully
Publish Code Coverage task log

Build run summary

Published files by the build

Code Coverage tab

Issue: The code coverage tab does not display the HTML reports

[S121Pradhan]

@pradudev Can you share the .runsettings file ?

[Caiptain1]

@pradudev I had the same thing. I had reached out to support for ADO but they weren't very helpful. I basically drove the full process and figured out the cause of this.

The findings

I came to the conclusion that if the VsTest task uploads the test results, your custom report being uploaded is ignored. When the VsTest task is uploading the test report to ADO, the coverage is an attachment to that and automatically gets uploaded. Since the pipeline already has code coverage registered it doesn't accept anything else. Since ADO doesn't merge the coverage files it kind of makes sense but then you can only publish the coverage once in the whole pipeline.

The workaround

• Run vstest from a powershell script
• Copy or move the test results file to a different location
• Publish the test results file (since the results have a relative path reference to code coverage, the publish task won't find it and you'll have no code coverage published)
• Convert the .coverage file to HTML report
• Publish code coverage

[Caiptain1]

P.S. Microsoft support said they won't do anything about it unless it gets community support on their developer communities. There was an idea raised some time ago to bring back the .coverage conversion directly in ADO: https://developercommunity.visualstudio.com/idea/366142/support-vstest-coverage-code-coverage-build-result.html
I asked some friends and colleagues for some support to the idea and it finally passed the threshold and got onto the roadmap for ADO.

[h0p3zZ]

I have tried a lot to solve his problem...

So first of all, the problem is the test task of the pipeline (in my case the DotNetCoreCLI@2 task). If this task collects the code coverage it is automatically published with the tests in the .coverage format. For some reason this push of the .coverage file is at the end of the pipeline (or job - this I do not know) and therefore overrides all the previously uploaded files.

The Solution

Use a Coverage Tool like OpenCover and collect the code coverage with that instead of the collection via the test task with CodeCoverage collection on.

Example:
OpenCover.Console.exe -target:"dotnet.exe" -targetArgs:"test {solution.sln}" -output:"{outputDir}\OpenCover.xml" {-oldstyle}

Afterwards convert the OpenCover file via the ReportGenerator task into Cobertura format.

And after that you can publish it as usual via the PublishCodeCoverageResult task.

You might have to try arround a little but with that solution in mind it will work.

[elninoisback]

Your issue is with Step 4. => "Run tests using above tool and generate .coverage file"

If a .coverage file is created and left alone, Azure DevOps will pick it up and use it over anything else you create later. So basically your coverage.xml or any generated HTML report is ignored at the Code Coverage tab.

So instead of using VSTest task and whatever you used beneath that, try adding a .Net Core Test to your pipeline and configure it to spit out "XPlat Code Coverage" using coverlet collector. Make sure NOT TO pass in any .runsettings file via --settings. Before doing any of that, you will need to add the NuGet coverlet.collector for all your test projects.

Then add your task to publish the coverage results. You don't need the report generator, it's now built-in.

That's all. Hopefully, you will be able to see the generated HTML getting rendered under Code Coverage tab.

[kenneth-westbrook]

Thank you @elninoisback your guide was VERY helpful! Got my stuff to publish after following your example.

[gioce90]

There is another way to achieve this (the Full coverage on the "Code coverage" tab on the build), without using coverlet.

Now thanks to microsoft/vstest#981 (comment)) is possible to use dotnet test with cobertura format:

  - task: DotNetCoreCLI@2
    displayName: 'DotNetCoreCLI Test with report (cobertura format)'
    inputs:
      command: test
      projects: '**/**.Tests.csproj' # if multiple test projects are to be executed, create a dirs.proj and use it here. That will ensure the tests are executed in parallel
      arguments: '--no-restore --collect "Code Coverage" --logger trx --results-directory "TestResults/Coverage/" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura'
      publishTestResults: true

You can also use publishTestResults: false instead, followed by reportgenerator, PublishTestResults and PublishCodeCoverageResults (with variable disable.coverage.autogenerate: 'true') . It depends of your preference on the reports.

This is good for the full coverage. For the diff coverage on the pull request instead, saddly you have to generate .coverage file. Actually I have two different pipelines (one for the full coverage as showed, the second for the diff coverage)

[MS-Manikandan]

• results

Hi @pradudev,
Could you let us know, do you able to convert the .coverage file to xml file, like the below command in powershell task

Powershell Task Inline command to convert to xml file

$coverageFile="$(Common.TestResultsDirectory)$(build.buildnumber)-$(build.buildid).coverage"
$xmlFile="$(Common.TestResultsDirectory)$(build.buildnumber)-$(build.buildid).xml"
&"$env:USERPROFILE.nuget\packages\microsoft.codecoverage\17.0.0\build\netstandard1.0\CodeCoverage\CodeCoverage.exe" analyze /output:$xmlFile $coverageFile
I used the above command in local it works fine, in the pipelines the coverage file exists and and I can execute the codeCoverage.exe, but I couldn't able to execute the above command altogether since it failed for no reason and I couldn't find why its failing only on the pipelines, have you faced this issue and could you share your workaround for this.

[fhnaseer]

@Manikandanmani You can also use dotnet-coverage cli tool to convert coverage files. https://docs.microsoft.com/en-us/dotnet/core/additional-tools/dotnet-coverage#dotnet-coverage-merge
dotnet-coverage merge -f xml sample.coverage

[MS-Manikandan]

@fhnaseer , Thanks for the info I'll try that.

[MS-Manikandan]

@fhnaseer, I could able to convert into xml file, but this file is generated under userprofile, also I have an Invalid file issue when I downloaded the coverage file by using the below powershell script, but if I directly download the same coverage file from the codecoverage tab in the build, the convertion works. could you take a look on this.

In this below picture,the coverage file downloaded are for the same build, not sure about the size difference.

1. The First File starts with 20220304.6...1968 is manually downloaded.
2. The second File with Name 9746 is downloaded by invoking a PS Script.

$UriOrga = "https://dev.azure.com/$($OrganizationName)/$($Project)/" 
$uriAccount = $UriOrga + "_apis/test/codecoverage?buildId=9746&flags=1&api-version=6.0-preview.1"
$json=Invoke-RestMethod -Uri $uriAccount -Method get -Headers $AzureDevOpsAuthenicationHeader

$downloadedFile="$env:userprofile\Downloads\9746.coverage"

Invoke-WebRequest $json.psobject.properties["value"].Value.codeCoverageFileUrl -OutFile $downloadedFile

[fhnaseer]

@Manikandanmani You can also provide output path in command line [-o|--output ]. Regarding downloading with ps scritp, I am not sure about it. But it seems like 9746.coverage file is not valid.

[simonachmueller]

@nehsharmMS mentioned here that MS has new preview option to enable for Code Coverage, does it only require .coverage file? Then coverlet etc. would be not needed.

[MS-Manikandan]

@nehsharmMS mentioned here that MS has new preview option to enable for Code Coverage, does it only require .coverage file? Then coverlet etc. would be not needed.

@simonachmueller, are you meaning to say that the .coverage file generated in the pipelines would also provides individual coverage metrics for each file covered ?, btw is this can be achieved in vstest.console.exe [ships with visual studio] or using dotnet test cli? correct me if i am wrong here.

[MS-Manikandan]

@Manikandanmani You can also provide output path in command line [-o|--output ]. Regarding downloading with ps scritp, I am not sure about it. But it seems like 9746.coverage file is not valid.

@fhnaseer , I found out the reason why the downloaded file was corrupted ,because I forgot to add the authentication as headers when downloading the coverage file, after adding the headers downloads the coverage file without any issues

Invoke-RestMethod -Uri $json.psobject.properties["value"].Value.codeCoverageFileUrl Get -ContentType "application/json" -Headers @{Authorization=("Basic {0}" -f $AzureDevOpsAuthenicationHeader)} -OutFile $downloadedFile

[simonachmueller]

@nehsharmMS mentioned here that MS has new preview option to enable for Code Coverage, does it only require .coverage file? Then coverlet etc. would be not needed.

@simonachmueller, are you meaning to say that the .coverage file generated in the pipelines would also provides individual coverage metrics for each file covered ?, btw is this can be achieved in vstest.console.exe [ships with visual studio] or using dotnet test cli? correct me if i am wrong here.

As I understood, there is a preview feature on Azure DevOps to use .coverage file to display code coverage in the build results without needing to generate html results separately.

[gioce90]

@simonachmueller As I understood, there is a preview feature on Azure DevOps to use .coverage file to display code coverage in the build results without needing to generate html results separately.

I never ear that. Is true?

[simonachmueller]

@gioce90 nope, unfortunately fake news from MS (seems like they tested it, but did not release).

[skotian2]

Is this issue resolved with the mentioned steps above ?

[gioce90]

@skotian2 is resolved only if you launch a task that use directly the cobertura format, as I described previously.
If you generate .coverage files (as mentioned in the first comment on this thread) you will not be able to see the report in the "Code coverage" tab (is a known issue already specified in this thread):

If a .coverage file is created and left alone, Azure DevOps will pick it up and use it over anything else you create later. So basically your coverage.xml or any generated HTML report is ignored at the Code Coverage tab.

But as solution you can:

• collect coverage with coverlet or
• collect coverage with the native Microsoft code coverage method, but specifying "cobertura" format. I show how here (please add a thumbs-up reaction if you like).

Returning to your question @skotian2 , if this issue is resolved, I see two possible answers:

• if we consider my solutions, yes
• if we think that the real problem is the one mentioned by @elninoisback , i.e if a .coverage file is generated then Azure DevOps ignores the files in .xml format, then no.

[agilenut]

To me, the real fix for this would be for Microsoft to update Azure DevOps so that you also get the diff PR integration if you use coverlet (not just .coverage). Then we could just use a single coverlet collector and get our reports displayed as well as get the benefit of the PR integration and coverage policies. If this was an option, I wouldn't have the need for multiple collectors. The fact that the .coverage file blocks the coverlet report from rendering is not good. But that wouldn't be an issue for me if I could just use coverlet. But I can't do that because I need the PR integration.

If the PR integration worked for coverlet, we could also use non-Windows build agents which would be fantastic.

I'd also be able to simplify a lot of my pipelines which are currently generating both coverage reports and publishing the coverlet report as a separate artifact that I have to download instead of having it render in the browser.

[gioce90]

be for Microsoft to update Azure DevOps so that you also get the diff PR integra

I think exactly like you. My solution is just a workaround on a problem that only Microsoft can address.

[github-actions]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days