Coverlet producing incomplete coverage results #1278
Comments
|
Added detail: the issue seems to come from processing multiple DLLs at once. When I pick one of the dlls which is producing 0 coverage and process that one as a single file, the results are good. Maybe I can pinpoint it to a specific project this way which might be causing issues, not sure. Update: it doesn't seem to be related to a certain project being in the list, but rather the sequence of the projects. If I place one of the projects reporting zero at the top of the list, now suddenly the coverage is 100% again. Place it back down and coverage disappears. So there seems to be an issue with handing Update 2: when I run the tests all seperately ( |
|
Hi @jsiegmund, first of all you're mixing the configuration for coverlet and MS CodeCoverage parameters, coverlet supports only these https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md#advanced-options-supported-via-runsettings can you try with: With a correct runsettings. PS: this param |
MarcoRossignoli
added
waiting for customer
|
Thanks for your reply. I'm getting back to it but need to carve out some time to do so. I'll reply back here. |
|
Sorry, here's a ping to let you know I did not forget but also didn't find the time to look into it. I'll report back asap. |
|
Not sure if I'm seeing the exact same scenario, but I'm getting empty coverage results when passing a DLL like this: dotnet test ./test/UnitTests/publish/UnitTests.dll --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=coberturaWhereas I get actual coverage results when passing a csproj directory like this: dotnet test ./test/UnitTests --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=coberturaOddly, this is happening in an Alpine Docker container, but outside the container on my Windows 10 host machine, both approaches work. |
|
@SeijiSuenaga I think that your scenario is different, the issue inside a docker is related to the fact that coverlet excludes dll if it's not able to locate the sources (you moved files inside a container and sources are no more available), we have an issue opened for that and we'll work on it. #1164 |
|
I'm also having an issue with incomplete coverage. Sometimes coverlet produces a complete coverage and sometimes it returns 0%. How can this be fixed? |
|
@olga-shtykova if you're out of a docker maybe you should move if possible to the collector integration, msbuild and .NET tools suffer sometimes of a defect related to test platform behavior, if you take a look at the documentation we have a section for it(the fist one https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/KnownIssues.md). New test platform version was been shipped with some mitigation but I'm not sure if it was been released yet, which version of test SDK are you using (check inside the csproj). |
|
@MarcoRossignoli Thank you for your answer, I've already looked through this section. I do use docker |
|
You should try to use the datacollector integration https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md also inside the docker msbuild integration suffers of the issue described in the known issue doc. |
|
I forgot to mention that I'm using Microsoft.NET.Test.Sdk" Version="17.0.0" |
|
@MarcoRossignoli Using coverlect.collector instead of coverlet.msbuild, in .gitlab-ci.yml do I need to reference the same way /p:CollectCoverage=true or somehow else? And can I use coverlet.console for report, e.g., |
|
The collector works alone you can avoid the /p:CollectCoverage and also the .net tool you can use simply dotnet test. Here you can find a full sample You can pass a run settings with the same options available for the tool and /p: CollectCoverage, you can find it in the documentation I've pointed above |
|
@MarcoRossignoli I appreciate your patience and willingness to help. I cannot find anything in the coverlet integration with VSTest documentation on how to visualize code coverage in gitlab CI. Maybe you can point me to the right direction? |
|
Coverlet will generate a cobertura XML file with the coverage and the visualization depends on the tool you're using, if you're on gitlab I think this is the documentation https://docs.gitlab.com/ee/user/project/merge_requests/test_coverage_visualization.html Visualization of coverage is out of the scope of coverlet, let's say that we generare the file that should be used by some "visualizer", usually I generate report with reportgenerator tool, but also some CI systems do that like gitlab. |
|
I'm experiencing similar behavior where I receive appropriate coverage reports on my development machine, but running in CI with an alpine-based image for a .net6.0 and coverlet collector. Microsoft.NET.Test.Sdk 17.2.0 The cobertura file is generated, but only with empty sources and packages nodes. |
|
I have the same issue with an empty cobertura output when testing with a pre-built dll: What I do:
I use these run settings: Microsoft.NET.Test.Sdk 17.1.0 I tried this on a dev machine and also in a container job on Azure DevOps, same result. Running against the .csproj file works properly. |
|
@alexgalie you should always pass and remove the |
|
@MarcoRossignoli thanks for the reply, after some experimentation I can confirm I have essentially reproduced @SeijiSuenaga 's issue above. The command line argument vs the The issue is the missing source files, which I have verified by copying the source files in the same path as where they were originally built. Only then the report is generated properly. |
|
@alexgalie to verify if that is the prioblem we can produce and check the logs...the skip reason are logged there https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/Troubleshooting.md#collectors-integration and yes the symptom can be the same |
|
@MarcoRossignoli I enabled the logging, below are a couple of fragments which are relevant from what I can think of. The first one is from the main The second is the full output of I am not including the actual test run because it is quite verbose and may contain some sensitive data. |
|
Do you see some comment like |
|
No I do not have this in any of the logs |
- Update VSTest to minimum supported version - Use the coverlet connector Due to known issues, the datacollector integration is suggested See coverlet-coverage/coverlet#1278 (comment) And https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md <!-- Minimum version 16.5.0 -->
- Update VSTest to minimum supported version - Use the coverlet connector Due to known issues, the datacollector integration is suggested See coverlet-coverage/coverlet#1278 (comment) And https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md <!-- Minimum version 16.5.0 -->
- Update VSTest to minimum supported version - Use the coverlet connector Due to known issues, the datacollector integration is suggested See coverlet-coverage/coverlet#1278 (comment) And https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md <!-- Minimum version 16.5.0 -->
- Update VSTest to minimum supported version - Use the coverlet connector Due to known issues, the datacollector integration is suggested See coverlet-coverage/coverlet#1278 (comment) And https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md <!-- Minimum version 16.5.0 -->
- Update VSTest to minimum supported version - Use the coverlet connector Due to known issues, the datacollector integration is suggested See coverlet-coverage/coverlet#1278 (comment) And https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md <!-- Minimum version 16.5.0 -->
|
This issue is stale because it has been open for 3 months with no activity. |
|
If I understand this correctly - the source files are needed for coverlet to work. I'm trying to use only build artifacts (produced w/ dotnet publish) from one azure pipeline job, and use them to produce coverage result in another job (s). I need to parallelize, as some of the tests are really heavy (integration) and take a lot of time to execute. Building each test project in individual jobs is an option, but as the builds are heavy as well, skipping them is a good idea. So, is there any supported scenario, in which coverage result can be collected, using pre-build binaries, and eventually the source files. I.e. in pseudocode: |
|
This issue is stale because it has been open for 3 months with no activity. |
|
Please verify whether the problem still exists using the latest coverlet 6.0.2 release. |
Hi there, we have kind of an oddball situation here which is potentially causing issues with Coverlet. This is a client project / client code so unfortunately I can only share the set-up and not the actual bits and pieces.
The solution is quite large with a total of 128 projects. These are all .NET Framework 4.8 SDK-style. We have WebAPI projects in there which normally you cannot use with SDK style, but using https://github.com/CZEMacLeod/MSBuild.SDK.SystemWeb we have converted these to SDK style as well. We're not using any test frameworks like xunit or such, and all package versions are on the latest (MSTest / Test SDK and Coverlet as well).
Due to the SDK style projects we cannot use dotnet test, that just does not work. We can use dotnet vstest however, or simply call vstest.console.exe so that is what we do. Coverlet collectors is set-up using a runsettings file which is not that special:
(note that this is the current state, I've been messing around with options to no avail)
Now when I run dotnet vstest supplying the full list of DLLs we want to check, I use the following command:
This runs and produces a coverage XML file, but for certain modules the coverage results are zero. Note: for certain modules, not for all! Overall coverage is reported to be 15% whereas Visual Studio when asked says 80%. So why is it missing things?
I've inspected the diagnostics output and found a number of
Hits file not foundlines which seem to correspond with the modules for which the output is not being shown. That makes sense, but why is it not finding these files? For other modules it showsHit file deleted, which is probably what you'd want to see? I searched the other diag files for hints which might explain why there are no hit files, but couldn't find any more leads. So I'm stuck now trying to find the root cause of this, any pointers where to look would be highly appreciated.The text was updated successfully, but these errors were encountered: