How to ExcludeFromCodeCoverageAttribute in a test framework scenario?

[jzabroski]

Description

If I run dotnet test --collect="Code Coverage", then it counts xunit classes towards code coverage statistics.

Coverlet addresses this problem by providing an ExcludeByAttribute where the user can define arbitrary attributes to use for exclusion purposes.

Steps to reproduce

1. Create a new .NET library project that has a class Foo that calculates A + B
2. Create a new xunit test project that tests class Foo
3. Run dotnet test --collect="Code Coverage"

Expected behavior

xunit assemblies not included in code coverage report.

Actual behavior

xunit assemblies included in code coverage report.

Diagnostic logs

n/a - please specify what you would want for this, and I can provide

Environment

Test SDK 17.5

[jzabroski]

Note: Brad Wilson, xunit maintainer, said this is not a real concern for him, since xunit does not collect code coverage stats for the project today.

However, I am curious what the VSTest engineers have to say about this base case.

[MarcoRossignoli]

cc: @jakubch1

[jakubch1]

Thanks for the report. We will work on excluding xunit dlls by default.

[jzabroski]

Oops. Hit close by accident while holding my newborn.

Just wanted to say that you should also think about Coverlet approach ExcludeByAttribute. For example, what about assertion libraries like Shoudly.

[jakubch1]

In latest version https://dev.azure.com/dnceng/public/_artifacts/feed/test-tools/NuGet/Microsoft.CodeCoverage/overview/17.7.0-preview.23203.7 we added default exclusion of few libraries. We are working on some more generic approach like Coverlet has so not closing this.

[gioce90]

We are working on some more generic approach like Coverlet has so not closing this.

Hi @jakubch1 , what does this approach consist of? Can you share some information about it? It would be nice to exclude from code coverage report all about third-party libraries (without using .runsettings files)

[jakubch1]

We added some default exclusions for xunit and moq libraries. Generic solution is still not implemented.

[gioce90]

I think this is related to #72

[jakubch1]

In latest version https://dev.azure.com/dnceng/public/_artifacts/feed/test-tools/NuGet/Microsoft.CodeCoverage/overview/17.10.0-preview-24073-02 we ported from Coverlet feature to optionally exclude modules based on existence of sources.

To enable this feature in command line you can use:

dotnet test --collect "Code Coverage;ExcludeAssembliesWithoutSources=MissingAll"

If you use config/runsettings you can add:

<ExcludeAssembliesWithoutSources>MissingAll</ExcludeAssembliesWithoutSources>

under <Configuration> tag.

Remember that this is adding additional processing to check existence of sources. You will get always better performance by configuring correctly <ModulePaths> regexes.

[cremor]

Shouldn't this be the default?
When would I ever care about coverage of an assembly for which I don't have any sources?

[jakubch1]

I don't want to make it default because of performance. To check if sources exist you need to start with dll path => read dll metadata => read PDB data and get all sources => in case of deterministic build calculate original paths => check each file existence. This can take time for big solutions

For <ModulePaths> regexes you take dll path and check regex => no disk operations.

To calculate coverage using our tooling, sources are not needed so you can imagine scenarios that you build test projects on 1 machine then move binaries to several test nodes and execute them there. We are able to collect coverage in such scenarios.