'dotnet test' in solution folder fails when non-test projects are in the solution

[bungeemonkee]

Description

Being able to run 'dotnet test' on a solution now is amazing! However, unless that solution contains ONLY test projects it always produces a failure result due to not finding the test sdk. In my experience it is rare for tests to be in a separate solution - they generally don't exist or are in the same solution as the code they are testing.

It seems logical to me that when running tests on a solution any projects without the test sdk should be ignored. If a project has the test sdk and no tests or is being tested in isolation and does not have the sdk that is likely an error. However if a project does not have the sdk and is being tested as part of a solution that is likely not an error.

Alternatively, if that feels too much like a breaking change then adding an option to filter by patterns on the project name could suffice. Something like dotnet test --projects "*.Tests.*". Since most test projects in my experience have some form of pattern to the name - usually involving the word 'test' - this would probably be sufficient for most cases.

Note that the existing --filter <EXPRESSION> switch is insufficient to prevent the exception, presumably because it is processed by the underlying test framework which happens after the missing sdk error is thrown.

Steps to reproduce

'''
dotnet new sln -n Solution
dotnet new console -n Program
dotnet sln Solution.sln add Program/Program.csproj
dotnet new mstest -n Tests
dotnet sln Solution.sln add Tests/Tests.csproj
dotnet test
$?
'''

Expected behavior

1. Existing tests run.
2. Projects with no test sdk reference are ignored.
3. Return code is zero.

Actual behavior

1. Exiting tests run.
2. Projects with no test sdk reference are run as test projects and fail due to an error.
3. Return code is non-zero. In this case it appears to be 1 but I'm not sure if that is consistent.

Environment data

dotnet --info output:
.NET Command Line Tools (2.0.0)

Product Information:
Version: 2.0.0
Commit SHA-1 hash: cdcd192

Runtime Environment:
OS Name: ubuntu
OS Version: 16.04
OS Platform: Linux
RID: ubuntu.16.04-x64
Base Path: /usr/share/dotnet/sdk/2.0.0/

Microsoft .NET Core Shared Framework Host

Version : 2.0.0
Build : e8b8861ac7faf042c87a5c2f9f2d04c98b69f28d

What is not clear form this environment output is that I am running in bash via the WSL.

[bencyoung]

We have the same issue. A filter or the ability to point it at a sub-directory would be great

dotnet test tests/

or something like that...

[peterichens]

Same problem here, the error thrown causes our CI to report failure. Ignoring projects without the test sdk would be good, or at least just logging a message/warning, rather than an error.

[kendrahavens]

@pvlakshm Could you add the dotnet-test label?

[johncrim]

Same issue appears to be covered here: Allow dotnet test to run on all test projects in a given directory dotnet/cli#705.

[livarcocc]

This issue was moved to microsoft/vstest#1129