New test files require rebuild to be picked up by VS Test explorer

[erik-inkapool]

When adding test files to a project the tests do not get picked up on first build, but they are picked up when rebuilding.

To repro, remove the "precompile": [ "dotnet specflow" ] line in the scripts section of my sample project, delete any .feature.cs files and rebuild the project. No tests should (correctly) have been discovered at this point. Now, add the line, back, rebuild the project. No tests have still been discovered (incorrect). Rebuild again, now the tests appear.

This may be a dotnetcore issue because I see slightly similar behaviour when just adding new non-specflow tests to my project (tests being discovered but not ran in that case), but should still be looked into.

[stajs]

Thanks for the report, I'll have a look when I get some time.

[smudge202]

I'm confident this is a tooling issue (problems between dotnetcore and the VS Test Runner). If you speak with @bradwilson he can probably confirm.

[bradwilson]

From the runner perspective, we never deal with source, only DLLs. It's definitely sounds like .NET CLI tooling is to blame.

My guess is that the build engine had already decided what files to compile before it runs your precompile step. This is why it works the second time: the file is already on disk before your precompile step runs.

[smudge202]

I'm pretty sure I can repro this with xunit alone (let me know if you want repro steps). I don't for a second think this is xunit's fault, I'm very aware the test runner interaction can be interesting at times. More wondered if this is something you've already seen reported, @bradwilson? Perhaps spoken with some VS guys about it or would know who to point us at?

[bradwilson]

.NET CLI bugs should be reported here: https://github.com/dotnet/cli

What do you mean you can repro it with just xUnit.net alone?

[smudge202]

I take that back. I remember in the past that I used to have to build before running tests in .Net Core (I have it in some old youtube videos I'm sure, but that was possibly in RC2 or even RC1). I see the behaviour has been fixed since. It used to be that everytime I added a new test (or renamed an existing one) I would have to build then run, despite the fact that running the tests does a build. That no longer seems to be the case.

It was never a great hardship so I never thought to investigate further.

[stajs]

Thanks for the input guys. It looks like this issue: https://github.com/dotnet/cli/issues/3807

And just to confirm it again, you have to also run dotnet test twice from the command line for tests to be picked up (so not just VS Test Explorer) indicating it is a .NET CLI issue.

[smudge202]

Yup, I was about to comment based on testing for another issue. Any file created during precompile is ignored; I'd guess that the file lists for compilation are generated before the precompile step currently. Works fine after first build/creation of each designer file - worthy of a readme mention I'm sure.

[smudge202]

I wonder... would it be worthwhile throwing an exception or returning a non-zero exit code if files are created by the specflow process so that the user can be informed that not all tests are going to be run until next build? It's yet another ugly-interim-hack, but it might be a useful one?

[stajs]

Yeah, definitely worth noting in the README. I'm also open to the idea of emitting a warning of some sort.

[stajs]

Accidentally closed. Reopening, even though the fix is with .NET CLI.