Eliminate the NetCore runner and the NUnit dotnet tool

[CharliePoole]

Discussion under #1493 have led to this decision. I'll excerpt some of it here so all the discussion can be in one place.

@manfred-brands commented:

I'm still confused about the why we have all these different ways to run tests. Historically yes, we used nunit(gui) to run the test. Is this still needed? Now we only use dotnet test for CI and VS Test Explorer to run any tests on all my multi-targeting libraries. The only reason for a non-dotnet command is to allow running tests on a system with no SDK installed. That particular situation will be addressed with the new MS self-contained runners.

@CharliePoole commented:

Regarding the various ways to run tests, you make a good point. I'm confused as well. Back when I was heavily involved in nunit, I worked on what I've been calling the "standard" console runner. It exists because of a perceived need to execute a number of different test assemblies, possibly with different runtime targets, in a single run, producing a single result file. Most of the complexity of the engine is there in order to deal with that requirement. If it ever ceased to be a need, then the runner and engine could be drastically simplified.

I do think you're right to ask why we need the dotnet nunit tool at all. I know that xunit.net dropped their dotnet tool a long time ago. If the tool didn't exist, there would still be four other ways to run tests: the standard console runner, nunitlite, dotnet test and the test explorer. Since the tool is created by subtracting features from the standard console runner, it clearly doesn't do anything the standard runner does. @OsirisTerje Is this just a matter of not introducing a breaking change? If so, let's take do-as-little-as-possible approach right now and plan on getting rid of it in V4.

@OsirisTerje commented:

I kind of like the ease of installing it, compared to the NUnit.Console, but when I look at the download numbers, they are very very low. Total is less than 200K. Last 6 weeks we're talking less than thousand per version.

With the upcoming self-executing new MS Testing platform adding into the set of runners, I agree this one can go. Sad, but.....

@CharliePoole commented:

FWIW, here's how I install NUnit.Console: choco install nunit-console-runner. I'm of the opinion that, for executable applications, chocolatey has all other means of installation beat. I agree that dotnet tool install is very easy for the initial installation, but it doesn't get me any extensions, whereas I can do choco install nunit-extension-some-extension and it just works.

This is an important decision so we want everyone affected to have a chance to express their opinions. In addition to @OsirisTerje and the rest of the @nunit team, the following folks are mentioned because they were recently involved in discussions about the netcore runner: @MichalMucek @kogoel @bouchraRekhadda @Sputnik24

Of course, anybody who uses it is welcome to comment.

[CharliePoole]

Backgound - Currently supported Runners

As background for this discussion, here is a list of the ways nunit tests may be run using the packages we create and distribute. Third-party runners are not listed since they will not influence this decision, at least as far as we know.

1. NUnitLite allows self-executing tests to run and produce output similar to the console runner. The test assembly itself is the executable, so one assembly at a time is run. It executes under the target framework specified by the user and makes use of the standard facilities of .NET to locate any dependencies needed.

2. The "standard" runner, NUnit.ConsoleRunner, executes under the .NET Framework. It can run multiple test projects in a single execution, including tests targeting multiple frameworks, and produces a single XML result file for all tests run. By default, it launches agent processes to execute tests. We currently bundle agents for .NET 4.6.2, .NET 6.0, .NET 7.0 and .NET 8.0. It can also run .NET Framework tests in-process, which makes debugging simpler..

3. The "netcore" runner, NUnit.ConsoleRunner.NetCore may be installed as a tool and runs using the command dotnet nunit. It runs under .NET 6.0 or under a higher runtime if 6.0 is not present. It can execute tests targeting any version of .NET Core equal to or lower than the version under which it runs. It runs only one assembly at a time and the tests always run in the same process as the runner itself.

4. A new experimental runner, NUnit.ConsoleRunner.Net80, was provided in the 3.18.3 release. It works the same as the "netcore" runner but targets .NET 8.0. It runs using the command dotnet nunit-net80.

5. The NUnit3.VisualStudio.Adapter is used to run tools in several ways

   • Using vstest.exe, although this is now somewhat obsolete.
   • Using the dotnet test command.
   • Running under the Visual Studio test explorer
     In each case, the adapter is the actual runner of tests. Like all of our runners (excepting NUnitLite) it makes use of the NUnit engine to run the tests. However, it really only uses a rather small subset of engine features because the execution environment, including a compatible process, has already been set up by the host.

This issue proposes to eliminate 3 and 4, retaining runners 1, 2 and 5.

[CharliePoole]

@manfred-brands Could you expand on your comment regarding the new MS test runners? My understanding is that this is a runner for tests using the MSTest framework. Not so?

[OsirisTerje]

@CharliePoole I can answer that. And No, it is not, it is a general new testing platform. It actually does the same as we do with NUnitLite, so that the test project becomes self-executing. In order to hook this up to the existing infrastructure, they have added and adapter into the NUnit3TestAdapter, which I will try to release a beta of as soon as I can. The test project is still selfexecuting, but will actually work through the NUnit3TestAdapter to hook into Visual Studio and dotnet test.
They have done similar with MSTest and I do believe also XUNit is on the same track now.
Over time NUnit should have its own native implementation of this new testplatform, and we could use NUnitLite as a base for that - althought @manfred-brands said earlier he would like to take a fresh look at NUnitLite and modernize it first.

[CharliePoole]

@OsirisTerje I can't find any link to the general platform, just the MSTest blog post describing executable tests as a new and brilliant idea they just invented. :-) Is the more general thing publicly announced anywhere?

[Sputnik24]

Hi @CharliePoole

thanks for tagging and informing me. Eliminating 3 would also mean eliminating the .net version specific agents and eliminating the option to create an ITestRunner using TestEngineActivator.CreateInstance().GetRunner(new TestPackage(dllfile)) ?

Of course, this would completely brake my use case. In my use case, I create a console application which creates an ITestRunner as mentioned above. Let me give some insights regarding my motivation and use case and why not using your agent or console:

• I need to run testcases from compiled dll files, not from source code
• Compared to your agent, I switched the roles of tcp client and server:
• The console application starts a WebSocket server where any WebSocket client can connect to and call commands to the ITestRunner like explore, run. The console app is running independent of connected clients and clients can also connect later on an already running test and just listen. If the clients disconnect, the tests continue to run or can be terminated.
• I have a platform independent GUI which runs under Windows, Linux, macOS or browser (therefore, websocket)
• I like the way of sending the xml reports around the ITestEventListener, which I provide over the websocket from the server to the client and processing through a TestModel

Edit: After re-reading your summary and my comment and discussing with my colleagues, I assume if you still provide the agents targeting net8.0 in Option 2 (and net9.0 ;) ), I still can use my own agent. Please inform me as soon as a working dev build is available. I will test it asap and give feedback.

[MichalMucek]

Thank you @CharliePoole for adding me.

I share @OsirisTerje's opinion on that for the most part. It's straightforward to install, and it would be sad if the .NET tool were gone.

It's very convenient for our case to have it. With 18 Windows machines running UI tests, all we need to do to update the NUnitConsole, is to bump up the version number in a tool manifest of repo. It's also not an issue to go back to a previous version when needed. Without it, we either have to work out an automatic way to provision all the machines (current and new) with a version of NUnitConsole, or update the Windows installation image which is not a straightforward process.

[manfred-brands]

@MichalMucek For my understanding, why can you not use dotnet test? What does the console runner bring that dotnet test does not?

[CharliePoole]

@Sputnik24 This issue proposes to remove the .NET Core builds of the console runner. It does not affect the engine at all. It also doesn't affect the existing "standard" runner (i.e. the console runner currently built under .NET framework which uses agents to run tests). Users would be left with one less way to run tests, but all targets currently supported would be supported. Those using the dotnet CLI would lose dotnet nunit but continue to use dotnet test, which relies on the adapter and the agent. We're looking for input from those who would potentially suffer from that change before making a final decision, but your use case of creating your own runner would not be affected.

It's a bit off topic for this issue, but fwiw the existing console runner (option 2) also reverses the typical client server model. It's agents are clients, which connect back to it for instructions. Communication is via TCP or Remoting at this time but the design allows alternate protocols to be added.

[CharliePoole]

@MichalMucek
Same question as @manfred-brands about dotnet test. In my case, I ask without a specific opinion. I'm not a user of either but they look the same to me in terms of capabilities. Additionally, the full runner (perhaps that's the term I should use rather than standard) has a greater set of capabilities than our dotnet runner. It's drawback, of course, is that a recent version of the dotnet framework must be installed in order to use it even if all your work is under dotnet core.

From the development point of view, I think it would be better to spend time migrating the standard runner to .NET 8.0 or higher rather than continuing to maintain 2 or 3 different builds of the runner each with different capabilities. Of course, if more people were working on this, that may be less of an issue.

[Sputnik24]

@Sputnik24
It's a bit off topic for this issue, but fwiw the existing console runner (option 2) also reverses the typical client server model. It's agents are clients, which connect back to it for instructions. Communication is via TCP or Remoting at this time but the design allows alternate protocols to be added.

I know, that's why I mentioned that I reversed the client/server logic of your agent and had to develop my own agent being the server. So, to sum up, I would not be affected by this change.

Again, thanks a lot for informing me.

[MichalMucek]

@manfred-brands @CharliePoole

@MichalMucek For my understanding, why can you not use dotnet test? What does the console runner bring that dotnet test does not?

Better filtering. It allows filtering the SpecFlow BDD tests by the examples. It's convenient and useful because different examples may have different categories. dotnet test does not have such a capability. It can at most run scenarios with a whole batch of examples.

[OsirisTerje]

It allows filtering the SpecFlow BDD tests by the examples.

I am not a Specflow guy. Can you show how you call the console "by examples"?

[CharliePoole]

@MichalMucek
The filtering issue makes me want to think a bit more. It would be good to have examples, including those that work with both dotnet test and dotnet nunit and those that only work with dotnet nunit.

Is filtering any different using SpecFlow vs NUnit by itself?

[MichalMucek]

@OsirisTerje To be precise, I mean filtering and listing (exploring in terms of NUnit).

Let's consider the below test case in the Gherkin language based on an imaginary PrinerHelper application:

Scenario Outline: PrintingHelper can connect to a printer
    Given PrinterHelper was started
    And "Color" printer was connected to PC using "<CableType>" cable
    When User clicks printers tab
    Then user will see "Color" printer on the list
@USB
Examples:
    | CableType |
    | USB       |
@Serial
Examples:
    | CableType |
    | Serial    |

Having two PCs, each with a printer connected using a different cable (PC1 - USB; PC2 - Serial). We want to run a specific example on a particular PC. Otherwise, the scenario will fail when running every example on each machine. To resolve it, we have to run each example separately and NUnitConsole allows that. Let's say there are about 800 scenarios with similar examples and we add more PCs to speed up the execution.

With the below command, we're able to obtain the IDs (they're the key) and test method names of scenario examples with @serial category. Obtained list can be split into smaller lists that could run in parallel on multiple PCs/agents.
dotnet tool run nunit tests.dll --test-name-format="{i} {M}" --noheader --where cat==Serial --explore

dotnet test --list-tests can list all the test scenarios with their examples, but that's about it. --list-tests doesn't work with --filter. However, the --filter works alone, but then again, it's not possible to split it into smaller lists that run in parallel.

Edit: I forgot to mention that SpecFlow is responsible for generating C# code with test methods from the scenarios written in Gherkin language. Test steps are bonded with methods written in C#.

[CharliePoole]

@MichalMucek
OK, that's a good example although the translation to C# is a bit of a black box I wish I understood better.

What I'm missing is this... what prevents you from using the standard console runner to generate your test lists?
Also, do you execute the tests using the SpecFlow runner or the console?

[MichalMucek]

@CharliePoole

What I'm missing is this... what prevents you from using the standard console runner to generate your test lists? Also, do you execute the tests using the SpecFlow runner or the console?

To be fair, there is no actual blocker for us. We prefer using the .NET tool because of its ease of installation in our test environment. Implementation-, maintenance-, and time-wise it is cheaper. Relying on the standard executable would mean either a few hours spent each time when updating the Windows installation image with a new version of the Console, or creating a custom way of provisioning automatically Windows machines with NUnitConsole. Chocolatey is a no-go for us in our test environment due to network infrastructure restrictions.

We execute the tests using the Console. SpecFlow alone can't do that. https://docs.specflow.org/projects/specflow/en/latest/Execution/Executing-SpecFlow-Scenarios.html

BTW SpecFlow development is practically abandoned. It has been taken over by fork project Reqnroll.

[CharliePoole]

@MichalMucek

That clears up some things for me, but the looking at the SpecFlow docs confused me. :-(

Their docs state that for test discovery and execution, you need

• The NUnit3TestAdapter
• Microsoft.NET.Test.Sdk

That implies use of dotnet test or the Visual Studio IDE rather than dotnet nunit. So are the docs out of date?

Thanks for the info about Reqnroll. I'll take a look at that - maybe the that team will be a bit more transparent than SpecFlow was since they are eliminating the non-open components.

[CharliePoole]

I'm reaching the conclusion that we can't really get rid of the tool at the moment for two main reasons:

1. Folks who don't have .NET Framework installed obviously can't use the standard runner.
2. Our tool provides filtering that's not avaiable using the dotnet test
3. Some folks just prefer it.

I'd like to do this in a way that minimizes the level of effort we put in, so I'm suggesting the following for 3.19...

1. Upgrade our NetCore package to .NET 8.0.
2. Drop the experimental .NET Core runner.

The following items are more properly part of issue #1492, so I'll repeat and enlarge on them there.

3. Stop building the engine for .NET 6.0 and .NET Standard 2.0, leaving only .NET 4.6.2 and .NET 8.0
4. Improve the .NET 8.0 engine to gradually include more features, which are currently left out by use of conditional compilation.

It may also be helpful to consider adding better filtering to our support of dotnet test.

What does everyone think of the above as a resolution?

[OsirisTerje]

I believe what you write is sound. I agree.

But
I still don't understand why it is said that dotnet test can't filter the same way as the console. It might be true that the built in filter in dotnet test doesn't have that capability, but dotnet test calls on the adapter which again calls the engine, and you can actually pass any NUnit filter down there from dotnet test. The statement to do this is the NUnit Where filter statement. See https://docs.nunit.org/articles/vs-test-adapter/Tips-And-Tricks.html#where and this blogpost too: https://blog.prokrams.com/2019/12/16/nunit3-filter-dotnet/ Also note that the Where filter can be passed down as a command line argument.

[CharliePoole]

I'll move on to issue #1492 and continue to build out a plan for 3.19. It looks like this one can be closed, but I'll let it sit for a few days to see if there are any other responses, esp. from @MichalMucek regarding the --where filter.