Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NUnit Engine 3.15.0 cannot load the test assembly #1180

Closed
nataliaheliova opened this issue Apr 12, 2022 · 20 comments
Closed

NUnit Engine 3.15.0 cannot load the test assembly #1180

nataliaheliova opened this issue Apr 12, 2022 · 20 comments
Assignees
@CharliePoole
Labels
Bug
Milestone
3.16.0

Comments

@nataliaheliova
Copy link

Hey there,

I have a .NET 6.0 console project which references NUnit.Engine 3.15.0. When I try to run the tests from test assembly (targeting .NET 6.0, referencing NUnit 3.13.3), I get the following exception:

NUnit.Engine.NUnitEngineException
  HResult=0x80131500
  Message=An exception occurred in the driver while loading tests.
  Source=nunit.engine.core
  StackTrace:
   at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage)
   at NUnit.Engine.Runners.DirectTestRunner.LoadPackage()
   at NUnit.Engine.Runners.DirectTestRunner.EnsurePackageIsLoaded()
   at NUnit.Engine.Runners.DirectTestRunner.RunTests(ITestEventListener listener, TestFilter filter)
   at NUnit.Engine.Runners.AbstractTestRunner.Run(ITestEventListener listener, TestFilter filter)
   at NUnit.Engine.Runners.MasterTestRunner.RunTests(ITestEventListener listener, TestFilter filter)
   at NUnit.Engine.Runners.MasterTestRunner.Run(ITestEventListener listener, TestFilter filter)
   at Program.Main(String[] args) in C:\Users\nathe\source\NUnit3_Repro\NUnit3_Repro\Program.cs:line 22

  This exception was originally thrown at this call stack:
    [External Code]

Inner Exception 1:
FileNotFoundException: Could not load file or assembly 'Tests, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.

The same works flawlessly when I downgrade to NUnit Engine 3.14.0. Is there any reason why this would not work with the latest NUnit Engine?

Please find attached solution which can be used to reproduce the issue.
NUnit3_Repro.zip
@CharliePoole
Copy link
Member

This is the same as nunit/nunit-v2-framework-driver#51 with the added detail that your code works under 3.14.0.

I closed the first issue as "not a bug" for reasons I explained there but I'll keep this one open for a while because I think the question of why it worked before is kind of interesting.

First, some questions for you...

  • Are you sure that all the versions and references in the example that works are identical, except for the version of the engine?
  • What are you trying to accomplish? Are you writing a test runner similar to nunit3-console?
  • Looking at the output directory in each case, are the contents the same? In particular, are the engine and the extension both present?

@nataliaheliova
Copy link
Author

Hey @CharliePoole,

thank you for detailed answers for both questions. Basically I'm trying to automate both NUnit 2 and 3 tests. Based on your answers I realized I can achieve that easily with nunit3-console, I missed this option before :) Thanks a lot for helping me to wrap my head about this.

Therefore this issue is not relevant for me but still interesting so here comes my answer: There is no extension referenced in this case (here I'm only trying to run NUnit3 tests) and I only reference NUnit.Engine. If I run the same code with NUnit.Engine 3.14.0, everything works fine. However with NUnit.Engine 3.15.0, I get the exception. The output is the same for both cases, apart from different versions of nunit.engine.api.dll, nunit.engine.core.dll and nunit.engine.dll.

@CharliePoole CharliePoole added this to the 3.15.1 milestone Jun 7, 2022
@CharliePoole CharliePoole modified the milestones: 3.15.1, 3.15.2 Jun 29, 2022
@Cylox
Copy link

Cylox commented Aug 3, 2022 •
edited
Loading

Hi @nataliaheliova and @CharliePoole ,

I'm facing the same issue when using NUnit.Engine 3.15.X from a .net60 console application. I've drilled it down to the fact that package version 3.14 of the engine still contained the assemblies for .netcoreapp31 but those were removed from 3.15 only leaving netstandard20 to use for a project targeting .net60. Therefore the NUnitNetStandardDriver is used and not the NUnitNetCore31Driver. NUnitNetStandardDriver is unable to load any assemblies that are not in the Default AppDomain's base directory or probing paths. Therefore it is impossible to load a test assembly from anywhere else.

Why have the .netcoreapp31 binaries been removed from NUnit.Engine 3.15 and newer?

NUnitNetCore31Driver uses it's own AssemblyLoadContext and is able to load test assemblies from any path.

@CharliePoole
Copy link
Member

@Cylox

I've left this issue inactive after @nataliaheliova 's last comment, but if it remains an actual problem for you, we can try to work it out.

The NUnitNetCore31Driver was removed because we were able to simplify the code without breaking any tests. So we'll need to investigate how you are calling the engine from your own code and how that differs from what the console runner is doing.

Can you add some more info about what you are doing in your code? Also, have you tried running your test assembly under the console runner?

@Cylox
Copy link

Cylox commented Aug 3, 2022 •
edited
Loading

@CharliePoole I've tried running it with the .net60 build of nunit3-console (3.15.2) and this uses NUnitNetCore31Driver internally (and therefore, this works). This is why I am surprised it's not in the engine package any more.

What I am doing in my code: I have a custom test runner built initially on .Net Framework which I am migrating currently to .net6. That runner has some special handling and selection / filtering logic for SpecFlow scenarios and also creates custom reports.
Until now the application executed the NUnit test runner in the default AppDomain and depended on NUnit to open up separate AppDomains for the test. This is also what NUnitNetStandardDriver seems to expect. This is not possible with .net6 any more.
I found a way to work around this by subscribing to the AppDomain.AssemblyResolve and using Assembly.LoadFrom to load the test assembly (since the application knows where it's located) and all of its dependencies (which therefore must be in the same directory) but I'm not sure if this is the way to go.

For me it seems like NUnitNetCore31Driver should be part of the NUnit.Engine package again because it is also used by nunit3-console.

@CharliePoole
Copy link
Member

@Cylox

Hang on a minute! Are we talking about nunit3-console.exe, which comes as part of the NUnit.ConsoleRunner package or nunit-console.exe, which is part of NUnit.ConsoleRunner.NetCore? (I'll grant you that the naming is confusing!)

The former runs under .NET Framework, but launches separate processes for the tests. The latter runs under .NET 6.0 and only runs tests in process. The reasons for this are historical - plus we are in transition toward 4.0.

@Cylox
Copy link

Cylox commented Aug 4, 2022 •
edited
Loading

@CharliePoole
Now I'm confused. I'm definitely talking about nunit3-console.exe that is part of the package NUnit.Console in version 3.15.2. This one: https://www.nuget.org/packages/NUnit.Console/3.15.2. Inside this package there is a bin\net6.0 folder. Inside that folder there is a nunit3-console.exe and nunit3-console.dll which is built for .NETCoreApp,Version=v6.0 (provided by ILSpy).

So I would say this is a .net6 build of nunit3-console, which does not run under .NET Framework. Am I missing something?

@CharliePoole
Copy link
Member

@cyclox
None of the nuget packages contain a bin folder at all. They have a tools folder containing everything.

The NUnit.Console package contains no software at all. It only references a set of other packages, including the main one, NUnit.ConsoleRunner.

The NUnit.ConsoleRunner package has a net20 directory and various agent directories. Directory agent/net6.0 exists, but doesn't contain a copy of nunit3-console.exe, only the agent exes.

Suggestion... take the package you are actually using and temporarily change the extension from .nupkg to .zip. Then you can unzip it and see what's there.

@Cylox
Copy link

Cylox commented Aug 4, 2022 •
edited
Loading

Hi @CharliePoole , thanks for taking your time with me. I was totally on the wrong path. Sorry for that. I had actually downloaded the .zip distribution of this NUnit.Console release from the GitHub Releases page, not the .nupkg file: https://github.com/nunit/nunit-console/releases/tag/3.15.2
For some reason I thought it was the nuget package and also wondered why there was no tools folder. This should be clarified now.
To start over: I've run my test successfully with the distribution of nunit3-console.exe in the bin\net6.0 folder of that zip file, and that folder includes the .netcoreapp31 version of the engine with the corresponding driver.

When I use the nuget package NUnit.ConsoleRunner.3.15.2.nupkg I can run the test successfully but of course only if I don't use the --inprocess tag so that the agent is used. For my own runner, I'm trying to do what the nunit3-console.exe .net6.0 build does that is contained in the NUnit.Console-3.15.2.zip package. That means: I want to run the test assembly directly without an agent. I cant do that however because it uses a version of the engine that is not distributed by the NUnit.Engine nuget package.

I hope that clarifies my problem. So either I need to go with the AssemblyResolve workaround and hope this doesn't break anything or I'll need to build the NUnit.Engine version for .netcoreapp31 myself. At least this are the solutions that come into my mind.

Thanks for your patience!

@CharliePoole
Copy link
Member

@Cylox OK, that clears up one bit at least. Before going further, I'm going to review the last few versions for consistency to make sure there wasn't a packaging error. I'll be back.

@cyclox
Copy link

cyclox commented Aug 4, 2022 via email

@CharliePoole
Copy link
Member

@cyclox Apparently github handles are case-sensitive. I'll be more careful. :-)

@cyclox
Copy link

cyclox commented Aug 4, 2022 via email

@CharliePoole
Copy link
Member

Looking at the contents of the zip package, I can see that it's a bit confusing. There are two completely different executable applications included with the same name, one under net20 and one under net6.0, whereas if you use nuget, there are two entirely different packages. Longer term, I think I need to change this.

@CharliePoole CharliePoole mentioned this issue Sep 24, 2022
Closed
@CharliePoole
Copy link
Member

@Cylox See issue #1225

@cyclox
Copy link

cyclox commented Sep 24, 2022 via email

@CharliePoole
Copy link
Member

@cyclox I think you're being notified because you were previously in the discussion due to my mistake. You'll have to unsubscribe from it.

@cyclox
Copy link

cyclox commented Sep 24, 2022 via email

@CharliePoole
Copy link
Member

Closing this since #1225 restored netcoreapp3.1 build of the engine.

@CharliePoole CharliePoole self-assigned this Oct 19, 2022
@CharliePoole CharliePoole modified the milestones: 3.15.3, 3.16 Oct 19, 2022
@CharliePoole
Copy link
Member

This issue has been resolved in version 3.16.0

The release is available on:
GitHub.
NuGet packages are also available NuGet.org and
Chocolatey Packages may be found at Chocolatey.org

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@CharliePoole CharliePoole

Labels
Bug
Projects
None yet
Milestone
3.16.0
Development

No branches or pull requests
4 participants
@CharliePoole @Cylox @nataliaheliova @cyclox