MSTest 3.0.1 is broken when using deployment items

[avivanoff]

Describe the bug

Found while verifying #1437.

Steps To Reproduce

1. Create UnitTest project in Visual Studio 2022.
2. Make sure target framework is .NET Framework 4.8.
3. Add .runsettings file.
4. Add a deployment item.
5. Build and execute unit tests using vstest.console.exe:
   "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" TestProject1\bin\Debug\net48\TestProject1.dll /Settings:"TestProject1\local.runsettings" /TestAdapterPath:TestProject1\bin\Debug\net48

Expected behavior

Tests execute.

Actual behavior

Tests do not execute.

Additional context

See attached sample project TestProject1.zip.

Tests output:

An exception occurred while invoking executor 'executor://mstestadapter/v2': Could not load file or assembly 'Microsoft.TestPlatform.CoreUtilities, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
Stack trace:
 
Server stack trace:
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Deployment.AssemblyLoadWorker.GetFullPathToDependentAssemblies(String assemblyPath, IList`1& warnings)
   at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
   at System.Runtime.Remoting.Messaging.StackBuilderSink.PrivateProcessMessage(RuntimeMethodHandle md, Object[] args, Object server, Object[]& outArgs)
   at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)
 
Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Deployment.AssemblyLoadWorker.GetFullPathToDependentAssemblies(String assemblyPath, IList`1& warnings)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Utilities.AssemblyUtility.GetFullPathToDependentAssemblies(String assemblyPath, String configFile, IList`1& warnings)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Utilities.DeploymentUtility.AddDependencies(String testSource, String configFile, IList`1 deploymentItems, IList`1 warnings)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Utilities.DeploymentUtility.AddDependenciesOfDeploymentItem(String deploymentItemFile, IList`1 filesToDeploy, IList`1 warnings)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Utilities.DeploymentUtilityBase.Deploy(IList`1 deploymentItems, String testSource, String deploymentDirectory, String resultsDirectory)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Utilities.DeploymentUtilityBase.Deploy(String source, IRunContext runContext, ITestExecutionRecorder testExecutionRecorder, IList`1 deploymentItems, TestRunDirectories runDirectories)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestDeployment.Deploy(IEnumerable`1 tests, IRunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle, TestRunCancellationToken cancellationToken)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestExecutor.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(LazyExtension`2 executor, Tuple`2 executorUriExtensionTuple, RunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.<>c__DisplayClass46_0.<RunTestInternalWithExecutors>b__0()
   at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformThread.<>c__DisplayClass0_0.<Run>b__0()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformThread.Run(Action action, PlatformApartmentState apartmentState, Boolean waitForCompletion)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.TryToRunInStaThread(Action action, Boolean waitForCompletion)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable`1 executorUriExtensionMap, Int64 totalTests)

[ms6073]

.NET framework 6.0.306 as well.

[Evangelink]

@avivanoff I have tried to download and run your sample project and I don't have any issue (disabling or not app domains):

.NET framework 6.0.306 as well.

@ms6073 Are you also having the same exception or is it something else? You are mentioning .NET 6 and there is a breaking change in .NET 6 related to DeploymentItems (the folder was changed to match what's done for .NET framework). If you have a different exception, please open a separate issue and we will handle it there.

[Evangelink]

Sorry @avivanoff I need some coffee, you said it's failing in command line. Will investigate it.

[Evangelink]

Alright so the first pass analysis shows that the issue was introduced in 2.2.7 (using your example versions up to 2.2.6 are good and then it starts to fail). I will investigate deeper and try to have this fixed for a 3.0.2, sorry for the trouble.

[avivanoff]

@Evangelink, 2.2.10 works just fine.

[Evangelink]

With the same test project? It's failing for me.

Using vstest.console.exe:

Using dotnet test:

[Evangelink]

I confirm that the issue is coming from a change that was introduce in 2.2.7 where some logging mechanism was added without checking for how app domain and assembly resolvers were setup.

[avivanoff]

With the same test project? It's failing for me.

@Evangelink, yes, everything is exactly the same. Our Azure DevOps pipelines have been working fine with 2.2.10, the problems started after upgrading to 3.3.x/17.4.x with absolutely no other changes.

[Evangelink]

I have no idea how it's possible... My only guess is that it's an ordering issue and you happen to have the dll touched before... I can reproduce the issue 100% of the time and asked for confirmation from my colleagues and they have the same behaviour using your repro and any version after 2.2.6 (and they also have a success when using 2.2.6 or before).

There is an open PR that is fixing the issue but I am now wondering if there is something else going on.

[avivanoff]

@Evangelink, would it be possible to provide a NuGet package I can test?

[Evangelink]

Hey @avivanoff, you can either download the previews here:

• MSTest.TestAdapter 4.0.0-preview-20221222-4
• MSTest.TestFramework 4.0.0-preview-20221222-4

or you can add the following feed <add key="test-tools" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/test-tools/nuget/v3/index.json" /> and restore version 4.0.0-preview-20221222-4.

If you confirm everything is good for you, I will cherry-pick the commit and do a 3.0.2 release with it.

[avivanoff]

@Evangelink, it worked. Please, release 3.0.2 as soon as possible. Thank you.

[avivanoff]

@Evangelink, any chance of getting 3.0.2 today?

[Evangelink]

@avivanoff Thanks for the confirmation! It's quite unlikely to be done today (I am CET based so it's already end of the day for me) and I have a few days of vacations so most likely it will be done on Tuesday. I will try to make it now but not sure if CI will be on my side :)

[avivanoff]

@Evangelink, let’s try.

[Evangelink]

Yep, branch created and pipeline started: https://github.com/microsoft/testfx/tree/rel/3.0.2

[avivanoff]

@Evangelink, one unit test failed. Re-run the pipeline?

[Evangelink]

For the release, we rely on some internal pipeline (that is still ongoing and is much slower because of many extra steps - validation, signing...).

[Evangelink]

Internal build failed, I triggered a new build but it's going to take too much time to complete so sadly it will have to wait.

@avivanoff Feel free to keep the 4.0.0 preview on your pipeline waiting for the 3.0.2.

[avivanoff]

@Evangelink, no one else can run the build?

[GaTechThomas]

I have this same issue in my Azure Pipelines, using vstest@2, no direct mstest reference. It's coming from MSTestAdapter:

MSTestAdapter failed to discover tests in class

I'm able to make the error stop by using the runsettings disablement of AppDomain, but the tests need the AppDomain, so we need another option.

Is it possible to make the vstest@2 task use the new version (or the old version)?

This is the task I'm referring to:

https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/vstest-v2?view=azure-pipelines

[ndenkha]

Hi guys, I'm having the same issue on VS 2022 LTSC 17.8.10. I upgraded my VS last night along with installing Python extensions and my test project running on .Net Framework 4.8 started failing with the same issue. First, I tried updating the MSTest Nuget to the latest 3.4.0 and it didn't solve the issue, with/out DisableAppDomain. The only thing that worked for me is downgrading MSTest to 2.2.6

So, is there a valid fix to this?

[wangyoutian]

Hi guys, I'm having the same issue on VS 2022 LTSC 17.8.10. I upgraded my VS last night along with installing Python extensions and my test project running on .Net Framework 4.8 started failing with the same issue. First, I tried updating the MSTest Nuget to the latest 3.4.0 and it didn't solve the issue, with/out DisableAppDomain. The only thing that worked for me is downgrading MSTest to 2.2.6

So, is there a valid fix to this?

Oh, yes. I came across similar issue, and a scramble for finding some help brings me here. I have to downgrade from 3.4 to 2.2.10 to work around the issue; It seems an issue more than one years ago:

#1437 (comment)

,reappears.

[Evangelink]

I have created a different issue so we can investigate it, please add comments there.