Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] In assembly parallel #296

Merged
merged 33 commits into from
Nov 29, 2017
Merged
Show file tree
Hide file tree
Changes from 30 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
41c9616
WIP: In assembly parallelization at method level.
Oct 20, 2017
621da1b
WIP: Make ClassInitialize/Cleanup and AssemblyInitialize/Cleanup sync…
Oct 20, 2017
87c41cb
Run test class/assembly cleanups at end of execution.
Oct 20, 2017
912cc54
Test Class parallelization.
AbhitejJohn Oct 20, 2017
95e912a
Added E2E tests.
AbhitejJohn Oct 20, 2017
9377a41
Merge pull request #2 from AbhitejJohn/in-assembly-parallel
AbhitejJohn Oct 20, 2017
550b70d
Runsettings support.
AbhitejJohn Oct 20, 2017
560411e
Made Parallel platform independent.
AbhitejJohn Oct 21, 2017
6262793
Merge pull request #3 from AbhitejJohn/in-assembly-parallel
AbhitejJohn Oct 21, 2017
69a2392
nit refactoring and test fixes.
AbhitejJohn Oct 21, 2017
9bd125b
Another nit fix.
AbhitejJohn Oct 21, 2017
c442547
Fixing test script to fail on test failures.
AbhitejJohn Oct 21, 2017
93f3d8b
Some cleanup and a few UTs.
AbhitejJohn Oct 22, 2017
26d68c8
Merge pull request #4 from AbhitejJohn/in-assembly-parallel
AbhitejJohn Oct 22, 2017
fa60189
Meged latest master.
AbhitejJohn Oct 30, 2017
4f1dcef
few PR comments
AbhitejJohn Oct 30, 2017
8c5b283
Merge pull request #5 from AbhitejJohn/in-assembly-parallel
AbhitejJohn Nov 10, 2017
ec72649
Changes based on spec.
AbhitejJohn Nov 20, 2017
053969f
Exception handling for settings, unit tests and Disable Parallelization.
AbhitejJohn Nov 20, 2017
4b37cba
Merge pull request #6 from AbhitejJohn/in-assembly-parallel
AbhitejJohn Nov 20, 2017
5b51599
Fixed up E2E tests.
AbhitejJohn Nov 21, 2017
7555ed1
Adding TestExecutionManager unit tests.
AbhitejJohn Nov 21, 2017
7821679
Merge pull request #7 from AbhitejJohn/in-assembly-parallel
AbhitejJohn Nov 21, 2017
ee36c06
Reverting the UWP sdk upgrade since PR build machines are not on the …
AbhitejJohn Nov 21, 2017
59882e6
Merge pull request #8 from AbhitejJohn/in-assembly-parallel
AbhitejJohn Nov 21, 2017
a2f5bfc
Updating localized files.
AbhitejJohn Nov 21, 2017
1b9a78b
Merge pull request #9 from AbhitejJohn/in-assembly-parallel
AbhitejJohn Nov 21, 2017
fcecc7c
Staging changes for STA/MTA of parallized tests and a test.
AbhitejJohn Nov 21, 2017
d26387a
The TaskScheduler.FromCurrentSyncContext did not work. It appears tha…
AbhitejJohn Nov 24, 2017
575ccc1
Merge pull request #10 from AbhitejJohn/in-assembly-parallel
AbhitejJohn Nov 24, 2017
4a4637e
Merge branch 'master' into in-assembly-parallel
smadala Nov 27, 2017
2f9c7ba
Really feel bad adding a sleep to a test - This can probably be handl…
AbhitejJohn Nov 28, 2017
256151e
Merge pull request #11 from AbhitejJohn/in-assembly-parallel
AbhitejJohn Nov 28, 2017
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 62 additions & 2 deletions TestFx.sln
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@

Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio 15
VisualStudioVersion = 15.0.26403.0
VisualStudioVersion = 15.0.26228.9
MinimumVisualStudioVersion = 10.0.40219.1
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "src", "src", "{FF8B1B72-55A1-4FFE-809E-7B79323ED8D0}"
EndProject
Expand Down Expand Up @@ -107,8 +107,11 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Automation.CLI", "test\E2ET
{98BA6D2C-1F3D-4636-8E1D-D4932B7A253D} = {98BA6D2C-1F3D-4636-8E1D-D4932B7A253D}
{BBC99A6B-4490-49DD-9C12-AF2C1E95576E} = {BBC99A6B-4490-49DD-9C12-AF2C1E95576E}
{B0FCE474-14BC-449A-91EA-A433342C0D63} = {B0FCE474-14BC-449A-91EA-A433342C0D63}
{4004757A-0080-4410-B90A-6169B20F151B} = {4004757A-0080-4410-B90A-6169B20F151B}
{4004757A-0082-4410-B94A-6166B20F153C} = {4004757A-0082-4410-B94A-6166B20F153C}
{0C522C84-B5AF-4DB6-96A2-9EFC15C88589} = {0C522C84-B5AF-4DB6-96A2-9EFC15C88589}
{0C139DA9-AA81-43A0-8625-7A873C0E2873} = {0C139DA9-AA81-43A0-8625-7A873C0E2873}
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220} = {CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}
{293597CE-CCF7-4C09-9ADD-17BF34C6E86F} = {293597CE-CCF7-4C09-9ADD-17BF34C6E86F}
{C6AC27D2-975B-4CF3-B16B-9C17907B2268} = {C6AC27D2-975B-4CF3-B16B-9C17907B2268}
EndProjectSection
Expand Down Expand Up @@ -159,7 +162,11 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "FxExtensibilityTestProject"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "PlatformServices.NetCore", "src\Adapter\PlatformServices.NetCore\PlatformServices.NetCore.csproj", "{2979FAA1-4786-43E8-B2E1-F01A0A91FECC}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "PlatformServices.NetCore.Unit.Tests", "test\UnitTests\PlatformServices.NetCore.Unit.Tests\PlatformServices.NetCore.Unit.Tests.csproj", "{655001BB-C00F-4EF7-A324-7F7C48656D35}"
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "PlatformServices.NetCore.Unit.Tests", "test\UnitTests\PlatformServices.NetCore.Unit.Tests\PlatformServices.NetCore.Unit.Tests.csproj", "{655001BB-C00F-4EF7-A324-7F7C48656D35}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "ParallelMethodsTestProject", "test\E2ETests\TestAssets\ParallelTestMethods\ParallelMethodsTestProject.csproj", "{4004757A-0082-4410-B94A-6166B20F153C}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "ParallelClassesTestProject", "test\E2ETests\TestAssets\ParallelTestClass\ParallelClassesTestProject.csproj", "{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}"
EndProject
Global
GlobalSection(SharedMSBuildProjectFiles) = preSolution
Expand Down Expand Up @@ -854,6 +861,54 @@ Global
{655001BB-C00F-4EF7-A324-7F7C48656D35}.Release|x64.Build.0 = Release|Any CPU
{655001BB-C00F-4EF7-A324-7F7C48656D35}.Release|x86.ActiveCfg = Release|Any CPU
{655001BB-C00F-4EF7-A324-7F7C48656D35}.Release|x86.Build.0 = Release|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Code Analysis Debug|Any CPU.ActiveCfg = Debug|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Code Analysis Debug|Any CPU.Build.0 = Debug|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Code Analysis Debug|ARM.ActiveCfg = Debug|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Code Analysis Debug|ARM.Build.0 = Debug|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Code Analysis Debug|x64.ActiveCfg = Debug|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Code Analysis Debug|x64.Build.0 = Debug|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Code Analysis Debug|x86.ActiveCfg = Debug|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Code Analysis Debug|x86.Build.0 = Debug|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Debug|Any CPU.Build.0 = Debug|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Debug|ARM.ActiveCfg = Debug|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Debug|ARM.Build.0 = Debug|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Debug|x64.ActiveCfg = Debug|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Debug|x64.Build.0 = Debug|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Debug|x86.ActiveCfg = Debug|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Debug|x86.Build.0 = Debug|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Release|Any CPU.ActiveCfg = Release|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Release|Any CPU.Build.0 = Release|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Release|ARM.ActiveCfg = Release|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Release|ARM.Build.0 = Release|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Release|x64.ActiveCfg = Release|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Release|x64.Build.0 = Release|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Release|x86.ActiveCfg = Release|Any CPU
{4004757A-0082-4410-B94A-6166B20F153C}.Release|x86.Build.0 = Release|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Code Analysis Debug|Any CPU.ActiveCfg = Debug|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Code Analysis Debug|Any CPU.Build.0 = Debug|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Code Analysis Debug|ARM.ActiveCfg = Debug|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Code Analysis Debug|ARM.Build.0 = Debug|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Code Analysis Debug|x64.ActiveCfg = Debug|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Code Analysis Debug|x64.Build.0 = Debug|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Code Analysis Debug|x86.ActiveCfg = Debug|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Code Analysis Debug|x86.Build.0 = Debug|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Debug|Any CPU.Build.0 = Debug|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Debug|ARM.ActiveCfg = Debug|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Debug|ARM.Build.0 = Debug|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Debug|x64.ActiveCfg = Debug|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Debug|x64.Build.0 = Debug|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Debug|x86.ActiveCfg = Debug|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Debug|x86.Build.0 = Debug|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Release|Any CPU.ActiveCfg = Release|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Release|Any CPU.Build.0 = Release|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Release|ARM.ActiveCfg = Release|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Release|ARM.Build.0 = Release|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Release|x64.ActiveCfg = Release|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Release|x64.Build.0 = Release|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Release|x86.ActiveCfg = Release|Any CPU
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220}.Release|x86.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
Expand Down Expand Up @@ -906,5 +961,10 @@ Global
{4004757A-0080-4410-B90A-6169B20F151B} = {D53BD452-F69F-4FB3-8B98-386EDA28A4C8}
{2979FAA1-4786-43E8-B2E1-F01A0A91FECC} = {24088844-2107-4DB2-8F3F-CBCA94FC4B28}
{655001BB-C00F-4EF7-A324-7F7C48656D35} = {F2D0BF2C-38F2-4244-80E3-4AAD1C3F4C89}
{4004757A-0082-4410-B94A-6166B20F153C} = {D53BD452-F69F-4FB3-8B98-386EDA28A4C8}
{CD0CA7CD-CED3-45FF-9F36-B1C8DF7A9220} = {D53BD452-F69F-4FB3-8B98-386EDA28A4C8}
EndGlobalSection
GlobalSection(ExtensibilityGlobals) = postSolution
SolutionGuid = {31E0F4D5-975A-41CC-933E-545B2201FAF9}
EndGlobalSection
EndGlobal
3 changes: 3 additions & 0 deletions src/Adapter/MSTest.CoreAdapter/Constants.cs
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ internal static class Constants

internal static readonly TestProperty DeploymentItemsProperty = TestProperty.Register("MSTestDiscoverer.DeploymentItems", DeploymentItemsLabel, typeof(KeyValuePair<string, string>[]), TestPropertyAttributes.Hidden, typeof(TestCase));

internal static readonly TestProperty DoNotParallelizeProperty = TestProperty.Register("MSTestDiscoverer.DoNotParallelize", DoNotParallelizeLabel, typeof(bool), TestPropertyAttributes.Hidden, typeof(TestCase));

#endregion

#region Private Constants
Expand All @@ -56,6 +58,7 @@ internal static class Constants
private const string TestCategoryLabel = "TestCategory";
private const string PriorityLabel = "Priority";
private const string DeploymentItemsLabel = "DeploymentItems";
private const string DoNotParallelizeLabel = "DoNotParallelize";

#endregion
}
Expand Down
2 changes: 2 additions & 0 deletions src/Adapter/MSTest.CoreAdapter/Discovery/TypeEnumerator.cs
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,8 @@ internal UnitTestElement GetTestFromMethod(MethodInfo method, bool isDeclaredInT

testElement.TestCategory = this.reflectHelper.GetCategories(method);

testElement.DoNotParallelize = this.reflectHelper.IsDoNotParallelizeSet(method);

var traits = this.reflectHelper.GetTestPropertiesAsTraits(method);

var ownerTrait = this.reflectHelper.GetTestOwnerAsTraits(method);
Expand Down
82 changes: 45 additions & 37 deletions src/Adapter/MSTest.CoreAdapter/Execution/TestAssemblyInfo.cs
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,14 @@ public class TestAssemblyInfo
private MethodInfo assemblyCleanupMethod;

private MethodInfo assemblyInitializeMethod;
private object assemblyInfoExecuteSyncObject;

/// <summary>
/// Initializes a new instance of the <see cref="TestAssemblyInfo"/> class.
/// </summary>
internal TestAssemblyInfo()
{
this.assemblyInfoExecuteSyncObject = new object();
}

/// <summary>
Expand Down Expand Up @@ -124,20 +126,23 @@ public void RunAssemblyInitialize(TestContext testContext)
throw new NullReferenceException(Resource.TestContextIsNull);
}

// If assembly initialization is not done, then do it.
if (!this.IsAssemblyInitializeExecuted)
lock (this.assemblyInfoExecuteSyncObject)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should check for IsAssemblyInitializeExecuted before attempting to request lock.

{
try
// If assembly initialization is not done, then do it.
if (!this.IsAssemblyInitializeExecuted)
{
this.AssemblyInitializeMethod.InvokeAsSynchronousTask(null, testContext);
}
catch (Exception ex)
{
this.AssemblyInitializationException = ex;
}
finally
{
this.IsAssemblyInitializeExecuted = true;
try
{
this.AssemblyInitializeMethod.InvokeAsSynchronousTask(null, testContext);
}
catch (Exception ex)
{
this.AssemblyInitializationException = ex;
}
finally
{
this.IsAssemblyInitializeExecuted = true;
}
}
}

Expand Down Expand Up @@ -193,36 +198,39 @@ public string RunAssemblyCleanup()
return null;
}

try
lock (this.assemblyInfoExecuteSyncObject)
{
this.AssemblyCleanupMethod.InvokeAsSynchronousTask(null);

return null;
}
catch (Exception ex)
{
var realException = ex.InnerException ?? ex;

string errorMessage;

// special case AssertFailedException to trim off part of the stack trace
if (realException is AssertFailedException ||
realException is AssertInconclusiveException)
try
{
errorMessage = realException.Message;
this.AssemblyCleanupMethod.InvokeAsSynchronousTask(null);

return null;
}
else
catch (Exception ex)
{
errorMessage = StackTraceHelper.GetExceptionMessage(realException);
var realException = ex.InnerException ?? ex;

string errorMessage;

// special case AssertFailedException to trim off part of the stack trace
if (realException is AssertFailedException ||
realException is AssertInconclusiveException)
{
errorMessage = realException.Message;
}
else
{
errorMessage = StackTraceHelper.GetExceptionMessage(realException);
}

return string.Format(
CultureInfo.CurrentCulture,
Resource.UTA_AssemblyCleanupMethodWasUnsuccesful,
this.AssemblyCleanupMethod.DeclaringType.Name,
this.AssemblyCleanupMethod.Name,
errorMessage,
StackTraceHelper.GetStackTraceInformation(realException)?.ErrorStackTrace);
}

return string.Format(
CultureInfo.CurrentCulture,
Resource.UTA_AssemblyCleanupMethodWasUnsuccesful,
this.AssemblyCleanupMethod.DeclaringType.Name,
this.AssemblyCleanupMethod.Name,
errorMessage,
StackTraceHelper.GetStackTraceInformation(realException)?.ErrorStackTrace);
}
}
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT license. See LICENSE file in the project root for full license information.

namespace Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution
{
using System;
using System.Security;
using Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Helpers;
using Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel;

internal class TestAssemblySettingsProvider : MarshalByRefObject
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we build an abstraction for TestSource? There seem to be multiple data associated with a test assembly and it is currently fragmented. E.g. we pass in source, maintain a state isDeploymentCompleted etc.. And now we're adding parallelization related configuration. These are effectively a per source property.

The TestSource could be serializable.

{
private ReflectHelper reflectHelper;

public TestAssemblySettingsProvider()
: this(new ReflectHelper())
{
}

internal TestAssemblySettingsProvider(ReflectHelper reflectHelper)
{
this.reflectHelper = reflectHelper;
}

/// <summary>
/// Returns object to be used for controlling lifetime, null means infinite lifetime.
/// </summary>
/// <returns>
/// The <see cref="object"/>.
/// </returns>
[SecurityCritical]
public override object InitializeLifetimeService()
{
return null;
}

internal TestAssemblySettings GetSettings(string source)
{
var testAssemblySettings = new TestAssemblySettings();

// Load the source.
var testAssembly = PlatformServiceProvider.Instance.FileOperations.LoadAssembly(source, isReflectionOnly: false);

var parallelizeAttribute = this.reflectHelper.GetParallelizeAttribute(testAssembly);

if (parallelizeAttribute != null)
{
testAssemblySettings.Workers = parallelizeAttribute.Workers;
testAssemblySettings.Scope = parallelizeAttribute.Scope;

if (testAssemblySettings.Workers == 0)
{
testAssemblySettings.Workers = Environment.ProcessorCount;
}
}

testAssemblySettings.CanParallelizeAssembly = !this.reflectHelper.IsDoNotParallelizeSet(testAssembly);

return testAssemblySettings;
}
}
}
Loading