Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable parallel mode for test collections and reenable flaky tests. #8772

Closed
wants to merge 26 commits into from

Conversation

dreddy-work
Copy link
Member

@dreddy-work dreddy-work commented Mar 9, 2023

Testing theory of "disabling test collection parallelism may be fixing flaky tests".

changes:

  • Serialize execution of unit tests on build agents.
  • Add delay of 500ms after input simulator input action before proceeding with next action/validation.
  • Prevent disposing Cursor that is not owned by process.
  • Add checks for Cursor/Handle creation and fail fast.
  • Block tests against hsplit.cur and vsplit.cur. They need special attention. Possibly a bug that has been there since Framework.
  • Remove collection definition that is not necessary anymore.
  • Add retry count (1).
  • Reenable tests from Epic: Epic: WinForms runtime test failures/flakyness #8607.
  • Tracking issue created to replace InputSimulator for a more permeant and reliable solution.

Related tests re-enabled:

fixes #8750 fixes #8748 fixes #8747 fixes #8746 fixes #8740 fixes #8734 fixes #8726 fixes #8723 fixes #8709 fixes #8682 fixes #8635 fixes #8634 fixes #8632 fixes #8619 fixes #8612 fixes #8611 fixes #8609 fixes #8604 fixes #8602 fixes #8601 fixes #8520 fixes #8521 fixes #8373 fixes #8223 fixes #7428 fixes #7407 fixes #7297 fixes #6610 fixes #6654 fixes #6714 fixes #6729 fixes #4212 fixes #3918 fixes #3647 fixes #3456 fixes #2856

Microsoft Reviewers: Open in CodeFlow

@dreddy-work dreddy-work requested a review from a team as a code owner March 9, 2023 19:40
@ghost ghost assigned dreddy-work Mar 9, 2023
@dreddy-work dreddy-work added the 🚫 * NO-MERGE * The PR is not ready for merge yet (see discussion for detailed reasons) label Mar 9, 2023
@dreddy-work
Copy link
Member Author

vsplit.cur and hsplit.cur are consistent in resulting disposed handle ( or handle may never be created).

@dreddy-work dreddy-work changed the title [Test] Disable parallel mode for test collections. Disable parallel mode for test collections and reenable flaky tests. Mar 13, 2023
@dreddy-work
Copy link
Member Author

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 2 pipeline(s).

@dreddy-work
Copy link
Member Author

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 2 pipeline(s).

Directory.Build.props Outdated Show resolved Hide resolved
@dreddy-work
Copy link
Member Author

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 2 pipeline(s).

@dreddy-work
Copy link
Member Author

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 2 pipeline(s).

Copy link
Member

@JeremyKuhne JeremyKuhne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A comment nit from me.

@@ -0,0 +1,5 @@
{
"maxParallelThreads": 1,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Making everything serial should have a follow up issue for tracking if it doesn't already. It is unfortunate that we can't add comments in json. Doing this will facilitate adding poorly written tests making it harder and harder to go back to parallel testing.

@@ -28,6 +28,10 @@
<EnableTrimAnalyzer Condition="$(IsTestProject) or $(IsAnalyzerProject) or $(MSBuildProjectDirectory.Contains('\tests\'))">false</EnableTrimAnalyzer>
</PropertyGroup>

<PropertyGroup Condition="'$(IsTestProject)' == 'true'">
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be a good place to have a comment for our re-parallelization tracking issue.

@ghost ghost added the 📭 waiting-author-feedback The team requires more information from the author label Mar 15, 2023
@dreddy-work
Copy link
Member Author

Closing as we are trying to remove all delays and a part of these changes are taken separately.

@ghost ghost removed the 📭 waiting-author-feedback The team requires more information from the author label Mar 16, 2023
@dreddy-work dreddy-work deleted the dev/dreddy/fixtestParellelmode branch March 27, 2023 16:54
@ghost ghost locked as resolved and limited conversation to collaborators Apr 26, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.