Flaky Clipboard Unit Tests

[weltkante]

.NET Core Version:
master

Have you experienced this same bug with .NET Framework?:
no

Problem description:
Noticed as a flaky test:

System.Windows.Forms.Tests.TextBoxBaseTests.TextBoxBase_Paste_InvokeNotEmpty_Success [FAIL]
Assert.Equal() Failure
� (pos 1)
Expected: abc
Actual: a
� (pos 1)
Stack Trace:
F:\workspace_work\1\s\src\System.Windows.Forms\tests\UnitTests\TextBoxBaseTests.cs(5957,0): at System.Windows.Forms.Tests.TextBoxBaseTests.TextBoxBase_Paste_InvokeNotEmpty_Success()

Test seems to assume clipboard is empty/nonfunctional (there is at least one more test along these, should review for more clipboard-dependent tests)

Also any clipboard tests need to be [Collection("Sequential")] to avoid overlapping with other clipboard tests. Is it worth making the whole test class sequential just due to that single test? should the test be moved?

Expected behavior:
Clipboard based tests initialize clipboard to be in known state and are marked for sequential execution

Minimal repro:
none, flaky test

[RussKie]

I wonder if we could employ Trait decorations, e.g. something like:

        [Trait(Category.Synchronous, Category.Clipboard)] // arbitrary strings represented by consts
        [WinFormsFact]
        public void TextBoxBase_Paste_InvokeNotEmpty_Success()
        {
            // ...
        }

...and then update our CI configs to run tests separately:

winforms/eng/ci.yml

Lines 98 to 107 in 0f90af4

	# Run Unit Tests
	# Tests are run with /m:1 to work around https://github.com/tonerdo/coverlet/issues/364
	- script: eng\cibuild.cmd
	-test
	-configuration $(_BuildConfig)
	$(_OfficialBuildIdArgs)
	/p:Coverage=$(_Coverage)
	/bl:$(BUILD.SOURCESDIRECTORY)\artifacts\log\$(_BuildConfig)\Test.binlog
	/m:1
	displayName: Run Unit Tests

        # Run Unit Tests
        # Tests are run with /m:1 to work around https://github.com/tonerdo/coverlet/issues/364
        - script: eng\cibuild.cmd
            -test
            -configuration $(_BuildConfig)
            $(_OfficialBuildIdArgs)
            /p:Coverage=$(_Coverage)
            /bl:$(BUILD.SOURCESDIRECTORY)\artifacts\log\$(_BuildConfig)\Test.binlog
            /m:1
           -trait "Synchronous=Clipboard"
          displayName: Run Unit Tests (Synchronous)

        - script: eng\cibuild.cmd
            -test
            -configuration $(_BuildConfig)
            $(_OfficialBuildIdArgs)
            /p:Coverage=$(_Coverage)
            /bl:$(BUILD.SOURCESDIRECTORY)\artifacts\log\$(_BuildConfig)\Test.binlog
            /m:1
           -notrait "Synchronous=Clipboard"
          displayName: Run Unit Tests

Got some inspiration from http://www.brendanconnolly.net/organizing-tests-with-xunit-traits/

[weltkante]

Failed test Clipboard_SetText_InvokeStringTextDataFormat_GetReturnsExpected in CI run of PR #3473:

Assert.Equal() Failure\r\n ↓ (pos 0)\r\nExpected: text\r\nActual: \r\n ↑ (pos 0)

Since clipboard tests are not [Collection("Sequential")] its possible another test (from a different class) cleared the clipboard and interfered with this test.

[weltkante]

Failed test TextBoxBase_Paste_InvokeNotEmpty_Success in CI run of PR #3511

[RussKie]

FYI I started looking into adding traits

[JeremyKuhne]

Failed again:

TextBoxBase_Paste_InvokeNotEmpty_Success
Assert.Equal() Failure\r\n           ↓ (pos 1)\r\nExpected: abc\r\nActual:   a\r\n           ↑ (pos 1)
 at System.Windows.Forms.Tests.TextBoxBaseTests.TextBoxBase_Paste_InvokeNotEmpty_Success() in /_/src/System.Windows.Forms/tests/UnitTests/TextBoxBaseTests.cs:line 5957

[RussKie]

One way of tracking rolling failures: https://runfo.azurewebsites.net/search/tests/?q=started%3A~14+definition%3A267+kind%3Arolling

[RussKie]

Related to #6269

[JeremyKuhne]

This is a matter of making sure everything that might touch the clipboard is in a non-parallelized collection and ensuring that everything has sufficient retries. I've had some success in the clipboard tests in my thirtytwo project with them running stably. See: https://github.com/JeremyKuhne/thirtytwo/blob/main/src/thirtytwo_tests/ClipboardTestCollection.cs

[Tanya-Solyanik]

@LeafShi1 - you already fixed this in #11606, right?

[LeafShi1]

TextBoxBase_Paste_InvokeNotEmpty_Success

Yes