Some of the unittests are flaky in CI

[zadjii-msft]

FYI, the current build failure is just the unit tests timing out, which seems to happen quite often in the CI build. I don't think there's actually anything wrong with the code itself.

Originally posted by @j4james in #3943 (comment)

A case in point build: link to ADO. (EDIT: Okay this is really bad now )

This has been happening for a couple weeks now, where sometimes a test run in CI will hang. This time it happened somewhere around these tests:

  √ TestResize
  √ TestCursorVisibility [15ms]
  √ ConptyOutputTestCanary [15ms]
  √ SimpleWriteOutputTest [15ms]
  √ WriteTwoLinesUsesNewline
  √ WriteAFewSimpleLines
##[error]The operation was canceled.
Finishing: Run Unit Tests

This has happened in the past before and was notoriously hard to pin down - last time it had to do with how VtIo/VtRenderer was getting dtor'd. Hopefully we can figure out what's causing this.

As an additional note - there's vso_ut.cmd in the repo that attempts to run the unittests the same way ADO does, but that script hasn't been updated in a while.

[j4james]

FYI, PR #4295 caused the unit tests to time out yesterday as well (link to ADO). The exact location where it stops is slightly different, but it looks to me like the test execution order is not guaranteed - I'm guessing because it's running multiple tests in parallel? If that's the case, it may be an interaction between two test running at the same time that is the source of the issue.

[DHowett-MSFT]

Yanking triage- we see this phenomenon everywhere, and we should close on why it happens.

[j4james]

Anyone wanting to try and reproduce this locally, I think this is what the test_cmd variable needs to be updated to in the vso_ut.cmd script to match the CI server:

set test_cmd=%VSTEST_PATH% ^
 "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\ConAdapter.Unit.Tests.dll" ^
 "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\ConParser.Unit.Tests.dll" ^
 "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\Conhost.Interactivity.Win32.Unit.Tests.dll" ^
 "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\Conhost.Unit.Tests.dll" ^
 "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\Terminal.Core.Unit.Tests.dll" ^
 "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\TextBuffer.Unit.Tests.dll" ^
 "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\Types.Unit.Tests.dll" ^
 "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\UnitTests_TerminalApp\Terminal.App.Unit.Tests.dll" ^
 "%OPENCON%\bin\%ARCH%\%_LAST_BUILD_CONF%\til.Unit.Tests.dll" ^
 /Settings:"%OPENCON%\src\unit.tests.%ARCH%.runsettings" ^
 /EnableCodeCoverage ^
 /logger:trx ^
 /TestAdapterPath:"%OPENCON%" ^
 %*

Also note that you'll likely have to update the VSTEST_PATH to match whatever version of Visual Studio you have installed.

That said, I've run this test probably dozens of times now, sometimes multiple instances at the same time, and I've not yet found a way to make it hang.

[zadjii-msft]

Welp, one of the builds from #4382 still got stuck, so that one wasn't the fix.

[miniksa]

#4384 is where I've been playing around and trying to fix this.

I've identified that Terminal.Core.Unit.Tests.dll is the one that appears to be causing the problem.

EDIT:

• I ran it tons of times with just the tests. When I have every test but that one, it's fine. When I reintroduce that one, it hangs.

[miniksa]

OK, I also ran it with only the terminal core tests and it hangs. So it's that guy. I'll try to dig in.