Fix a deadlock during ConPTY shutdown #14160

lhecker · 2022-10-07T18:03:12Z

Problem:

Calling RundownAndExit tries to flush out the last frame from VtEngine
VtEngine indirectly calls RundownAndExit if the pipe is gone via VtIo
RundownAndExit is called by other parts of OpenConsole
RundownAndExit must be [[noreturn]] for most parts of OpenConsole
VtIo itself has a mutex ensuring orderly shutdown
In short, doing a thread safe orderly shutdown requires us to hold both,
a lock in RundownAndExit and VtIo at the same time, but while other parts
need a blocking RundownAndExit, VtIo needs a non-blocking one
Refactoring the code to use optionally non-blocking RundownAndExit
requires refactoring and might prove to be just as buggy

Solution:

Simply don't call RundownAndExit in VtEngine at all
In case the write pipe breaks:
- VtEngine will close the handle
- The client should notice that their read pipe is broken and
  close their write pipe sooner or later
- Once we notice that our read pipe is broken, we call RundownAndExit
- RundownAndExit might call back into VtEngine but
  without a pipe it won't do anything
In case the read pipe breaks or any other part calls RundownAndExit:
- We call RundownAndExit
- RundownAndExit might call back into VtEngine and depending on whether
  the write pipe is broken or not it will simply write into it or ignore it

Closes #14132
Pretty sure this also applies to #1810

Validation Steps Performed

Open 5 tabs and run MSYS2's bash --login in each of them
Enter-VsDevShell in another tab
Close window
5 tab processes are killed instantly, 1 after ~3s ✅
Replace conhost with OpenConsole via sfpcopy
Launch Dozens of Git Bash tabs in VS Code
Close them randomly
Remaining ones still work, processes are gone ✅

DHowett

[2:20 PM] Dustin Howett
did you git blame the line that introduced the shutdown mutex to see if it fixed a bug?
[2:20 PM] Dustin Howett
if so, can you test against that bug?

Requesting changes during due diligence :)

src/host/VtIo.cpp

DHowett · 2022-10-07T19:25:35Z

src/host/VtIo.cpp

+    // ConsoleInputThreadProcWin32 calls VtIo::CreatePseudoWindow,
+    // which calls CreateWindowExW, which causes a WM_SIZE message.
+    // In short, this function might be called before _pVtRenderEngine exists.
+    // See PtySignalInputThread::CreatePseudoWindow().


since the window message loop happens on a different thread, what happens if we get a window message for visibility while we are shutting down? we will come in here, lock console, and stall... right?

The only function that blocks is LockConsole. If it does, that's fine since RundownAndExit() will hold the console lock during shutdown until the process exits. No code depends on the window thread to proceed during the shutdown so there's no "circular dependency" as is the case with VtEngine.

lhecker · 2022-10-07T20:02:00Z

Requesting changes during due diligence :)

When I suggested that lock, my concern was that the caller of SetWindowVisibility() runs on a different thread than the code that calls CloseOutput(). Accessing _pVtRenderEngine without mutex would then lead to a race condition.

But now that I've read the code a bit closer, I've realized that all we do is .release() the object and not close it. So we might as well not reset or release it at all, thus avoid needing another mutex.

DHowett

I didn't submit a single annoying nit comment from a week ago lol

src/host/VtInputThread.cpp

DHowett · 2022-10-11T23:22:47Z

src/host/VtIo.cpp

-    // thread can't trigger a CloseOutput and release the
-    // _pVtRenderEngine out from underneath us.
-    std::lock_guard<std::mutex> lk(_shutdownLock);
+    auto& gci = ::Microsoft::Console::Interactivity::ServiceLocator::LocateGlobals().getConsoleInformation();


Actually, i am suddenly worried this is a deadlock factory. We call SetWindowVisibility off the I/O thread when Terminal says "I got focus." The PseudoWindow itself may also take the console lock to send Terminal messages. Is there a condition that will result in Terminal receiving and processing a message while the console lock is held, and then trying to dump a focus message back into the VtIo pipe?

The previous code already acquired the console lock here. All I did was move the if condition inside the locked area. If we have such a bug then we should have had it before this PR is/was merged.

DHowett

Let's give it a shot then!

lhecker · 2022-10-12T14:04:27Z

src/host/VtInputThread.cpp

@@ -94,7 +93,7 @@ VtInputThread::VtInputThread(_In_ wil::unique_hfile hPipe,
 DWORD WINAPI VtInputThread::StaticVtInputThreadProc(_In_ LPVOID lpParameter)
 {
    const auto pInstance = reinterpret_cast<VtInputThread*>(lpParameter);
-    return pInstance->_InputThread();
+    pInstance->_InputThread();


FYI _InputThread is now [[noreturn]] because CloseInput is [[noreturn]]. Hence, no return is needed, despite this function "returning" a DWORD.

miniksa · 2022-10-13T21:05:46Z

Test with WSL calling out to this to run a Windows executable (ensure you receive all output from the Windows binary back to the Linux half and that the process exits correctly).
Also test with VScode's integrated terminal.

If both are "fine" then this should be "fine".

miniksa · 2022-10-13T21:07:21Z

Hate that I'm saying this, but you should probably check this change on a OneCore edition as well :|

miniksa

Code looks fine to me. It's just a matter of the effects. You will want to test the various integrations I listed if you didn't already (building this as conhost.exe and subbing into the various Windows OS images) before merging.

DHowett · 2022-10-13T22:07:02Z

Test with WSL calling out to this to run a Windows executable (ensure you receive all output from the Windows binary back to the Linux half and that the process exits correctly).

A good way would be to run, like, bc or something... using WSL interop.

I'll help L with a onecore build. It can be.. let's say tedious. :D

lhecker · 2022-10-13T23:08:12Z

I've tested two scenarios now:

Opening a pwsh tab running wsl inside, running pwsh inside that and then running bc.exe.
Then close the tab.
Running an .exe that exits really quickly.

In both cases both the old code as well as the new code caused OpenConsole to exit immediately. However one time I had a leftover/hung conhost.exe in headless mode hanging around out of nowhere. The good news is that this is the OS' conhost and not the one I patched via sfpcopy, so whatever this does: This doesn't make it worse. (I ran my OpenConsole as conhost via sfpcopy already to test VS Code and that worked fine as well.)

DHowett · 2022-10-13T23:16:44Z

I can't get sirep working, but it does tear down properly in onecore (not tested onecoreuap yet); I can't get WSL to exhibit an output truncation bug either.

ghost · 2022-10-13T23:17:12Z

Hello @DHowett!

Because this pull request has the AutoMerge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.

Problem: * Calling `RundownAndExit` tries to flush out the last frame from `VtEngine` * `VtEngine` indirectly calls `RundownAndExit` if the pipe is gone via `VtIo` * `RundownAndExit` is called by other parts of OpenConsole * `RundownAndExit` must be `[[noreturn]]` for most parts of OpenConsole * `VtIo` itself has a mutex ensuring orderly shutdown * In short, doing a thread safe orderly shutdown requires us to hold both, a lock in `RundownAndExit` and `VtIo` at the same time, but while other parts need a blocking `RundownAndExit`, `VtIo` needs a non-blocking one * Refactoring the code to use optionally non-blocking `RundownAndExit` requires refactoring and might prove to be just as buggy Solution: * Simply don't call `RundownAndExit` in `VtEngine` at all * In case the write pipe breaks: * `VtEngine` will close the handle * The client should notice that their read pipe is broken and close their write pipe sooner or later * Once we notice that our read pipe is broken, we call `RundownAndExit` * `RundownAndExit` might call back into `VtEngine` but without a pipe it won't do anything * In case the read pipe breaks or any other part calls `RundownAndExit`: * We call `RundownAndExit` * `RundownAndExit` might call back into `VtEngine` and depending on whether the write pipe is broken or not it will simply write into it or ignore it Closes #14132 Pretty sure this also applies to #1810 ## Validation Steps Performed * Open 5 tabs and run MSYS2's `bash --login` in each of them * `Enter-VsDevShell` in another tab * Close window * 5 tab processes are killed instantly, 1 after ~3s ✅ * Replace conhost with OpenConsole via sfpcopy * Launch Dozens of Git Bash tabs in VS Code * Close them randomly * Remaining ones still work, processes are gone ✅ (cherry picked from commit 1774cfd) Service-Card-Id: 86174637 Service-Version: 1.16

Problem: * Calling `RundownAndExit` tries to flush out the last frame from `VtEngine` * `VtEngine` indirectly calls `RundownAndExit` if the pipe is gone via `VtIo` * `RundownAndExit` is called by other parts of OpenConsole * `RundownAndExit` must be `[[noreturn]]` for most parts of OpenConsole * `VtIo` itself has a mutex ensuring orderly shutdown * In short, doing a thread safe orderly shutdown requires us to hold both, a lock in `RundownAndExit` and `VtIo` at the same time, but while other parts need a blocking `RundownAndExit`, `VtIo` needs a non-blocking one * Refactoring the code to use optionally non-blocking `RundownAndExit` requires refactoring and might prove to be just as buggy Solution: * Simply don't call `RundownAndExit` in `VtEngine` at all * In case the write pipe breaks: * `VtEngine` will close the handle * The client should notice that their read pipe is broken and close their write pipe sooner or later * Once we notice that our read pipe is broken, we call `RundownAndExit` * `RundownAndExit` might call back into `VtEngine` but without a pipe it won't do anything * In case the read pipe breaks or any other part calls `RundownAndExit`: * We call `RundownAndExit` * `RundownAndExit` might call back into `VtEngine` and depending on whether the write pipe is broken or not it will simply write into it or ignore it Closes #14132 Pretty sure this also applies to #1810 ## Validation Steps Performed * Open 5 tabs and run MSYS2's `bash --login` in each of them * `Enter-VsDevShell` in another tab * Close window * 5 tab processes are killed instantly, 1 after ~3s ✅ * Replace conhost with OpenConsole via sfpcopy * Launch Dozens of Git Bash tabs in VS Code * Close them randomly * Remaining ones still work, processes are gone ✅ (cherry picked from commit 1774cfd) Service-Card-Id: 86178271 Service-Version: 1.15

ghost · 2022-10-18T21:51:04Z

🎉Windows Terminal v1.15.2874 has been released which incorporates this pull request.:tada:

Handy links:

#14160 didn't fix #14132 entirely. There seems to be a race condition left where (on my system) 9 out of 10 times everything works correctly, but sometimes OpenConsole exits, while pwsh and bash keep running. My leading theory is that the new code is exiting OpenConsole faster than the old code. This prevents clients from calling console APIs, etc. causing them to get stuck. The old code (and new code) calls `ExitProcess` when the ConPTY pipes break and I think this is wrong: In conhost when you close the window we only call `CloseConsoleProcessState` via the `WM_CLOSE` event and that's it. Solution: Remove the call to `RundownAndExit` for ConPTY. During testing I found that continuously printing text inside msys2 will cause child processes to only exit slowly one by one every 5 seconds. This happens because `CloseConsoleProcessState` calls `HandleCtrlEvent` without holding the console lock. This creates a race condition where most of the time the console IO thread is the one picking up the `CTRL_CLOSE_EVENT`. But that's problematic because the `CTRL_CLOSE_EVENT` leads to a `ConsoleControl` call of type `ConsoleEndTask` which calls back into conhost's IO thread and so you got the IO thread waiting on itself to respond. Solution: Don't race conditions. ## Validation Steps Performed * `Enter-VsDevShell` and close the tab Everything exits after 5s ✅ * Run msys2 bash from within pwsh and close the tab Everything exits instantly ✅ * Run `cat bigfile.txt` and close the tab Everything exits instantly ✅ * Patch `conhost.exe` with `sfpcopy`, as well as `KernelBase.dll` with the recent changes to `winconpty`, then launch and exit shells and applications via VS Code's terminal ✅ * On the main branch without this modification remove the call to `TriggerTeardown` in `RundownAndExit` (this speeds up the shutdown). Run (msys2's) `bash.exe --login` and hold enter and then press Ctrl+Shift+W simultaneously. The tab should close and randomly OpenConsole should exit early while pwsh/bash keep running. Then retry this with this branch and observe how the child processes don't stick around forever anymore. ✅

Problem: * Calling `RundownAndExit` tries to flush out the last frame from `VtEngine` * `VtEngine` indirectly calls `RundownAndExit` if the pipe is gone via `VtIo` * `RundownAndExit` is called by other parts of OpenConsole * `RundownAndExit` must be `[[noreturn]]` for most parts of OpenConsole * `VtIo` itself has a mutex ensuring orderly shutdown * In short, doing a thread safe orderly shutdown requires us to hold both, a lock in `RundownAndExit` and `VtIo` at the same time, but while other parts need a blocking `RundownAndExit`, `VtIo` needs a non-blocking one * Refactoring the code to use optionally non-blocking `RundownAndExit` requires refactoring and might prove to be just as buggy Solution: * Simply don't call `RundownAndExit` in `VtEngine` at all * In case the write pipe breaks: * `VtEngine` will close the handle * The client should notice that their read pipe is broken and close their write pipe sooner or later * Once we notice that our read pipe is broken, we call `RundownAndExit` * `RundownAndExit` might call back into `VtEngine` but without a pipe it won't do anything * In case the read pipe breaks or any other part calls `RundownAndExit`: * We call `RundownAndExit` * `RundownAndExit` might call back into `VtEngine` and depending on whether the write pipe is broken or not it will simply write into it or ignore it Closes #14132 Pretty sure this also applies to #1810 ## Validation Steps Performed * Open 5 tabs and run MSYS2's `bash --login` in each of them * `Enter-VsDevShell` in another tab * Close window * 5 tab processes are killed instantly, 1 after ~3s ✅ * Replace conhost with OpenConsole via sfpcopy * Launch Dozens of Git Bash tabs in VS Code * Close them randomly * Remaining ones still work, processes are gone ✅

Problem: * Calling `RundownAndExit` tries to flush out the last frame from `VtEngine` * `VtEngine` indirectly calls `RundownAndExit` if the pipe is gone via `VtIo` * `RundownAndExit` is called by other parts of OpenConsole * `RundownAndExit` must be `[[noreturn]]` for most parts of OpenConsole * `VtIo` itself has a mutex ensuring orderly shutdown * In short, doing a thread safe orderly shutdown requires us to hold both, a lock in `RundownAndExit` and `VtIo` at the same time, but while other parts need a blocking `RundownAndExit`, `VtIo` needs a non-blocking one * Refactoring the code to use optionally non-blocking `RundownAndExit` requires refactoring and might prove to be just as buggy Solution: * Simply don't call `RundownAndExit` in `VtEngine` at all * In case the write pipe breaks: * `VtEngine` will close the handle * The client should notice that their read pipe is broken and close their write pipe sooner or later * Once we notice that our read pipe is broken, we call `RundownAndExit` * `RundownAndExit` might call back into `VtEngine` but without a pipe it won't do anything * In case the read pipe breaks or any other part calls `RundownAndExit`: * We call `RundownAndExit` * `RundownAndExit` might call back into `VtEngine` and depending on whether the write pipe is broken or not it will simply write into it or ignore it Closes #14132 Pretty sure this also applies to #1810 ## Validation Steps Performed * Open 5 tabs and run MSYS2's `bash --login` in each of them * `Enter-VsDevShell` in another tab * Close window * 5 tab processes are killed instantly, 1 after ~3s ✅ * Replace conhost with OpenConsole via sfpcopy * Launch Dozens of Git Bash tabs in VS Code * Close them randomly * Remaining ones still work, processes are gone ✅ (cherry picked from commit 1774cfd)

#14160 didn't fix #14132 entirely. There seems to be a race condition left where (on my system) 9 out of 10 times everything works correctly, but sometimes OpenConsole exits, while pwsh and bash keep running. My leading theory is that the new code is exiting OpenConsole faster than the old code. This prevents clients from calling console APIs, etc. causing them to get stuck. The old code (and new code) calls `ExitProcess` when the ConPTY pipes break and I think this is wrong: In conhost when you close the window we only call `CloseConsoleProcessState` via the `WM_CLOSE` event and that's it. Solution: Remove the call to `RundownAndExit` for ConPTY. During testing I found that continuously printing text inside msys2 will cause child processes to only exit slowly one by one every 5 seconds. This happens because `CloseConsoleProcessState` calls `HandleCtrlEvent` without holding the console lock. This creates a race condition where most of the time the console IO thread is the one picking up the `CTRL_CLOSE_EVENT`. But that's problematic because the `CTRL_CLOSE_EVENT` leads to a `ConsoleControl` call of type `ConsoleEndTask` which calls back into conhost's IO thread and so you got the IO thread waiting on itself to respond. Solution: Don't race conditions. ## Validation Steps Performed * `Enter-VsDevShell` and close the tab Everything exits after 5s ✅ * Run msys2 bash from within pwsh and close the tab Everything exits instantly ✅ * Run `cat bigfile.txt` and close the tab Everything exits instantly ✅ * Patch `conhost.exe` with `sfpcopy`, as well as `KernelBase.dll` with the recent changes to `winconpty`, then launch and exit shells and applications via VS Code's terminal ✅ * On the main branch without this modification remove the call to `TriggerTeardown` in `RundownAndExit` (this speeds up the shutdown). Run (msys2's) `bash.exe --login` and hold enter and then press Ctrl+Shift+W simultaneously. The tab should close and randomly OpenConsole should exit early while pwsh/bash keep running. Then retry this with this branch and observe how the child processes don't stick around forever anymore. ✅ (cherry picked from commit b6b1ff8) Service-Card-Id: 87207831 Service-Version: 1.15

#14160 didn't fix #14132 entirely. There seems to be a race condition left where (on my system) 9 out of 10 times everything works correctly, but sometimes OpenConsole exits, while pwsh and bash keep running. My leading theory is that the new code is exiting OpenConsole faster than the old code. This prevents clients from calling console APIs, etc. causing them to get stuck. The old code (and new code) calls `ExitProcess` when the ConPTY pipes break and I think this is wrong: In conhost when you close the window we only call `CloseConsoleProcessState` via the `WM_CLOSE` event and that's it. Solution: Remove the call to `RundownAndExit` for ConPTY. During testing I found that continuously printing text inside msys2 will cause child processes to only exit slowly one by one every 5 seconds. This happens because `CloseConsoleProcessState` calls `HandleCtrlEvent` without holding the console lock. This creates a race condition where most of the time the console IO thread is the one picking up the `CTRL_CLOSE_EVENT`. But that's problematic because the `CTRL_CLOSE_EVENT` leads to a `ConsoleControl` call of type `ConsoleEndTask` which calls back into conhost's IO thread and so you got the IO thread waiting on itself to respond. Solution: Don't race conditions. ## Validation Steps Performed * `Enter-VsDevShell` and close the tab Everything exits after 5s ✅ * Run msys2 bash from within pwsh and close the tab Everything exits instantly ✅ * Run `cat bigfile.txt` and close the tab Everything exits instantly ✅ * Patch `conhost.exe` with `sfpcopy`, as well as `KernelBase.dll` with the recent changes to `winconpty`, then launch and exit shells and applications via VS Code's terminal ✅ * On the main branch without this modification remove the call to `TriggerTeardown` in `RundownAndExit` (this speeds up the shutdown). Run (msys2's) `bash.exe --login` and hold enter and then press Ctrl+Shift+W simultaneously. The tab should close and randomly OpenConsole should exit early while pwsh/bash keep running. Then retry this with this branch and observe how the child processes don't stick around forever anymore. ✅ (cherry picked from commit b6b1ff8) Service-Card-Id: 87207832 Service-Version: 1.16

ghost · 2022-12-14T20:45:13Z

🎉Windows Terminal Preview v1.16.3463.0 and v1.16.3464.0 has been released which incorporates this pull request.:tada:

Handy links:

2 new ConPTY APIs were added as part of this commit: * `ClosePseudoConsoleTimeout` Complements `ClosePseudoConsole`, allowing users to override the `INFINITE` wait for OpenConsole to exit with any arbitrary timeout, including 0. * `ConptyReleasePseudoConsole` This releases the `\Reference` handle held by the `HPCON`. While it makes launching further PTY clients via `PROC_THREAD_ATTRIBUTE_PSEUDOCONSOLE` impossible, it does allow conhost/OpenConsole to exit naturally once all console clients have disconnected. This makes it unnecessary to having to monitor the exit of the spawned shell/application, just to then call `ClosePseudoConsole`, while carefully continuing to read from the output pipe. Instead users can now just read from the output pipe until it is closed, knowing for sure that no more data will come or clients connect. This is especially useful in combination with `ClosePseudoConsoleTimeout` and a timeout of 0, to allow conhost/OpenConsole to exit asynchronously. These new APIs are used to fix an array of bugs around Windows Terminal exiting either too early or too late. They also make usage of the ConPTY API simpler in most situations (when spawning a single application and waiting for it to exit). Depends on #13882, #14041, #14160, #14282 Closes #4564 Closes #14416 Closes MSFT-42244182 ## Validation Steps Performed * Calling `FreeConsole` in a handoff'd application closes the tab ✅ * Create a .bat file containing only `start /B cmd.exe`. If WT stable is the default terminal the tab closes instantly ✅ With these changes included the tab stays open with a cmd.exe prompt ✅ * New ConPTY tests ✅

Fix a deadlock during ConPTY shutdown

83d14f0

lhecker requested a review from DHowett October 7, 2022 18:04

DHowett requested changes Oct 7, 2022

View reviewed changes

ghost added Needs-Author-Feedback The original author of the issue/PR needs to come back and respond to something and removed Needs-Author-Feedback The original author of the issue/PR needs to come back and respond to something labels Oct 7, 2022

lhecker added 2 commits October 7, 2022 22:02

Fix issues, address feedback

2782af5

Remove obsolete test

e1a8d06

DHowett reviewed Oct 10, 2022

View reviewed changes

src/host/VtInputThread.cpp Outdated Show resolved Hide resolved

Address feedback

06a56fa

carlos-zamora added a commit that referenced this pull request Oct 11, 2022

PRE-MERGE #14160 Fix a deadlock during ConPTY shutdown

957c548

DHowett reviewed Oct 11, 2022

View reviewed changes

DHowett approved these changes Oct 11, 2022

View reviewed changes

lhecker added the Needs-Second It's a PR that needs another sign-off label Oct 12, 2022

ghost requested review from zadjii-msft, PankajBhojwani and carlos-zamora October 12, 2022 14:03

lhecker commented Oct 12, 2022

View reviewed changes

lhecker mentioned this pull request Oct 12, 2022

Seemingly random crash #14192

Closed

miniksa approved these changes Oct 13, 2022

View reviewed changes

DHowett added the AutoMerge Marked for automatic merge by the bot when requirements are met label Oct 13, 2022

ghost merged commit 1774cfd into main Oct 13, 2022

ghost deleted the dev/lhecker/14132-shutdown-deadlock branch October 13, 2022 23:17

lhecker mentioned this pull request Oct 22, 2022

Wait for clients to exit on ConPTY shutdown #14282

Merged

lhecker mentioned this pull request Dec 1, 2022

Reported hang when minimized during a razzle build #14134

Closed

zadjii-msft mentioned this pull request Dec 5, 2022

ClosePseudoConsole API hanging #1810

Closed

lhecker mentioned this pull request Dec 13, 2022

Add support for start /B and FreeConsole #14544

Merged

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix a deadlock during ConPTY shutdown #14160

Fix a deadlock during ConPTY shutdown #14160

lhecker commented Oct 7, 2022 •

edited

Loading

DHowett left a comment

DHowett Oct 7, 2022

lhecker Oct 7, 2022

lhecker commented Oct 7, 2022

DHowett left a comment

DHowett Oct 11, 2022

lhecker Oct 11, 2022

DHowett left a comment

lhecker Oct 12, 2022

miniksa commented Oct 13, 2022

miniksa commented Oct 13, 2022

miniksa left a comment

DHowett commented Oct 13, 2022

lhecker commented Oct 13, 2022

DHowett commented Oct 13, 2022

ghost commented Oct 13, 2022

ghost commented Oct 18, 2022

ghost commented Dec 14, 2022

Fix a deadlock during ConPTY shutdown #14160

Fix a deadlock during ConPTY shutdown #14160

Conversation

lhecker commented Oct 7, 2022 • edited Loading

Validation Steps Performed

DHowett left a comment

Choose a reason for hiding this comment

DHowett Oct 7, 2022

Choose a reason for hiding this comment

lhecker Oct 7, 2022

Choose a reason for hiding this comment

lhecker commented Oct 7, 2022

DHowett left a comment

Choose a reason for hiding this comment

DHowett Oct 11, 2022

Choose a reason for hiding this comment

lhecker Oct 11, 2022

Choose a reason for hiding this comment

DHowett left a comment

Choose a reason for hiding this comment

lhecker Oct 12, 2022

Choose a reason for hiding this comment

miniksa commented Oct 13, 2022

miniksa commented Oct 13, 2022

miniksa left a comment

Choose a reason for hiding this comment

DHowett commented Oct 13, 2022

lhecker commented Oct 13, 2022

DHowett commented Oct 13, 2022

ghost commented Oct 13, 2022

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (@msftbot) and give me an instruction to get started! Learn more here.

ghost commented Oct 18, 2022

ghost commented Dec 14, 2022

lhecker commented Oct 7, 2022 •

edited

Loading

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.