Introduce a mechanism for passing through DCS data strings (#9307)

This PR introduces a mechanism via which DCS data strings can be passed
through directly to the dispatch method that will be handling them, so
the data can be processed as it is received, rather than being buffered
in the state machine. This also simplifies the way string termination is
handled, so it now more closely matches the behaviour of the original
DEC terminals.

* Initial support for DCS sequences was introduced in PR #6328.
* Handling of DCS (and other) C1 controls was added in PR #7340.
* This is a prerequisite for Sixel (#448) and Soft Font (#9164) support.

The way this now works, a `DCS` sequence is dispatched as soon as the
final character of the `VTID` is received. Based on that ID, the
`OutputStateMachineEngine` should forward the call to the corresponding
dispatch method, and its the responsibility of that method to return an
appropriate handler function for the sequence.

From then on, the `StateMachine` will pass on all of the remaining bytes
in the data string to the handler function. When a data string is
terminated (with `CAN`, `SUB`, or `ESC`), the `StateMachine` will pass
on one final `ESC` character to let the handler know that the sequence
is finished. The handler can also end a sequence prematurely by
returning false, and then all remaining data bytes will be ignored.

Note that once a `DCS` sequence has been dispatched, it's not possible
to abort the data string. Both `CAN` and `SUB` are considered valid
forms of termination, and an `ESC` doesn't necessarily have to be
followed by a `\` for the string terminator. This is because the data
string is typically processed as it's received. For example, when
outputting a Sixel image, you wouldn't erase the parts that had already
been displayed if the data string is terminated early.

With this new way of handling the string termination, I was also able to
simplify some of the `StateMachine` processing, and get rid of a few
states that are no longer necessary. These changes don't apply to the
`OSC` sequences, though, since we're more likely to want to match the
XTerm behavior for those cases (which requires a valid `ST` control for
the sequence to be accepted).

## Validation Steps Performed

For the unit tests, I've had to make a few changes to some of the
`OutputEngineTests` to account for the updated `StateMachine`
processing. I've also added a new `StateMachineTest` to confirm that the
data strings are correctly passed through to the string handler under
all forms of termination.

To test whether the framework is actually usable, I've been working on
DRCS Soft Font support branched off of this PR, and haven't encountered
any problems. To test the throughput speed, I also hacked together a
basic Sixel parser, and that seemed to perform reasonably well.

Closes #7316

src/terminal/parser/IStateMachineEngine.hpp

@@ -20,6 +20,8 @@ namespace Microsoft::Console::VirtualTerminal
     class IStateMachineEngine
     {
     public:
+        using StringHandler = std::function<bool(const wchar_t)>;
+
         virtual ~IStateMachineEngine() = 0;
         IStateMachineEngine(const IStateMachineEngine&) = default;
         IStateMachineEngine(IStateMachineEngine&&) = default;
@@ -36,6 +38,7 @@ namespace Microsoft::Console::VirtualTerminal
         virtual bool ActionEscDispatch(const VTID id) = 0;
         virtual bool ActionVt52EscDispatch(const VTID id, const VTParameters parameters) = 0;
         virtual bool ActionCsiDispatch(const VTID id, const VTParameters parameters) = 0;
+        virtual StringHandler ActionDcsDispatch(const VTID id, const VTParameters parameters) = 0;
 
         virtual bool ActionClear() = 0;
 

src/terminal/parser/InputStateMachineEngine.cpp

@@ -446,6 +446,21 @@ bool InputStateMachineEngine::ActionCsiDispatch(const VTID id, const VTParameter
     return success;
 }
 
+// Routine Description:
+// - Triggers the DcsDispatch action to indicate that the listener should handle
+//      a control sequence. Returns the handler function that is to be used to
+//      process the subsequent data string characters in the sequence.
+// Arguments:
+// - id - Identifier of the control sequence to dispatch.
+// - parameters - set of numeric parameters collected while parsing the sequence.
+// Return Value:
+// - the data string handler function or nullptr if the sequence is not supported
+IStateMachineEngine::StringHandler InputStateMachineEngine::ActionDcsDispatch(const VTID /*id*/, const VTParameters /*parameters*/) noexcept
+{
+    // DCS escape sequences are not used in the input state machine.
+    return nullptr;
+}
+
 // Routine Description:
 // - Triggers the Ss3Dispatch action to indicate that the listener should handle
 //      a control sequence. These sequences perform various API-type commands

src/terminal/parser/InputStateMachineEngine.hpp

@@ -146,6 +146,8 @@ namespace Microsoft::Console::VirtualTerminal
 
         bool ActionCsiDispatch(const VTID id, const VTParameters parameters) override;
 
+        StringHandler ActionDcsDispatch(const VTID id, const VTParameters parameters) noexcept override;
+
         bool ActionClear() noexcept override;
 
         bool ActionIgnore() noexcept override;

src/terminal/parser/OutputStateMachineEngine.cpp

@@ -636,6 +636,24 @@ bool OutputStateMachineEngine::ActionCsiDispatch(const VTID id, const VTParamete
     return success;
 }
 
+// Routine Description:
+// - Triggers the DcsDispatch action to indicate that the listener should handle
+//      a control sequence. Returns the handler function that is to be used to
+//      process the subsequent data string characters in the sequence.
+// Arguments:
+// - id - Identifier of the control sequence to dispatch.
+// - parameters - set of numeric parameters collected while parsing the sequence.
+// Return Value:
+// - the data string handler function or nullptr if the sequence is not supported
+IStateMachineEngine::StringHandler OutputStateMachineEngine::ActionDcsDispatch(const VTID /*id*/, const VTParameters /*parameters*/) noexcept
+{
+    StringHandler handler = nullptr;
+
+    _ClearLastChar();
+
+    return handler;
+}
+
 // Routine Description:
 // - Triggers the Clear action to indicate that the state machine should erase
 //      all internal state.

src/terminal/parser/OutputStateMachineEngine.hpp

@@ -39,6 +39,8 @@ namespace Microsoft::Console::VirtualTerminal
 
         bool ActionCsiDispatch(const VTID id, const VTParameters parameters) override;
 
+        StringHandler ActionDcsDispatch(const VTID id, const VTParameters parameters) noexcept override;
+
         bool ActionClear() noexcept override;
 
         bool ActionIgnore() noexcept override;