Ctrl-Space inserts quotes using the German keyboard layout

[mardukbp]

Describe the bug

In PowerShell 7.0.3 with the module PSReadline 2.1 and the following configuration

%HOMEDIR%\Documents\PowerShell\Microsoft.PowerShell_profile.ps1

Import-Module PSReadLine
set-psreadlineoption -predictionsource history

pressing Ctrl+Space inserts " when the German keyboard layout is selected.
With the Spanish layout this key chord is passed to PowerShell, which then
suggests a command based on the history (like fish shell).

Environment (please complete the following information):

• OS: Windows 10
• Version: wezterm 20201101-103216-403d002d

To Reproduce

Assuming wezterm is configured to execute PowerShell.

1. Install the above versions of PowerShell and PSReadline
2. Create the configuration script mentioned above
3. Select the German keyboard layout
4. Launch wezterm
5. Press Ctrl+Space

Configuration

local wezterm = require 'wezterm';

return {
   font = wezterm.font_with_fallback({
    "Consolas",
    "DengXian"
   }),
   font_size = 16.0,
   hide_tab_bar_if_only_one_tab = true,
   default_prog = {"pwsh.exe"},  --PowerShell 7
   keys = {
      {key="F", mods="SHIFT|CTRL", action=wezterm.action{Search={CaseInSensitiveString=""}}},
      {key="l", mods="ALT", action="ShowLauncher"},
    },
}

Expected behavior

The key chord Ctrl+Space should be passed to the shell or any other process running in wezterm.

Screenshots

None.

Session Recording

None.

Additional context

PowerShell 7 is cross-platform. However, the keybinding Ctrl+Space is only defined on Windows.

[wez]

I can confirm that this only appears to happen for me when using wezterm + pwsh. I couldn't reproduce this with cmd.exe or the windows terminal preview.

I don't really know anything about powershell, but this little program:

while (1) { echo $Host.UI.RawUI.ReadKey() }

appears to show keypresses and it looks like it is not interpreting the keypresses the same way under wezterm as under the others.

If you run wsl and run xxd and press CTRL-space, it shows that wezterm is correct emitting ASCII NUL for CTRL-space ^@.

Turning up debugging in wezterm itself I can see that it is definitely recognizing this keypress as CTRL-space and definitely sending ASCII NUL.

My best guess is that this is something funky with codepages; the new windows PTY layer has a lot of issues around this, and it isn't clear how much of those are terminal "host" side vs. "client" side. I was thinking of this one microsoft/terminal#1802 and that is in turned linked to microsoft/terminal#7777

@miniksa / @DHowett-MSFT: do you have guidance on how a terminal emulator should be sending input data to the PTY? Today wezterm writes a UTF-8 byte sequence. Is there something special that should be done by wezterm as part of setting up the PTY to ensure that this is interpreted correctly?

[DHowett]

ConPTY is always UTF-8 in and out. This may be related to changes we made in conhost that aren't yet available in a Windows build? I recall us doing something to disambiguate Ctrl+Shift+2 from Ctrl+Space, but I cannot find it on account of I'm on my phone 😄

[zadjii-msft]

I'm pretty sure what you're seeing here is actually part of what was solved with microsoft/terminal#4999. Ctrl+space is one of those tricky things, because in Win32, NUL != Ctrl+space, but in VT, Ctrl+space == NUL. See also microsoft/terminal#2865

[wez]

FWIW, wezterm bundles and uses a build of OpenConsole.exe and conpty.dll from this rev: microsoft/terminal@9045266

[wez]

Not on my windows machine right now, so I'm speculating: it might be worth trying to run that bundled openconsole.exe directly and see if CTRL-space behaves the same way there? I'm wondering if this might be something that needs consideration in pwsh.exe's input handling?

[wez]

microsoft/terminal#6454 sounds like exactly this scenario: DEU, Ctrl-space -> " in powershell

[mardukbp]

@wez I just downloaded OpenConsole.exe from the wezterm repo. Ctrl-Space works as expected using the German keyboard layout in pwsh.exe.

[wez]

After digging through the linked issues, I found that although they are marked as resolved the resolution requires that a terminal implement a proprietary extension; this really wasn't made clear in the issues.
https://github.com/microsoft/terminal/blob/main/doc/specs/%234999%20-%20Improved%20keyboard%20handling%20in%20Conpty.md

That spec suggests that conpty unconditionally sends ^[[?9001h to the terminal to request a win32 specific key encoding mode, however, I've not seen it sent into wezterm so something feels incomplete in the implementation.

Additionally, the extension is problematic because the encoded data is not something that eg: a linux based terminal can produce for a remote win32 host because the encoding requires knowing virtual key codes and scan codes from win32 keyboard and language selection. This is referred to as a hypothetical future but this is in fact the situation today for any terminal remoting in to the windows host. This also applies for win32 native wezterm: we use a portable high fidelity representation of the key press but there isn't a way to portably resolve the win32 specific encoding required to represent that key.

FWIW, I think it's a shame that the CSI-u encoding (referred to as libtickit in the linked spec) wasn't selected as that has support in neovim and iTerm2, resolves the ambiguous key encoding problem and doesn't require the remote terminal to have inner knowledge of opaque details like win32 keyboard layouts.

As a workaround:
I found that starting pwsh.exe with ENG selected and then switching to DEU allows ctrl-space to be interpreted correctly(!)

So the state of things for now is that this isn't fixable within wezterm given the current implementation in conpty.

[mardukbp]

I found that starting pwsh.exe with ENG selected and then switching to DEU allows ctrl-space to be interpreted correctly(!)

Yeah, I know. Thanks anyway.

So the state of things for now is that this isn't fixable within wezterm given the current implementation in conpty.

I don't understand how openconsole.exe is used by wezterm. If openconsole.exe does not have this issue, then why does
wezterm do?

[wez]

wezterm uses conpty.dll to create a pty. Due to the way that those things work in windows, that causes openconsole.exe to be spawned in the background to act as a server for processing pty activity. When run in that mode it doesn't spawn a UI; it just processes data going across the pty pipes.

when openconsole.exe has a UI, it performs key input processing and encodes the results as appropriate input records, and powershell can use the win32 console APIs to directly access those. When run without a UI it doesn't (and cannot!) do those things and has to convert to/from the underlying win32 structures.

The issue here is that in the DEU keymap the standard terminal encoding for CTRL-Space is ambiguous, and that the way that the pty layer wants to resolve the ambiguity is by requiring the terminal emulator to use a proprietary, non-portable encoding for keyboard input to disambiguate it.

[mardukbp]

I see. Thank you so much for taking the time to explain!

As a workaround I defined a different shortcut for the completion menu in my $profile file:

Set-PSReadLineKeyHandler -Key Ctrl+r -Function MenuComplete

[wez]

xtermjs/xterm.js#2357 may have some useful stuff to follow up on around the win32-input-mode.

[DHowett]

Wow! You might be the first external developer to use W32IM! Please let us know -- we never made it public in the Windows SDK so that we could have some time to get community feedback 😄

[wez]

@DHowett my initial impression is that it is very win32 focused, which is kinda the point, but I'm left feeling uneasy about:

• I no longer have control over how keys are actually encoded to VT aware apps. (well, I can choose not to W32IM encode a given key, but that sounds nasty). I believe that this means that neither TERM=xterm nor TERM=wezterm are 100% accurate any more, and there isn't a way to reflect that to the apps running inside the terminal, because we don't know if W32IM is enabled until after we've started everything up and can't retroactively change the TERM environment. This obviously doesn't matter for classic win32 console apps, but it matters more for VT apps.
• remoting/tunneling scenarios where !windows and windows interact. I feel like there are good odds that an issue in this area will crop up. I left in a switch to disable W32IM just in case.
• Compatibility with potential future support of https://sw.kovidgoyal.net/kitty/keyboard-protocol/ in wezterm, although this might be more of a general "I hope the escape sequences are passed through" concern, especially because we re-discovered that undercurl escapes are swallowed by conpty today in our element channel

So, the good news is: nothing concrete to call out, but I don't feel that this is really the protocol I'd love to have. There's lots of legacy stuff we all need to deal with around terminals, both unix and win32, so I can live with not loving it :-)

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.