Discussion: Find a way to make OSC7 work

[zadjii-msft]

I think we all agree that we want a sequence that'll work for a client app to set the current working directory of the Terminal. How exactly that's enabled is still up for discussion. There's already existing user scripts, tools, etc that are using OSC7 to set the CWD of the Terminal. However, those who are using it are using it to set the path relative to the "container" - e.g. in WSL, the "container root" is /, but as far as the OS is concerned, that path is actually \\wsl$\DistroName\.

I'm aware we've already got #3158 tracking adding support for OSC7, but there's a ton of folks that are all following that discussion as the "enable opening a new tab with the same directory", and I'd rather keep this discussion more focused.

(That being said - the rest of this post is basically my verbatim notes, turned into a thread. I've got some though experiments below for what we could do here, but none of them are particularly good.)

For the completeness of discussion, let's also refer to

• #8166 OSC 9;9 - the ConEmu sequence for setting the working directory
• #7526 Support for the file:// URI when emitted as part of OSC 8 hyperlinks
• #331 Users wanting to be able to drag/drop a file onto the Terminal running WSL, and have the pasted path be in WSL format, not Windows
• #592 Less importantly, but users also want to be able to set a
  startingDirectory as a WSL path, not just as a Windows path.

Let's imagine emitting this sequence in the following scenarios:

• From cmd.exe
• From powershell (Core?) on Windows
• From WSL
• From WSL running in a non-WSL profile
  • This scenario is important, because we can't use the profile to determine
    how we'd want to convert the path
• From anything over ssh
  • This most definitely won't work. Fortunately, we can use the hostname to
    prevent these sequences from being accepted. It's up to the user to make
    sure that the machine they're ssh'd to doesn't have the same hostname.
• From inside a Docker container
• from inside cygwin
• Inside tmux or screen, or any other ort of multiplexer that might need to get in between the client app and the terminal emulator
• When emitted by a remote machine connected to with the Azure Cloud Shell

OSC7 Conclusions

(I originally wrote this for #7668 (comment))

Basically, we've got two simultaneously incompatible issues. Either we:

• Support this sequence with no "smart" path translation. We'll need to
  literally use whatever directory they emit (assuming the hostname matches)
  • Users who are currently using OSC 7 in their *nix scripts and emitting file://$hostname/$PWD in WSL, cygwin, will get unexpected behavior. We'll try and treat $PWD as a path to a Windows directory - which might be a path that exists. When they try to open a new tab with that same directory, it'll open not in the distro the sequence originated in, but in the Windows filesystem. Not great!
• Find some way to magically translate paths provided by OSC7 into the correct Windows path. WSL clients won't need to change any of the scripts they were already using for this to "just work".
  • There's no heuristic way of doing this, without the client application providing some extra information to us. We can't inspect the process tree to figure out if the process that's emitting this sequence is a WSL process or a docker process or an ssh, and even if we could, then we wouldn't be able to correlate wsl.exe to the distro within WSL that's being used. Similarly, we can't use the profile to do this, because the user might be using WSL w/in their "Command Prompt" profile (for example).
  • We could introduce some other sequence for additional metadata to be provided. Something like "I know I'm a WSL, and I'm <distroName>, when I emit paths, handle them specially". If we did introduce such a new custom sequence, then we're not really faithfully implementing OSC7 now are we? For clients to work as they did before, they'd still need to add emitting this custom sequence

I think at the end of the day, I agree with @j4james. If we implement OSC7 as requiring a Windows-style path, then that'll necessitate that client applications will need to be modified to work right on Windows, and I hate that idea.

OSC9;9 - the ConEmu solution

This one doesn't seem that bad. Since it's a windows-first sequence, it doesn't need to know anything else about who's emitting the path. We can always assume it has been emitted as a Windows path. On the surface, this seems to work great, save for one scenario - ssh. This sequence doesn't accept any sort of hostname, it only takes the path. So if you were ssh'd into another machine, and that one emitted a OSC9;9, then there's no way of determining that sequence wasn't intended for the current machine.

I think I'm okay with us implementing this as-is for the time being though. It's not a perfect sequence, but it's good enough, and doesn't come with the compat baggage that OSC7 does. It can certainly be supported with the caveat "This won't work in an ssh session, or in any scenario where it's emitted by a remote connection. It will always be treated as a path on the local machine".

---

NOTE: The following is just brainstorming from me. I'm gonna file a new issue
to track the discussion of this proposal, because I don't want to further
muddle the discussion here.

Creating a custom sequence

While noodling on all this, I considered, "what would we want for a perfect set working directory sequence"? Why isn't OSC7 good enough for us? I think my analysis is that the thing emitting the sequence shouldn't need to know if it's inside a container, inside a WSL, running in an ssh session. The client application should be able to emit the same sequence regardless, and the Terminal should be able to just do the right thing.

The WSL path -> Windows path thing is tricky, but could we work around it somehow?

I'm thinking it might be useful to introduce some new "path metadata" sequence, that controls how paths emitted by OSC7 are handled. While this wouldn't strictly be in-line with the rest of the OSC7 spec, it would allow these sequences to be used otherwise unmodified on Windows, and might help make emitting these sequences in containers more useful. I'm thinking something like

Idea 1 - K/V Pairs used by the Terminal to reverse-engineer the path

OSC 9001 ; 1 ; key = value ST

Which lets the client specify a set of key, value pairs that should be used by the terminal when handling paths emitted by the client, and that's it.

This sequence would have to be emitted on startup of something like WSL, docker, or ssh, and they'll be responsible for setting these values for the terminal^[1]. So WSL would emit something like:

OSC 9001 ; 1 ; wslDistro = Ubuntu ST
OSC 9001 ; 1 ; hostname = %COMPUTERNAME% ST

The Terminal could then use these pieces of information to make smarter decisions about paths.

^[1]: Obviously, this could be emitted during login in .profile or something else, as a stopgap while we wait for WSL to emit something like this. I'd think users would want to use it sooner than later, and if there's a way for the shell to determine it's running in WSL, then fine, I won't stop users from writing new code that'll work both now and in the future.

Idea 1 conclusion

This is a terrible idea, because it requires the terminal emulator to be aware of any and all of these possible scenarios. Every terminal emulator out there would need to be manually updated to be able to handle WSL paths, cygwin paths, docker paths. If any other container-like environment was created, then the terminal emulator would also need to be updated to have other logic for that case as well.

Also, how does nesting these things work? How would the logic of wsl->cmd->cygwin work?

Idea 2 - Specify a commandline for path translation

The thing that's creating the nesting pushes/sets a path transform onto the Terminal. So wsl.exe would say

^[]9001;2;wslpath.exe -w ^[\

Then, when the client emits a path, we'd use the provided commandline to translate the path for us. We'll run the given commandline, and if the exit code is non-zero, we'll use the output of that commandline as the real path.

We'd need both a transform for outbound paths (paths emitted by the client) and inbound paths (paths created by the Terminal, e.g. the drag-drop path situation).

Idea 2 conclusion

Obviously this is a terrible idea. All you have to do is have your script emit ^[]9001;2;malicious.exe^[\ and now you've got a malicious exe running on every prompt, with the path that's being used by the user.

Also, how would this work with ssh? we'd filter the paths out by hostname, but what if you ssh'd to a Windows machine, then ran wsl in there? That WSL would inform the terminal to treat paths like they were WSL paths. However, WSL is running on the remote machine, not the local one! The terminal can't use that wslpath.exe

Idea 3 - Static path translation mapping

Static path translation table. The containerizer (wsl, cygwin, docker) sets up a list of paths and what they should be mapped to. E.g.

^[]9001;3;/mnt/c=C:\^[\
^[]9001;3;/=\\wsl$\Ubuntu\^[\

The above example would set up two pairs of mappings (/mnt/c -> C:/), and
(/ -> \\wsl$\Ubuntu\). When the terminal emulator receives a path from OSC7 or OSC8,
then it'll look through the table and find the mapping with the longest prefix
match for the given path.

So, if someone emits ^[]7;file://localhost/mnt/c/foo^[\, both / and /mnt/c
will match, but we'll take /mnt/c since that's the longer match. We'll then
substitute that prefix with whatever it's mapped to, so /mnt/c/foo becomes
c:/foo

We'll do the reverse when someone drag/drops a path onto the Terminal

• How would this work with ssh?
• How would this work with cygwin?
• How would this work with tmux?
• How would this work with multiple nesting?
  • Simple example - wsl -d Ubuntu, then pwsh.exe, then wsl -d Debian. Now
    we're in Debian, and we want the paths to be translated to \\wsl$\Debian\,
    not \\wsl$\Ubuntu\. That's fine, when WSL enters the Debian instance,
    it'll emit the sequence again, re-mapping / to the new root. However, if
    we exit Debian, now the pwsh.exe would need to reconfigure the roots
    itself.
  • Heck, even simpler - wsl -d Ubuntu then pwsh. Does pwsh then set up the mappings itself?
  • Is it the responsibility of the person emitting them to clean up after
    itself? That's almost never the case with other sequences. (consider SGR
    sequences. If you care, then you'll assume the child polluted the state, and
    reset.) People who care about the paths being remapped should assume that
    any child subprocess might change them.
    • So that would mean that after every command (like, as a part of the
      prompt), the shell would need to set up the mappings again. This is
      basically the same as just emitting a Windows path in OSC7 to begin
      with! Like, the only benefit we get is that child processes would
      inherit your mapping by default, but they could always overwrite it.
• How could this be used maliciously?
• Path slashes! If the path we drag is c:\foo\bar, then will we generate
  /mnt/c/foo\bar or /mnt/c/foo/bar?
  • Recall that cmd.exe can be a bit of a stickler about paths with \ vs /.
• Is = a valid path character in either *nix or NTFS? If it is, then
  we'll need something else as a delimiter that's not a path character

Idea 3 conclusion

IMO this is no worse than asking people to change their scripts to emit a
Windows path in OSC7. If they already have to change their scripts, then adding
this mapping script once in their profile or wherever will make it just work for
all the tools they're using, not just the ones they can change

Nevermind. This is just about as bad as just forcing people to use Windows paths in the prompt in the first place.

Idea 4 - stick the metadata right in the OSC7 sequence

Could we add additional parameters to OSC7, after the path, that a terminal emulator could use to say "Ah yes, this path might be inside a container, lets handle it specially"? This kinda admits that we won't be able to add any other sequence that'll make OSC7 work on its own without modification. If the takeaway from idea 3 was "no matter what we do, the shell needs to reconfigure the path mapping whenever a child exits", then I'm starting to worry that could be extrapolated to any design we make.

Idea 4 Conclusion

So I guess, if we're going to be asking people to modify their existing tools that use OSC7, then why not just use OSC7 as-is, with the caveat "If this is running on Windows, it's gotta emit the Windows-relative path"

Reference

The following post by @TBBle is incredibly useful

#7668 (comment)

Anyway, how to generate the URL is not the main issue for Windows Terminal, it's how to parse it. Ideally, we want to receive URLs that are already in the right format to just pass through urlmon or similar, and get back a UNC or filesystem path.

To get everything interoperating well, we need to handle (off hand):

• Paths from CMD, which are going to need to be file://localhost/$CWD, because CMD only knows DOS paths.
• Paths from PowerShell, which could be DOS path or UNC path. PowerShell can tell these apart, because $executionContext.SessionState.Path.CurrentLocation knows if it's on a 'Drive' or not, so this can easily do different things for either type of path.
• Paths from cygwin-based systems, e.g., Git Bash or MSYS2), which I believe can only see DOS paths, and can convert from cygwin paths using cygpath --mixed ${PWD}, and then it looks just like CMD, above.
• • Edit: Actually you can cd to UNC paths in the cygwin virtual filesystem, and in this case, the local PWD is the UNC path. Helpfully, cygpath --mixed ${PWD} works for all these cases anyway.
• • Edit: Also, using --mixed instead of --windows so the slashes are already correct, so the OSC 7 URL can be literally file://$(hostname)/$(cygpath --mixed $PWD) and it seems that'll always work.
• Paths from WSL, which from within the WSL session are Unix paths, and have two major mappings:
• • /mnt/X/... where X is a drive-letter should map through to a DOS path for that Drive.
• • Otherwise, it needs to to a UNC path //wsl$/${WSL_DISTRO_NAME}/${CWD}.
• • Edit: I kind-of wish WSL has something like cygpath now, to hide all the above.
• Paths from an SSH session, which we ideally want to ignore early. However, these may look identical to a UNC path's file:// URL. That might actually be desirable, if I am SSH'd to a machine on my LAN, and then launch a PowerShell tab and it does actually go into the same directory, via UNC. However, that'd be an unusual CIFs layout (NFS 3 worked that way though...) so I can't see this as being a common use-case.

Happily, it seems ${WSL_DISTRO_NAME} exists, and I assume it's inserted by the wsl shell launcher, so it is possible to generate the full correct file:// URL in one's PS1 env-var. This even means you can key off the presence of ${WSL_DISTRO_NAME} (or ${WSL_INTEROP} perhaps) to share the same OSC 7 generation code with out-of-WSL environments.

One possible approach to the above: is that if everyone generating OSC 7 commands for Windows paths agrees to only generate a modified-'legacy file:// URL' format, i.e. file://<gethostbyname()>///<UNC hostname>/<UNC path> or file://<gethostbyname()>/<DOS Path>, then WT could recognise when <gethostbyname()> matches the current $COMPUTERNAME, and replace it with localhost (making it a correct "legacy" file:// URL) before passing it through to the URL handler to get the actual working directory.

Then any other hostname, except 'localhost' and 'empty string', could be treated as 'remote' and ignored. For 'localhost' and 'empty string' hosts, we'll just trust the user isn't generating those from a remote host. There's nothing else we can do there.

This seems the closest in-spirit to the Freedesktop file:// URL specification, and means that URLs generated for OSC 7 on Windows will be semantically-similar to URLs generated for OSC 7 on UNIX-type systems, and can be parsed the same way.

Other tabs I had open:

• https://gitlab.freedesktop.org/terminal-wg/specifications/-/merge_requests/7
• Implement handling for OS escape code 7 alacritty/alacritty#2937
• https://stackoverflow.com/questions/14018280/how-to-get-a-process-working-dir-on-windows
• https://conemu.github.io/en/AnsiEscapeCodes.html#ConEmu_specific_OSC

---

I'll keep thinking about this, but let's use this as a place to continue thinking about how we might do right by the users here.

[j4james]

I don't know if I'm missing something, but this really doesn't seem like it should be that complicated to me. If the path looks something like X:\foo\bar or \\foo\bar, then we assume it's a normal Windows path which we can handle as is. If it's something like /foo/bar, then we assume it's going to require some translation based on the current profile.

If the profile command line is something like wsl -d DISTRO then I figured we could just prefix the path with \\wsl$\DISTRO (or something along those lines - I'm probably oversimplifying, but I'm sure there's an algorithm for this). Worst case we have to launch an instance of wslpath to do the translation for us and pipe back the result.

And I'm assuming cygwin could be handled in the same way. Either there's an algorithm we can use to work out the correct path ourselves, or we launch an instance of cygpath to do the translation for us.

If we're concerned about performance, we could always do something like translating just the root path, and then reuse that translation for future paths sharing the same root. But in most scenarios where the path is needed, the overhead probably wouldn't even been noticed.

Yes this is going to require a lot of special case code for every variant of shell we want to support, and it's not going to be pretty. But I'd much rather have that mess isolated in the Terminal instead of every application and shell script having to deal with it.

And remember this problem is specific to Windows Terminal. Once wsl supports Linux GUI applications, there are assumedly going to be plenty of other terminal apps running on wsl that won't have this problem. So if you're expecting applications to treat us differently, it's not good enough to detect that they're in a wsl instance - they need to tell that they're running in Windows Terminal, which is not something we've wanted to encourage.

And before anyone starts bringing up edge cases where this won't work, ask yourself whether that is a realistic scenario that users would expect to work? And there is always still the option of a custom sequence to handle special cases, or requiring the path be in a particular format. The important thing is that the out-of-the-box solution handles the 99% of cases where the user is just launching a regular shell, and having the terminal recognise a standard OSC 7 sequence to track the active directory.

[zadjii-msft]

The edge case that I immediately think of is running WSL from a "Command Prompt" profile. If you enter any one of these "containered" environments from a profile that doesn't match, then immediately any logic that uses the profile to do the translation isn't going to work. If that's an acceptable outcome, then yea, this is easy to do. Heck, we could just slap a "wslDistro": "Ubuntu" property in the dynamically generated profiles, and use the existence of that property let us identify if a profile is a WSL distro and which one it is.

I'm still holding out hope that there's a way to do it without that, but the hope is fading fast.

it's not good enough to detect that they're in a wsl instance - they need to tell that they're running in Windows Terminal, which is not something we've wanted to encourage.

You're absolutely right about that. I'll make sure to keep that in mind, that being in WSL does not guarantee that the path must be Windows-friendly, only that being in WT does. And I'm sure as heck not going to ship anything that forces developers to check if they're in WT.

[j4james]

The edge case that I immediately think of is running WSL from a "Command Prompt" profile.

Is this really that common though? And what would a user expect to happen in this scenario if they opened a "duplicate" tab? Are they expecting it to open a cmd shell starting in the directory they were in in the WSL shell? Even if we did know how to map that, it doesn't seem like a reasonable expectation to me. Unless the path happened to be /mnt/c/something it's probably not even feasible for a cmd shell. I'd think a more reasonable behaviour would be to use the last directory from the base profile, and that's something we could easily do.

[skyline75489]

In #7668 I choose to move the burden to shells, which make things easier on the terminal side. And yes it can properly handle “running WSL from a "Command Prompt" profile”, since it’s profile-agnostic. James is leaning towards keeping the ugliness inside terminal, in a case-by-case fashion to handle the path transformation in WSL, Cygwin, etc.

One of the reasons why I like the shell approach, is that I feel the ugliness is relatively smaller than the terminal approach. I mean it’s bash/zsh, right? People have already been putting everything in it since those shells were born. The other reason is that during the discussion in #7668 I found that the path transformation isn’t just simple string replacement. To transform the path competently, we need wslpath, which is inside the WSL environment. This just adds even more ugliness in the terminal approach.

Now that I think of it, neither of them is a perfect solution. The tricky part of is the nature of WSL. It’s a complete Linux environment yet has the ability to interact with the Windows world. It’s so special that none of the VT sequences was ready to handle it.

In conclusion, WSL is, after all, a Microsoft product. If we choose the terminal approach, I think it’s fair to handle it in WT, to make life easier for people. I’m not feeling the same for Cygwin or other kinds of profiles, though.

[j4james]

I mean it’s bash/zsh, right?

No, it's potentially used in other applications too. For example, egmontkob added an option to Midnight Commander that would make it report it's current directory via the OSC 7 sequence too. If we don't implement OSC 7 correctly then we just aren't going to be able to interoperate with apps like that. And if we aren't benefiting from interoperability, then why bother implementing the sequence in the first place?

[DHowett]

if we aren't benefiting from interop ... why bother

Fully agree. We should accept what apps generate and attempt to determine, to the best of our knowledge, how to interpret that.

If this means that our autogenerated WSL distribution profiles act better[1] than "User ran wsl.exe from cmd and now something acts silly", I think that I'm willing to accept it.

1: (assuming we do some profile magic to make them "aware" of their namespace prefix)

[skyline75489]

Great! I think we have an agreement that the terminal approach is the correct way to handle OSC 7.

Still I’d like to make ConEmu’s OSC 9;9 to work. This sequence needs integration from shell at the time it was born. And it’s clearly Windows-first. I’d like to think it a pioneer on our way towards a more complete OSC 7 solution, which presumably takes longer time.

I want people to have a working even though not perfect solution. I’ll copy #7668 to another PR that uses OSC 9;9. Together with all the shell scripts, people can enjoy the CWD feature if they are willing to tweak the shells. Sounds good?

[zadjii-msft]

I think we're all in agreement here. Glad this discussion was able to bear fruit after all! I'll close this thread now that we have a path forward with `OSC7. Thanks everyone!