TCP mode?

[keithw]

Allow degenerate use of TCP and HTTP CONNECT modes for people who can't use UDP.

[jhnlsn]

I would also like to see something like this as well. UDP has been disable on our network.

[sourcefrog]

I wonder if you could in fact just run mosh's delta-updating protocol over the ssh channel, which will be even more likely to just fit in with people's firewalls, and still possibly faster than plain ssh. Maybe it's a separate issue.

[andreasvc]

sourcefrog's idea is better, SSH is already required so using that connection reduces firewall issues to zero. It would be neat if UDP connectivity would be autodetected with fallback to SSH.

[saurik]

@andreasvc The problem with using the SSH connection is that once it dies, in order to reestablish it, you will need to re-authenticate with the server using the SSH authentication mechanisms. At that point, you honestly are going to get most of the remaining benefits of mosh by just using autossh (an old and venerable project that has been maintained over many years) and attaching to a remote screen/tmux session.

To do this, you can use a command like the one below. The main problem with it is that autossh has an irritating timing fallback algorithm when it fails; so, you open your laptop, it starts trying to reconnect, and the attempts get slower and slower before your networking even comes online, and so sometimes you have to wait 10-30 seconds after your Internet kicks back in before it reconnects. Fixing that, however, is obviously a trivial patch.

function cysh() {
    AUTOSSH_GATETIME=0 autossh -M 0 -t -e none \
        -o ServerAliveInterval=2 -o ServerAliveCountMax=4 \
        $1 screen -dr $2
}

[andreasvc]

@saurik: I don't know about "most". I think the predictive echo of mosh is the most important benefit, which compensates for latency. So I think mosh-over-ssh would be useful.

[saurik]

@andreasvc That ability is totally independent of networking: simply mosh to localhost and your terminal will have type-ahead prediction on it; it is the ability to drop intermediate results that makes that something to tie together with a networking layer (otherwise mosh could simply be a patch to tmux or screen that smooths the latency of any application run underneath it, remote or local).

[andreasvc]

OK predictive echo was the wrong example; but what about sourcefrog's suggestion, to "run mosh's delta-updating protocol over the ssh channel", isn't that possible? If not, why? I understand that UDP has advantages, but maintaining state on both ends makes for a more efficient protocol with TCP/SSH as well, I think. Such a watered-down mosh version would be very useful with firewalls that only allow TCP port 22, and by ensuring mosh works wherever ssh works it becomes a drop-in replacement.

[keithw]

@saurik Well, not exactly -- the early versions of mosh did have client-only local echo as you describe. But unfortunately we need a (a) fast and (b) accurate way to detect false-positive and true-positive predictions, and so we ended up needing to do the current server-side "echo ack" mechanism to have a smooth experience, because of network jitter and other reasons. So mosh to localhost and then ssh remotehost will produce glitches you won't see with mosh remotehost.

[keithw]

My current thinking of how we will do mosh-over-ssh (possibly -over-HTTP-proxy) is that we will just run a local proxy that will take UDP packets (or Unix domain datagrams) and pipe them over SSH and turn them into UDP (or Unix domain datagrams) on the server side. That way we can just treat the whole SSH connection like an EV-DO link or any other link that might have varying RTT, and get most of the benefits of the mosh timing and varying frame rate.

[saurik]

@keithw I am quite interested in that; I would have expected there to be know way to tell the difference between those two environments. If I type a character into mosh-client, it is sent to a remote system and received by mosh-server, mosh-server will then place it into the pseudo-terminal to be read by the running shell. At some point later it will be echoed back to the terminal, and will appear to mosh-server. In comparison, were I to type it into mosh-client to be sent to a local copy of mosh-server, and it to then be placed into the pseudo-terminal and read by ssh, sent over the network to another machine, copied into that pseudo-terminal and then later echoed back, read to copy over ssh, and then finally displayed through to mosh-server, is not not accurate to claim that all of those later steps are simply encapsulated in the running shell? Put differently, how is being connected to a running copy of ssh different than being connected to a running copy of screen or even vim? Does mosh rely on some kind of expected delay-to-echo? Would I run into the same problem with a very slow process, or mosh to a remote server followed by ssh to yet a different server?

[saurik]

@andreasvc I explained in my earlier comment to you that "the problem with using the SSH connection is that once it dies, in order to reestablish it, you will need to re-authenticate with the server using the SSH authentication mechanisms". To make this more clear: if you use the autossh mechanism I described (which is clunky as hell), you will notice that every time it reconnects you need to re-authenticate; this may be reasonable using a private key, but only if that private key does not have a passphrase on it; if you have a passphrase on your key (a good idea, although on OS X the keychain mechanism for handling that is pretty cool) or use password authentication the experience becomes really irritating as you reconnect (especially so as you may be typing along and suddenly be given a password prompt), and with challenge-response authentication it becomes utter torture as you reach for your hardware token.

[saurik]

@keithw I was actually discussing the "tunnel UDP over SSH" option with someone last night, but given that you will still need to have a mechanism to maintain the active SSH channel (such as autossh) it almost makes sense to relegate that to its own general-purpose project that manages arbitrary UDP channels over arbitrary TCP channels: it could then be layered over an underlying SSH port forwarding, but would be free to be used in other contexts as well.

[andreasvc]

Reconnecting SSH is painless if you use a private key with an ssh-agent (contrary to what you say, it does not have to be passwordless); I assume the OS X keychain is similar to what Gnome/KDE provides: one password prompt which is remembered as long as the ssh-agent runs. Even if you don't use private keys, the mosh client could remember the password so you don't have to retype it. So I don't see SSH re-authentication as a problem. On the contrary, running everything over SSH will appease the skeptics who say that mosh does not have a proven security record.

I think sending the UDP packets over SSH directly, as Keith seems to suggest, is faster than using a TCP connection within SSH. When I ssh to another host within an ssh session, it is reasonably fast; however, if I add a port forward to that same ssh server and connect to that, I get noticeable latency.

[saurik]

@andreasvc Fair enough: I thought that ssh-agent only was related to forwarding passkeys to remote hosts. I will learn more about ssh-agent so I do not make similar mistakes in the future (and it may also help some of my use cases). Sorry, and thanks!

[keithw]

@saurik: Yes, we need some timeout on echoes or else we can never detect a false-positive prediction. (E.g., the user hit "x" when the cursor was at (5,15); we predicted an "x" would appear but it has been 45 seconds and still no "x". Does that mean the application is in a no-echo state and our prediction was wrong? Or should we keep waiting and keep the prediction on the screen?)

Early versions of mosh tried to do all the timeouts on the client side, but that didn't turn out to work very well because we would sometimes time out when really it was just network jitter causing the echo to be delayed. So now the only timeout is on the server side, and the client requires an explicit "echo ack" before it will time out a prediction.

The server will send an "echo ack" for each keystroke that has been presented to the application for at least 50 ms (ECHO_TIMEOUT in src/statesync/completeterminal.h). So, yes, in effect we are saying that for something to count as a correct prediction of an echo, it has to be echoed by the application within 50 ms of when the app gets the keystroke. Otherwise we score it as an incorrect prediction.

[zedinosaur]

Just to mention it but there's TCP Minion which allows datagrams over TCP with unordered delivery (but still connection oriented and guaranteed delivery). Might be interesting to support that for environments where only TCP is available.

[pickfire]

+1, so that people could use mosh with tor.

[ghost]

+1

[rzr]

@zedinosaur said :

there's TCP Minion which allows datagrams over TCP

Have you been using it with mosh ?

For the curious ones :
http://dedis.cs.yale.edu/2009/tng/papers/pfldnet10-slides.pdf

[beweisj]

+1 This is the only reason I never adopted mosh. (see https://serverfault.com/q/390916 for what lengths users go to)

[mkorkalo]

+1 to mosh over SSH automatic fallback, it makes much sense to me. I also don't have a problem with ssh reconnection, it's still way better than direct SSH for the high latency use cases.

[UralZima]

+1. Very need this

[tobimensch]

I'm currently stuck with throttled 3G, and even though UDP works in principle, it looks like my provider is crippling it in every possible way. When many packages are sent in a short timeframe the data loss can be anywhere from 45% to 100%. Even though 64kbits ought to be plenty fast for the application I want to use, this leads to mosh losing contact for minutes at a time.

I'd like to create a bounty for whoever can implement a mosh over tcp mode, that is also as efficient as possible. (Doesn't really make sense when it ends up not being better/faster than regular ssh)

Is anyone interested in implementing it? What amount of money would you need to get motivated enough to actually complete this feature?

I can only chip in a little bit. Who else would donate for this?

This feature request has been open for years. It's obviously possible to solve this theoretically, which means no developer was motivated enough to do it so far. That's why I think a bounty could help to fix this.

[domo141]

@tobimensch: I've just done mosh support to my ioiomxtx tunneling and multiplexing software at

https://github.com/domo141/ioiomxtx

basically mosh datagrams multiplexed to TCP tunnel, with help of ldpreload library for port matching.

Works pretty well, and could be ported to work over other software than mxtx (e.g. ssh).

[tobimensch]

@domo141
Sounds great. Can you write a short tutorial on how to use this with mosh?

[domo141]

@tobimensch
I added tl;dr; section to ioiomxtx README.md which shows minimal steps to try mosh.

I also added mxtx-mosh.md which describes how mosh(1) and mxtx-mosh(1) sets up mosh connections.

Now that I've thought it at bit it would not be too hard to modify mxtx-dgramtunneld to connect via
ssh instead of mxtx for tunneling mosh traffic is one so desired...

[jdstroy]

In case anyone else stumbles across this problem....

This feature will be necessary for environments that only have Internet access through a proxy server (such as a corporate SOCKS proxy, and not just going through Tor).

[fancsali]

Yes, that would be the dog's. I reckon, going through a Proxy (SOCKS or HTTP CONNECT) would be an instant hit: that would be also flexible:

• You really want Tor – check;
• Corporate proxy – check;
• OpenSSH has a built in SOCKS feature, so the tunnel-over-ssh – check!

[fancsali]

Just to add to my earlier comment: I've been thinking about this, and I reckon, we're actually looking at two separate – but connected – issues here:

Problem 1: What do do, if we have no UDP connectivity? (But direct TCP is fine)

I think, this is the easier one. I've done a quick and dirty UDP-TCP forwarder in Python, that prefixes the datagram from a UDP socket with a two-byte lenght field, and pushes that through a TCP one. If the TCP side disconnects, just tries to reconnect. MOSH is more than happy to talk to the server through that. I took the liberty to peek into the code for mosh, and I reckon, something along those lines shouldn't be too hard to add.

Problem 2: What to do if even TCP is only available through some proxy?

This is a bigger problem, IMHO, as the nuances of the different proxies vary. Also, I can see a rather large problem around the problems around #285. If we cannot even do an ssh, then the whole setup becomes inconvenient. Unfortunately using LD_PRELOAD is not always feasible either (e.g. MacOSX will not let you much around with that and run SSH)

However, this second issue is easier to solve with out-of-the-box tools, I believe. Worst case I set up a tcp tunnel with socat, and make ssh connect locally:

$ socat tcp4-listen:60000 socks4a:<proxy-ip>:<hostname>:<port>,socksport=1080

# ~/.ssh/config
Host myserver
    HostName localhost
    Port 8022

... or something along those lines.

I reckon, the above broad scope is perhaps the reason, there have been no volunteers for this; even with the possibility of a bounty, as proposed by @tobimensch.

So, I'd like to propose let's limit the scope to the bare minimum, perhaps, and try to find some help like that.

[xanoni]

Could someone please update interested users (e.g., me :) on if it's now possible to use mosh with sshd behind a tor hidden service, and how? I'm OK with a hack, if it performs well. Thank you.

[andersk]

@xanoni That’s not possible: Mosh uses UDP connections to IP addresses, and Tor hidden services have neither UDP connections nor IP addresses. You could try routing a VPN over Tor and running Mosh over the VPN, but it’s probably going to perform worse than SSH.

[xanoni]

Oh okay. I skipped over the thread and it seemed people had found solutions (e.g., this here #13 (comment)), but I didn't check the details. Sad.

EDIT: Seems it might be a solution, but there hasn't been a lot of activity in the repository ... see domo141/ioiomxtx#1

@domo141 can ioiomxtx tunnel mosh over Tor?

[xanoni]

For people in this thread who were trying to use Mosh with Tor but haven't found a solution, yet ... you might want to look at the I2P protocol instead, the C++ implementation (i2pd) natively supports UDP.

See here: https://i2pd.readthedocs.io/en/latest/user-guide/tunnels/#udp-tunnels

[ghost]

how can we use mosh-tcp now?

[OliverOffing]

For people in this thread who were trying to use Mosh with Tor but haven't found a solution, yet ... you might want to look at the I2P protocol instead, the C++ implementation (i2pd) natively supports UDP.

There are instructions here on how to set up Mosh over I2P, although the initial connection was inconsistent.

[fancsali]

Great article. This solves the use-case of accessing your hidden machine. (Though, as I2P seems to be much less popular, I reckon there are much fewer nodes than Tor, hence the stability/performance might be a hit and miss, as alluded to in the article too.) The other use-case when I want to access (my) ordinary server(s) on public IP from any dodgy/restricted network when out and about is still an issue. I2P to clearnet relaying is not really a thing.

…

On Thu, 26 May 2022 at 20:59, Oliver Offing ***@***.***> wrote: For people in this thread who were trying to use Mosh with Tor but haven't found a solution, yet ... you might want to look at the I2P protocol instead, the C++ implementation (i2pd) natively supports UDP. There are instructions here on how to set up Mosh over I2P <https://stacker.news/items/31972>, although the initial connection was inconsistent. — Reply to this email directly, view it on GitHub <#13 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMXX6FT3CZKFLWK42B4SZ3VL7Q2PANCNFSM4AK3JJZQ> . You are receiving this because you commented.Message ID: ***@***.***>

[sudo-gera]

Secure Socket Funneling is useful here.

How to use:

Prepare 3 terminal tabs on client.

Tab 1:

ssh -p REM_PORT REM_HOST -L 8011:127.0.0.1:8011 -L 2222:127.0.0.1:22 # change last 22 to actual port of remote ssh server

(now you're logged in on remote server)

cd ssf-linux-x86_64-3.0.0/ # or where is it installed
./ssfd

Tab 2:

cd ssf-linux-x86_64-3.0.0/ # or where is it installed
./ssf 127.0.0.1 -U 3333:127.0.0.1:3333

Tab 3:

mosh --ssh='ssh -p 2222' -p 3333 127.0.0.1