TCP performance better than MPTCP

[Monika-Research]

Hi Matt & others,
I have been working with MPTCP since 2017. So I was only using out of tree implementation so far. Now that upstream implementation is supporting most of the basic requirements of MPTCP, I tried using it.

I have 2 ubuntu 22.04 systems (Kernel - 5.15) with a fast Ethernet direct connection and a busy wlan connection between them. The Ethernet connection is kept as the primary flow and wlan as subflow (usedip mptcp signal/subflow configurations on the server & client)

Following are the results obtained using mptcpize iperf3
MPTCP throughput- eth: 60-70 Mbps and wlan0: 1.7-1.8 Kbps (this was expected)

However only TCP communication (single path over eth) gave better results
TCP throughput - 160-170Mbps

One of the main design goal of MPTCP is to ensure that it achieves atleast same throughput as that of TCP (over its best path). Is this getting violated?

I never experienced with out-of-tree implementation.

Kindly request your comments on the same, Matt.

Cheers,
Monika

[matttbe]

Hi Monika,

I have been working with MPTCP since 2017

Nice, 5 years!

I have 2 ubuntu 22.04 systems (Kernel - 5.15)

Do you mind re-doing the tests with a more recent kernel installed on both the client and the server side? (e.g. 6.0 or ideally by compiling our export branch). A lot of modifications have been done and there are some optimisations on the scheduler side: https://github.com/multipath-tcp/mptcp_net-next/wiki#changelog

Note that on Ubuntu, packages are available there: https://kernel.ubuntu.com/~kernel-ppa/mainline/ (e.g. take the v6.0 version)

MPTCP throughput- eth: 60-70 Mbps and wlan0: 1.7-1.8 Kbps (this was expected)

Why is it expected to have 1.7-1.8 Kbps on the WiFi side? That looks like a bad situation for MPTCP: it should handle that but trying to use it will most probably be a cost in term of buffer and CPU utilisation and maybe the upstream kernel doesn't handle this "extreme" case well yet.

Could you also redo some tests with MPTCP but only one path?
And with multiple paths but multiple connections in parallel (e.g. iperf3 -Z -P 10)?
Depending on the network conditions, you could be limited by the different buffers (net.ipv4.tcp_{r,w,}mem) but also by the IO, networking queues, CPU, etc. The best would be to take traces (with a recent kernel) and analyse them.

One of the main design goal of MPTCP is to ensure that it achieves atleast same throughput as that of TCP (over its best path). Is this getting violated?

Of course not. The goal is to get the best for all the different paths but this is a quite complex job to do. If you still have a big difference with a recent kernel when switching from 1 to 2 paths with this very bad second path and if you are not limited by buffers/CPU, then there is something to investigate.

Cheers,
Matt

[pabeni]

MPTCP throughput- eth: 60-70 Mbps and wlan0: 1.7-1.8 Kbps (this was expected)

Why is it expected to have 1.7-1.8 Kbps on the WiFi side? That looks like a bad situation for MPTCP: it should handle that but trying to use it will most probably be a cost in term of buffer and CPU utilisation and maybe the upstream kernel doesn't handle this "extreme" case well yet.

Addendum: could you also please specify the link speed ? You mentioned fast ethernet but the measured plain TCP B/W you reported is well above the fast ethernet limit (100Mbps). Are there any other flows running on the same link?

[Monika-Research]

Firstly thank you @matttbe & @pabeni for an immediate response.

I am yet to try the new kernel version however tried tuning the TCP buffer parameters and usage of multiple parallel connections.

This is exactly an extreme scenario as rightly said by Matt. The WiFi connection is used by close to say 50 ppl in office. So its kind of a poor link. On the other hand, the ethernet connection is Cat6A 500 MHz which is capable of supporting upto 10 Gbps. I was just curious how MPTCP would behave in case of a good and bad subflow. It would be nice if it can choose good over bad subflow instead trying to use both which would degrade the performance.

Please ignore the numbers I shared previously. Those were from wireshark (conversations/TCP/A->B(bps)).
For some reason they are not giving the same results as iperf. So please see ignore them and see the results shown by iperf3 below:

1. Single path TCP
   Throughput: 925 Mbps

2. MPTCP Single path (eth primary flow)
   Throughput: 925 Mbps

3. MPTCP 2 subflows (eth and WiFi)
   Throughput: Average over 10 samples - 316 Mbps

I changed the TCP buffer parameters to the following:
net.core.wmem_max=12582912
net.core.rmem_max=12582912
net.ipv4.tcp_rmem=10240 87380 12582912
net.ipv4.tcp_wmem= 10240 87380 12582912

4. MPTCP 2 subflows - Receiver buffer parameters tuned
   Avg. Throughput - 452 Mbps
   In this case, I observed that some samples were able to reach upto 650 Mbps. Tuning buffer parameters did help.

5. MPTCP 2 subflows - recv buff + parallel connections 10
   Nothing changed. We got the same performance as single connection.

Next when I test with newer kernel version, I will also log throughput per interface. I generally use nicstat. Do you have any recommendation? Also, I would like to double confirm if the scheduler used is BLEST?

Note: My intention of these tests are to find out how MPTCP reacts to different scenarios so that I can draw some usecases.

[matttbe]

Thank you for these first tests.

see the results shown by iperf3

Which parameters do you use with IPerf3? Is it an upload/download? Regarding the throughput, do you look at the view from the receiver side?

In this case, I observed that some samples were able to reach upto 650 Mbps. Tuning buffer parameters did help.

I don't know what's the latency on the different networks you use but you might need to go higher than 12MB for the buffers.

We got the same performance as single connection.

Mmh, strange. Then going higher than 12MB should not change anything (and I guess you are not limited by a very small tcp_mem I suppose.

Next when I test with newer kernel version

Yes, that would be good not to investigate on an old kernel.

I will also log throughput per interface. I generally use nicstat.

It depends what you want to see but ifstat is maybe what you would like to have: ifstat -i eth0,wlan0 -btT for example

Also, I would like to double confirm if the scheduler used is BLEST?

There is only one packet scheduler in this upstream kernel so far.

[matttbe]

Hi @Monika-Research

By chance, do you have any more input to share?

[Monika-Research]

Hi @matttbe,

Firstly, sorry about the delayed response.

I didn't proceed with this any further for two reasons

1. Currently, I have few dependencies on the kernel version and can't move up to 6.0
2. The main idea was to set up MPTCP over multiple mesh interfaces. So, I started working towards the key goal. Btw, I could get this to working fine. However, I had to disable mptcpd at some point as it was quite confusing with endpoint rules and I updated the endpoint configurations manually.

Now, I have another issue with backup flow case. I will discuss it on a new thread.

Thanks for your time and support!

[matttbe]

Hi @Monika-Research

I think we can close this ticket: it should be fixed by tickets #332 (and #345).

Any objections?

[Monika-Research]

Of course sure, thank you!