Manage listening sockets for upstream userspace PM

[mjmartineau]

The MPTCP upstream Linux community has selected a userspace PM architecture that has the userspace daemon managing the MPTCP listening sockets rather than having the kernel create additional in-kernel listeners based on the userspace PM netlink commands:

https://lore.kernel.org/mptcp/48686ee-4d79-c9fd-35d5-593b9ec9742b@linux.intel.com/

Another option was to try to have the kernel handle token lookup for incoming MP_JOINs on any address and port, but that option still had some tricky corner cases and involved an increasing number of TCP code modifications.

---

For this listening socket feature, mptcpd needs to optionally create MPTCP listening sockets for relevant addresses and ports before sending MPTCP_PM_CMD_ANNOUNCE for that endpoint on any MPTCP connection and after sending MPTCP_PM_CMD_REMOVE for all MPTCP connections. This is to guarantee the advertised addr/port is available to handle incoming joins.

Timing of this socket creation should be flexible. The listener may be created well in advance, for example when mptcpd initializes or configuration is modified and system-level policy creates an endpoint to share across all MPTCP connections in the entire namespace. Or, a listener may be created dynamically if the listener is only relevant for a specific MPTCP server or client application. Plugins must be able to manage this behavior. In this dynamic case, the listener may be a shared resource between multiple tracked MPTCP connections - I'm not sure if mptcpd should handle that sharing at a lower level, or leave it up to each plugin to manage.

Listening socket creation needs to be optional, since the advertisement could be for a non-local address - for example, a load balancer.

The MPTCP listeners created by mptcpd should not accept() any incoming new TCP or MPTCP connections on these listening sockets, unless it does so in order to immediately close() the attempted connection. mptcpd should listen(sock, 0) (backlog == 0) in order to minimize possible kernel resource use by the backlogged connection queue. Ideally the kernel would not even send SYN-ACK packets for attempted connections on these listening sockets, and would only proceed with handshakes for MP_JOIN SYN packets.

[ossama-othman]

PR #199 now has an early implementation of a MPTCP listener manager. It is not yet configurable through the mptcpd configuration file or command line. Plugins also cannot yet specify whether or not to listen. We'll probably need an additional boolean flag parameter for mptcpd_pm_add_addr() to allow plugins to exert such control.

[matttbe]

Hi Ossama,

PR #199 now has an early implementation of a MPTCP listener manager. It is not yet configurable through the mptcpd configuration file or command line. Plugins also cannot yet specify whether or not to listen. We'll probably need an additional boolean flag parameter for mptcpd_pm_add_addr() to allow plugins to exert such control.

Now that this PR has been merged, does it mean that the MPTCP listener manager is available somehow? Or do you mean: the code is there but it is not possible to use it for the moment? Or enabled by default?

[ossama-othman]

Hi Matthieu,

Now that this PR has been merged, does it mean that the MPTCP listener manager is available somehow? Or do you mean: the code is there but it is not possible to use it for the moment? Or enabled by default?

Listening socket "policy" is still not configurable through the mptcpd configuration file or the command line, but the listener manager code is there and enabled.

Mptcpd will implicitly create a listening socket for addresses passed to mptcpd_pm_add_addr(), or increment the reference count on the "listener" if a listening socket already exists and is tracked by mptcpd. Calling mptcpd_pm_remove_addr() will cause the listener reference count to be decremented, and closed if the reference count drops to zero.

Mptcpd plugins can also exert some control over listening sockets by getting access to the global mptcpd listener manager object, and explicitly listen on a given address or close the listener as needed, e.g.:

// mptcpd_lm API in  <mptcpd/listener_manager.h>

struct mptcpd_lm *lm  = mptcpd_pm_get_lm(pm);  // "pm" passed to plugin ops

struct sockaddr *sa = ...;   // struct sockaddr_in or sockaddr_in6

int result = mptcpd_lm_listen(lm, sa);
...
int result = mptcpd_lm_close(lm, sa);

This allows plugins to create listener pools, for example, before advertising addresses.

[ossama-othman]

Should we close this issue, and open another one specifically for mptcpd configuration file and command based listener policy configuration?

[matttbe]

@ossama-othman thank you for your reply. I just shared the info on multipath-tcp/mptcp_net-next#293

Should we close this issue, and open another one specifically for mptcpd configuration file and command based listener policy configuration?

Probably better yes but this is up to you :-)