Multihome option selecting egress interface instead of leaving decision to OS routing table

[taktv6]

Describe the bug
When using --multihome option according to the man page the expected behavior is that packets sent from server to client will be send with the source IP being set to what the client used as the destination IP to reach the server.
This in general works, but has an unexpected side effect: The return packets are sent out the same interface where the clients packets were received on. Effectively ignoring the servers routing table.

We're running a load balanced setup with an IPVS based L4 load balancer in tunnel mode with direct server retrun (DSR) setup in front of our OpenVPN servers.
The load balancers are encapsulating traffic into an IPIP tunnel to steer the packets to a specific selected backend. On the openvpn server node we have an IPIP tunnel interface like this:

lb-vpn: ip/ip remote any local 192.0.2.0 ttl inherit

The tunnel is unidirectional (as this is a DSR setup) and we expected the egress packets to just leave the machine on an interface that is the result of a regular routing table lookup. instead we see the packets being sent into the IPIP tunnel interface.

To Reproduce
Setup openvpn server with --multihome enabled, where ingress traffic is received on one interface and default route points out another. You'll see the egress packets leave on the wrong interface.

Expected behavior
Egress interface/next-hop selection should always be left to the operating systems routing table.

Version information (please complete the following information):

• OS: Ubuntu 22.04
• OpenVPN version: 2.6.14-0ubuntu0.24.04.1

Additional context
Add any other context about the problem here.

[cron2]

Your description makes sense and #856 will nicely fix it for your use case. Alas, the code is there for a reason - in other setups the egress interface needs to be symmetric, as next-hop routers/ISPs might do BCP38 source validation.

Non-trivial to fix, except if we add yet another option.

[taktv6]

Well, it may break current behavior (which can be seen as valid or broken, depending on use case).
However, with my PR, behavior would match what is documented.
What shall we fix? Behavior or documentation?

From the man page:

              Configure a multi-homed UDP server. This option needs to be used
              when a server has more than one IP address (e.g. multiple inter‐
              faces, or secondary IP addresses), and is not using  --local  to
              force binding to one specific address only. This option will add
              some extra lookups to the packet path to ensure that the UDP re‐
              ply  packets are always sent from the address that the client is
              talking to. This is not supported on all platforms, and it  adds
              more processing, so it's not enabled by default.

              Notes:

                     • This option is only relevant for UDP servers.

                     • If  you  do an IPv6+IPv4 dual-stack bind on a Linux ma‐
                       chine with multiple IPv4 address, connections  to  IPv4
                       addresses  will  not work right on kernels before 3.15,
                       due to missing kernel support for the IPv4-mapped  case
                       (some  distributions have ported this to earlier kernel
                       versions, though).

The documentation does not mention the packet being sent out a specific interface.

IMHO:
If the return path matters, this should be solved with policy routing. Not inside OpenVPN.
The --multihome option should not exist as a packet sent to a client should always be sourced with the appropriate source address that was used by the client to reach the server. Always. Otherwise it just will not work.

[cron2]

The world is slightly bigger than Linux, and not all other OSes have a really good policy routing framework.

WRT documentation, the documentation does not state anything about egress interfaces, so it can not really be called a documentation bug. Maybe an omission. But yeah, whatever we do in the end needs to come with a documentation update.

[ordex]

@taktv6 have you checked this behaviour when using DCO as well?

[taktv6]

@ordex we have observed this behavior with DCO enabled.

@cron2: The world is for sure bigger than just Linux. But is it OpenVPN's job to provide workarounds around OS limits? Linux, FreeBSD, OpenBSD and Solaris/Illumos all have support for policy routing. The ones that don't are Windows, OS X and NetBSD.

If the specified egress interface is an Ethernet interface, this "feature" doesn't work as it won't be possible to craft the necessary Ethernet header (due to lack of destination MAC address). It would only work with point-to-point style interfaces (like ppp, ipip, gre, etc.). I'm not sure which OS would just ignore the ifindex and perform a route lookup (and thus just ignore the ifindex) and which would just drop it.

I think the ideal way forward is:

1. Make sure that response packets are always coming from the source IP that the client was talking to. Everything else just doesn't work. This should be default behavior under all conditions (including when no local statement exists) and is in fact what one can regularly expect from other UDP servers out there as well.
2. If the answer to the question Is it OpenVPN's job to provide workarounds around OS limits? is Yes: Leave the multihome feature untouched as is and adjust the documentation to be explicit about it. If the answer is No (IMHO it should be): Just drop the multihome feature altogether.

[schwabe]

@taktv6 workaround that fix problems for one person often break it for others.

And it is not just "leave mtulihome" untouched. The DCO implementation is a complete new implementation. So it is not leaving multihome untouched. It is rather that we have to figure out how to emulate something similar like multihome with the in kernel networking.

@taktv6 and this is an open-source project, feel free to implement it yourself and contribute it.

[cron2]

@schwabe well, that is what @taktv6 actually did, in #856. We're just a bit in disagreement on what OpenVPN's role should be here, and what the defaults should be (and if the defaults should change, risking breakage for users that expect the existing behaviour).

For a new project with no existing setups, it's somewhat easier to be "philosophically correct" on how a software should behave.

OTOH, we need to double check what DCO is doing in this situation (... on FreeBSD, Linux and Windows), and definitely update documentation.

The discussion on whether there should be a --multihome switch at all (or whether it should just be default) is also quite old. There is quite a bit of platform specific code involved, and at some point in time, things just broke with --multihome (e.g. older Linux versions, using v4-mapped-v6). Now that's all old news, so this is a good time to revisit what we should do today.

[taktv6]

I'm happy to fix anything once there is an agreement what is wanted.
That's what this discussion is there for.