Improve the logic for XFF and internal/external client detection

[brian-pane]

Description:
Per the discussion in PR 2559, Envoy's logic for determining whether any given request is internal or external, and what the trusted address of the origin client is, has become very complicated. And even with this complexity, the mechanism isn't sufficiently general to cover the needs of all users.

One potential improvement would be to create a plug-in interface so Envoy users can plug in their own logic. Another would be to simplify the configuration semantics for the current functionality.

[slovdahl]

I just stumbled upon this problem while trying to figure out why my local Envoy in a Docker container allowed my curl requests to spoof the x-envoy-external-address header when having Envoy configured to act as an edge proxy. It took me a while to figure out that the problem was that the source IP of my requests 🙂

For reference, the configuration I use (inspired by https://www.envoyproxy.io/docs/envoy/v1.12.0/configuration/http/http_conn_man/headers#config-http-conn-man-headers-x-forwarded-for):

static_resources:
  listeners:
    - address:
        socket_address:
          address: 0.0.0.0
          port_value: 80

      filter_chains:
        - filters:
            - name: envoy.http_connection_manager
              config:
                stat_prefix: ingress_http
                codec_type: AUTO

                use_remote_address: true
                xff_num_trusted_hops: 0

                route_config:
                  name: local_route
                  virtual_hosts:
                    {
                      name: local_service,
                      domains: ["*"],

                      routes: [
                        {
                          match: { prefix: "/" },
                          route: { cluster: backend },
                        }
                      ]
                    }

                http_filters:
                  - name: envoy.router

Doing a curl request like this:

$ curl -sv -H "x-envoy-external-address: 1.2.3.4" http://localhost/foo

..and the traffic dump shows that the backend request actually got the spoofed x-envoy-external-address:

We would like to put Envoy in front of all incoming HTTP traffic to our backend services. The problem is that our edge might see untrusted traffic from both internal and external IP addresses. Having read through some discussions in e.g. #2559 and #4383, it sounds like just another configuration option is not the way forward here? Not asking for a solution, just want to start a discussion about the right way to solve this problem.

[slovdahl]

I just realized that another way of solving our use-case would be if Envoy supported removing any x-forwarded-for header on an incoming request, before Envoy sets it itself. Or is this somehow possible already? I have tried using request_headers_to_remove, but that just causes the x-forwarded-for header to be completely removed from the request that the backend sees as well.

[robinp-tw]

@slovdahl having stirred some waters around this, found that in the connection manager ConnectionManagerUtility::mutateRequestHeaders (which determines internal-ness of request as well) is called before the route is actually selected (and its modifiers like request_headers_to_remove could take effect). It kind of makes sense, you don't want to footgun yourself with removing XFF prematurely, and making Envoy believe the request is internal when it is in fact not.

our edge might see untrusted traffic from both internal and external IP addresses

Interesting. Sounds like you desire that internal traffic to be treated external, But with the default use_remote_address=true && xff_num_trusted_hops=0 setup used for edge router, the internal remote address will be trusted & treated as internal. If you control those internal sources, you could arrange to set XFF header on their requests, which will make the edge proxy treat them as external with those settings (Note that the XFF header sent upstream would then by default contain the original XFF that was sent + the IP of the internal source, which might have further consequences down the chain... but you can control append behavior, or use those route filters to remove).

[slovdahl]

Thanks for the insights, @robinp-tw 👍

If you control those internal sources

Unfortunately not. In this case, all incoming requests are actual end-user browsers, some behind NAT and a public IP, and some on IP addresses considered internal by Envoy.

One thing that just occurred to me (I haven't checked yet), but could a Lua script help here in some way? 🤔