Harden Envoy against untrusted upstream servers

[yanavlasov]

This is top level issue for tracking tasks and progress of hardening Envoy against untrusted upstream servers. More specific sub-tasks are split into its own issues and linked to this issue.

Objective

1. Fix existing known high priority issues with untrusted upstream servers, sufficient to allow Envoy core components to be hardened against untrusted upstream servers.
2. Modify the Envoy OSS threat model to include untrusted upstream servers.

Background

Presently Envoy’s threat model considers all upstream data and control plane infrastructure to be trusted. This includes upstream servers, auxiliary servers invoked by filters, such as remote authz server, control plane endpoints or user controlled portions of configuration. This approach meets requirements of all existing Envoy’s production deployments which are mostly reverse proxy to trusted production services or service mesh. As a result most security related fixes and improvements have been applied to client (downstream) requests only. For example mitigations to HTTP/2 flood attacks have been developed for downstream requests only and an upstream server may cause Envoy to consume excessive resources by flooding it with control frames.

However, a number of upcoming Envoy based products are required to be hardened against untrusted components of upstream infrastructure. For example L7ILB proxies requests to customer controlled upstream servers, and a buggy or malicious upstream server in multitenant mode may cause outage or intercept data of other customers.

This design document focuses on hardening against untrusted upstream servers. Hardening against other aspects of upstream infrastructure is covered elsewhere.

Overview

The entire scope can be split into several areas. This is preliminary list and it may change as we make progress:

• Implement error handling in codecs for cases outside of the processing of incoming client data.
• Achieve parity of security and other bug fixes between downstream and upstream use cases.
• Audit implementation of HCM and Router filters for potential problems with untrusted upstream servers.
• Audit error and invalid response handling in health checking.
• Audit cluster manager.
• Extend fuzzing to handling of upstream responses.

[antoniovicente]

Could we update the Envoy threat model to mention our goal to defend against untrusted upstreams? Bonus points for specifying a target completion date for this effort.

The reason for the update in threat model is to increase awareness about this effort. Going forward we should try to harden downstream and upstream at the same time. In cases where doing the upstream improvements at the same time is impractical, we should file issues to track the upstream hardening deficiency and associate it with this issue.

[htuch]

@yanavlasov where do you think hardening core filters that process upstream inputs fits here? I think some filters, e.g. router, process upstream headers and we need to have confidence that they are robust to problematic inputs.