Support for proxy authentication

[laurensb]

Description:

I would like to be able to specify a username and password for authenticating with a proxy server. Ideally the mechanism introduced in #4198 is extended to allow setting the following environment:

HTTP_PROXY=https://user:password@hostname:port
HTTPS_PROXY=http://user:password@hostname:port

So note the addition of both a protocol (http or https) and a username and password. So HTTP connection can be proxied through a proxy server requiring a secure HTTP connection and likewise HTTPS connections can be proxied using a UNsecure connection to the proxy server. This format is the common format used by many applications e.g. curl.

Ideally this same configuration would apply to all outgoing HTTP(S) requests including federation (see #8859 and #8660).

Rationale:

Most corporate environment require all outgoing connections to go through an authenticated proxy. Given the potential sensitivity of data contained within Synapse excluding it from this policy is usually not an option.

Implementation:

#6239 introduces basic proxy support by manually implementing this functionality, presumably because the Twisted HTTP client does not (completely) support proxies. Rather than implement proxy authentication manually may I be so bold as to suggest replacing the Twisted client library with another library, e.g. 'requests', assuming is it compatible with the Twisted server framework. Authentication in general is tricky to implement securely, using a well tested, widely used library, prevents accidently introducing vulnerabilities.

[clokep]

@anoadragon453 Was this fixed by #9657?

[anoadragon453]

@clokep Somewhat. Current we only support proxy authentication with the HTTPS_PROXY environment variable, and specifying a protocol is not supported.

[clokep]

Specifying the protocol seems to be #9090 though.

[anoadragon453]

That's true... though we're still missing the ability to authenticate to an HTTP proxy (though I debate the usefulness of authenticating over a plaintext connection...).

[laurensb]

I agree authentication over plain HTTP seems like a questionable idea. However in certain environment the internal network is considered secure even without encryption e.g. inside a Kubernetes cluster.

It would be nice if synapse could also be deployed in such environments.