Proxy setup for agent

[mostlyjason]

Let's document how users can set up and use proxies with Elastic Agent.

Related:

• [Fleet] Add support for proxy in Elastic Agent beats#25319
• Introduce httpcommon package in libbeat (add support for Proxy) beats#25219 (comment)
• https://github.com/elastic/ingest-dev/issues/1051

[urso]

@mostlyjason I added this issue to the task list in the meta-issue.

We have many issues about proxy support, so it can be quite confusing to figure out how it is supposed to work and which settings we have available. elastic/beats#25021 contains the most details how proxy support works. Issue https://github.com/elastic/ingest-dev/issues/1051 contains a diagram with communication channels and a list of potential places proxies could be.

Anyways, let me give you a short summary how proxy support works.
By default, if nothing is configured in Agent, the HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables are used to configure a HTTP 1 proxy endpoint (HTTP CONNECT).

From Go docs about environment variables support:

ProxyFromEnvironment returns the URL of the proxy to use for a given request, as indicated by the environment variables HTTP_PROXY, HTTPS_PROXY and NO_PROXY (or the lowercase versions thereof). HTTPS_PROXY takes precedence over HTTP_PROXY for https requests.
The environment values may be either a complete URL or a "host[:port]", in which case the "http" scheme is assumed. An error is returned if the value is a different form.
A nil URL and nil error are returned if no proxy is defined in the environment, or a proxy should not be used for the given request, as defined by NO_PROXY.

In addition to support for environment variables, users can configure proxy support via config files. We have proxy support in a many places/namespaces throughout Agent/Beats. All accept the same set of settings. elastic/beats#25219 Did introduce missing settings to a few places, guaranteeing consistency between components (check the PR description for more details). Proxy settings:

• proxy_disable: If set, no proxy will be used. Environment variables will be ignored.
• proxy_url: If set, the configured URL is used as proxy for all connection attemps by the component.
• proxy_headers: Additional headers to be send to the HTTP proxy during HTTP CONNECT.

Fleet enrolmment

Proxy types

Proxies can come in different flavors and support different protocols. Proxy support for HTTP Connect (HTTP 1 only) and Socks5 is supported.

HTTP 1 connections should still be able to upgrade to HTTP 2 after a successful HTTP CONNECT (that's some detail we might not want to document though).

From Go docs:

The proxy type is determined by the URL scheme. "http", "https", and "socks5" are supported. If the scheme is empty, "http" is assumed.

Proxy authentication

Some environments might require users to authenticate themselves with the proxy. We do not have explicit settings for authentication. The proxy_url accepts optional username and password like http://<username>:<password>@<proxy host>/. If keys/tokens are given to users, the proxy_headers setting should be used. We do not support more sophisticated authentication schemes for the proxy connection.

Agent enrollment into Fleet:

When enrolling an Agent into Fleet, we might not have a configuration file yet. In addition to the environment variables, the enrollment command accepts the proxy settings via CLI (PR elastic/beats#26514). Namely --proxy-url <url>, --proxy-disabled, --proxy-header <header name>=<value>. The --proxy-header CLI flag can be used multiple times to add additional headers.

During enrollment internal configuration files might be written. The proxy settings will be copies over to those configuration files.

Notes:

• Beats features not exposed or used via Agents did also gain proxy support (See: Introduce httpcommon package in libbeat (add support for Proxy) beats#25219 PR description). For example did we add proxy support for Kibana as well: Introduce httpcommon package in libbeat (add support for Proxy) beats#25219
• The Heartbeat HTTP monitor will ignore the environment variables, to not confuse system proxy setup with the system to be monitored. One can still configure proxy_url and proxy_headers in heartbeat if that functionality is required.
• Proxy support only has been unified for everything HTTP in Agent/Beats. Features (Inputs) using other network protocols have not been updated.

[mostlyjason]

Awesome thanks @urso!

@dedemorton is this enough info for the docs team to convert it into a public doc? I'll reassign this back to you but let me know if you need more information.

[dedemorton]

Yes, it's enough for us to get started, assuming the related PRs and issues describe the final product behavior. Can you ask someone on the dev team to confirm that the info in those PRs/issues is correct?

[mostlyjason]

@dedemorton i dont see anything incorrect there but let me know if anything looks inconsistent

[urso]

The details in the PRs and given here should be complete

[mostlyjason]

@andresrc you mentioned this issue needs scoping in an email. Do you see anything that you feel should be left out or is missing? I think users need to understand how to set the setting or environment variables, the values they can take, and caveats. You mentioned some issues discovered during testing?

[andresrc]

@mostlyjason what I meant is that in the diagram linked in https://github.com/elastic/ingest-dev/issues/1051 there are multiple connections that might be impacted by the proxy settings, and in many scenarios we might find a mix of them. So maybe we should focus on some of them first.

[mostlyjason]

Ok can we focus first on the proxy settings that Steffen just added here elastic/beats#25219? This is new functionality and he will be leaving soon and we may need his input. I believe that is the connections with numbers 3, 4, 6 and 7 in the diagram. @urso please correct me if I'm wrong. Is that a reasonable place to start?

[urso]

added here elastic/beats#25219? This is new functionality and he will be leaving soon and we may need his input. I believe that is the connections with numbers 3, 4, 6 and 7 in the diagram.

Correct, this PR adds proxy support (and consistent set of settings) to all these connections. There is another PR for 3 and 5 in the Beats repository. The connection 5 is done in Fleet Server repo and Beats repo. All in all, we introduce the very same set of settings in each place: proxy_url, proxy_headers, proxy_disabled and support for the HTTP_PROXY, HTTPS_PROXY, and NOPROXY environment variables.

[EricDavisX]

There was a bug that was fixed as part of 7.14.1 which has been confirmed as fixed. I don't know which #s it is per above, but I bet it is 3,4,6, and 7. The testing remains to be done for all other connections, though they may (probably) already work. Just fyi.

[EricDavisX]

@lthach2 FYI, and it might be helpful to review + collaborate on the Docs work for proxy enhancements

[EricDavisX]

Proxy support had a few bugs that are included as part of 7.15, the Agent team finished testing and this remains the only remaining piece to close out the meta/tracking ticket for Agent proxy support.

[dedemorton]

I have an initial PR up here: #1239

I went through the rat king of issues to figure this out as best I could, but I need someone to work with me on testing and filling in the missing details because I do not have an environment where I can test this setup.

It sounds like @michel-laterman tested this at one point, and maybe @lthach2 can help?

Who wants to help me take this (American) football over the finish line for 7.16? Add a comment to #1239 if you can help. Thanks!

[gbanasiak]

I think doc update in #1239 lacks description of where to put environment settings such as HTTP_PROXY. Based on customer feedback it's not immediately clear that if Elastic Agent runs as a service in Linux they should add the setting in /etc/sysconfig/elastic-agent instead of changing the system-wide user profile.

[dedemorton]

@gbanasiak Can you add your comment to the review of #1239? That way, reviewers can see your comment and discuss it. Thanks!