Add proxy setup docs for Elastic Agent and Fleet (#1239)

* Add proxy setup docs for Elastic Agent and Fleet

* Add changes from review

* More changes from review

* Apply additional chagnes from the review

* Add section about where to set env vars

* Add revisions from review

* Resolve reamining commments from mostlyjason

docs/en/ingest-management/fleet-agent-proxy-support.asciidoc

@@ -0,0 +1,317 @@
+[[fleet-agent-proxy-support]]
+= Use a proxy server with {agent} and {fleet}
+
+Your organization’s security strategy and other considerations may require you
+to use a proxy server between some components in your deployment. For example,
+you may have a firewall rule that prevents endpoints from connecting directly to
+{es}. In this scenario, you can set up the {agent} to connect to a proxy, then
+the proxy can connect to {es} through the firewall.
+
+Support is available in {agent} and {fleet} for connections through HTTP Connect
+(HTTP 1 only) and Socks5 proxy servers.
+
+This guide describes where a proxy server is allowed in your deployment and how
+to configure proxy settings for {agent} and {fleet}. The steps for deploying the
+proxy server itself are beyond the scope of this article.
+
+NOTE: Some environments require users to authenticate with the proxy. There are
+no explicit settings for proxy authentication in {agent} or {fleet}, except the
+ability to pass credentials in the URL or as keys/tokens in headers, as
+described later.
+
+[discrete]
+[[elastic-agent-proxy-config]]
+== When to configure proxy settings
+
+Configure proxy settings for {agent} when it must connect through a proxy server
+to:
+
+* Download artifacts from `artifacts.elastic.co` for subprocesses or binary
+upgrades
+* Send data to {es}
+* Retrieve agent policies from {fleet-server}
+* Retrieve agent policies from {es} (only needed for agents running {fleet-server})
+
+image::images/agent-proxy-server.png[Image showing connections between {agent}, {fleet-server}, and {es}]
+
+If {fleet} is unable to access the Elastic Package Registry because {kib} is
+behind a proxy server, you may also need to set the registry proxy URL
+in the {kib} configuration.
+
+image::images/fleet-epr-proxy.png[Image showing connections between {fleet} and the Elastic Package Registry]
+
+[discrete]
+[[host-proxy-env-vars]]
+== Set default proxy settings in host environment variables
+
+Set environment variables on the host to configure default proxy settings.
+The {agent} uses host environment settings by default if no proxy settings are
+specified elsewhere. You can override host proxy settings later when you
+configure the {agent} and {fleet} settings. The following environment variables
+are available on the host:
+
+|===
+|Variable |Description
+
+|`HTTP_PROXY`
+|URL of the proxy server for HTTP traffic. 
+
+|`HTTPS_PROXY`
+|URL of the proxy server for HTTPS traffic.
+
+|`NO_PROXY`
+|IP addresses or domain names that should not use the proxy. Supports patterns.
+|===
+
+The proxy URL can be a complete URL or `host[:port]`, in which case the `http`
+scheme is assumed. An error is returned if the value is a different form.
+
+[discrete]
+[[where-to-set-proxy-env-vars]]
+=== Where to set proxy environment variables
+
+The location where you set these environment variables is platform-specific and
+based on the system manager you're using. Here are some examples to get you
+started. For more information about setting environment variables, refer to the
+documentation for your operating system.
+
+* For Windows services, set environment variables for the service in
+the Windows registry.
++
+This PowerShell command sets the
+`HKLM\SYSTEM\CurrentControlSet\Services\Elastic Agent\Environment` registry
+key, then restarts {agent}:
++
+[source,yaml]
+----
+$environment = [string[]]@(
+  "HTTPS_PROXY=https://proxy-hostname:proxy-port",
+  "HTTP_PROXY=http://proxy-hostname:proxy-port"
+  )
+
+Set-ItemProperty "HKLM:SYSTEM\CurrentControlSet\Services\Elastic Agent" -Name Environment -Value $environment
+
+Restart-Service "Elastic Agent"
+----
+
+* For Linux services, the location depends on the distribution you're using.
+For example, you can set environment variables in:
+
+** `/etc/systemd/system/elastic-agent.service` for systems that use `systemd` to
+manage the service. To edit the file, run:
++
+[source,shell]
+----
+sudo systemctl edit --full elastic-agent.service
+----
++
+Then add the environment variables under `[Service]`
++
+[source,shell]
+----
+[Service]
+
+Environment="HTTPS_PROXY=https://my.proxy:8443"
+Environment="HTTP_PROXY=http://my.proxy:8080"
+----
+
+** `/etc/sysconfig/elastic-agent` for Red Hat-like distributions that don't use
+`systemd`.
+
+** `/etc/default/elastic-agent` for Debian and Ubuntu distributions that don't
+use `systemd`.
++
+For example:
++
+[source,shell]
+----
+HTTPS_PROXY=https://my.proxy:8443
+HTTP_PROXY=http://my.proxy:8080
+----
+
+After adding environment variables, restart the service.
+
+[discrete]
+[[proxy-settings-in-agent-policy]]
+== Set {agent} proxy settings in the agent policy
+
+Proxy settings in the {agent} policy override proxy settings specified by
+environment variables. This means you can specify proxy settings for {agent}
+that are different from host or system-level environment settings. The following
+proxy settings are valid in the agent policy:
+
+|===
+|Setting | Description
+
+|`proxy_url`
+| (string) URL of the proxy server. If set, the configured URL is used as a
+proxy for all connection attempts by the component. The value may be either a
+complete URL or a `host[:port]`, in which case the `http` scheme is assumed. If
+a value is not specified through the configuration, then proxy environment
+variables are used. The URL accepts optional `username` and `password` settings
+for authenticating with the proxy. For example:
+`http://<username>:<password>@<proxy host>/`.
+
+|`proxy_headers`
+| (string) Additional headers to send to the proxy during CONNECT requests. You
+can use this setting to pass keys/tokens required for authenticating with the
+proxy.
+
+|`proxy_disable`
+| (boolean) If set to `true`, all proxy settings, including the `HTTP_PROXY` and
+`HTTPS_PROXY` environment variables, are ignored.
+
+|===
+
+[discrete]
+=== Set the proxy for communicating with {es}
+
+To set the proxy for communicating with {es}, specify proxy settings under
+`outputs` in the agent policy. The procedure for doing this depends on
+whether you're running {fleet}-managed or standalone agents:
+
+* For {fleet}-managed agents, specify proxy settings in the {kib} UI:
++
+--
+. Log in to {kib} and go to *Management > {fleet}*.
+
+. Click *{fleet} settings*.
+
+. Under *Elasticsearch output configuration (YAML)*, specify proxy settings for
+connecting to {es}. The proxy settings you specify here are applied to all
+{agent}s enrolled in {fleet}.
++
+[role="screenshot"]
+image::images/fleet-proxy-settings-ui.png[Screen capture of Fleet settings UI showing proxy_url setting]
+--
+
+* For standalone agents, specify proxy settings the `elastic-agent.yml` file. For
+example:
++
+[source,yaml]
+----
+outputs:
+  default:
+    api_key: API-KEY
+    hosts:
+    - https://10.0.1.2:9200
+    proxy_url: http://10.0.1.7:3128
+    type: elasticsearch
+----
+
+For more information, refer to <<elastic-agent-configuration>>.
+
+[discrete]
+=== Set the proxy for downloading artifacts
+
+experimental::[]
+
+To set the proxy for downloading artifacts or binary upgrades from
+`artifacts.elastic.co`, add proxy settings under `agent.download` in the
+either the `fleet.yml` file for {fleet}-managed agents, or
+the `elastic-agent.yml` file for standalone agents.
+
+For example:
+
+[source,yaml]
+----
+agent:
+  id: fe277816-8736-4871-9de0-4850d0af21e5
+  download.proxy_url: http://10.0.1.3:3128
+  logging.level: info
+----
+
+IMPORTANT: There is currently no way to configure the `agent.download.proxy_url`
+setting through a command-line flag or the {fleet} UI. If this capability is
+added in the future, the setting you specify here will be overridden.
+
+[discrete]
+[[cli-proxy-settings]]
+== Set the proxy for retrieving agent policies from {fleet}
+
+If there is a proxy between {agent} and {fleet}, specify proxy settings on the
+command line when you install {agent} and enroll in {fleet}. The settings you
+specify at the command line are added to the `fleet.yml` file installed on the
+system where the {agent} is running.
+
+NOTE: If {kib} is behind a proxy server, you'll still need to
+<<epr-proxy-setting,configure {kib} settings>> to access the package registry.
+
+The `enroll` and `install` commands accept the following flags:
+
+|===
+| CLI flag | Description
+
+|`--proxy-url <url>`
+|URL of the proxy server. The value may be either a complete URL or a
+`host[:port]`, in which case the http scheme is assumed.  The URL accepts optional
+username and password settings for authenticating with the proxy. For example:
+`http://<username>:<password>@<proxy host>/`.
+
+|`--proxy-disabled`
+|If specified, all proxy settings, including the `HTTP_PROXY` and `HTTPS_PROXY`
+environment variables, are ignored.
+
+|`--proxy-header <header name>=<value>`
+|Additional header to send to the proxy during CONNECT requests. Use the
+`--proxy-header` flag multiple times to add additional headers. You can use
+this setting to pass keys/tokens required for authenticating with the proxy.
+
+|===
+
+For example:
+
+[source,sh]
+----
+elastic-agent install -f --url="https://10.0.1.6:8220" --enrollment-token=TOKEN --proxy-url="http://10.0.1.7:3128" --fleet-server-es-ca="/usr/local/share/ca-certificates/es-ca.crt" --certificate-authorities="/usr/local/share/ca-certificates/fleet-ca.crt"
+----
+
+NOTE: This command requires default policies to be loaded in {fleet}. Default
+policies are loaded automatically when you visit {fleet} for the first time. If
+you're not sure whether default policies are loaded, log in to {kib} and go to
+*Management > {fleet}*.
+
+The command in the previous example adds the following settings to the
+`fleet.yml` policy on the host where {agent} is installed:
+
+[source,yaml]
+----
+fleet:
+  enabled: true
+  access_api_key: API-KEY
+  hosts:
+  - https://10.0.1.6:8220
+  ssl:
+    verification_mode: ""
+    certificate_authorities:
+    - /usr/local/share/ca-certificates/es-ca.crt
+    renegotiation: never
+  timeout: 10m0s
+  proxy_url: http://10.0.1.7:3128
+  reporting:
+    threshold: 10000
+    check_frequency_sec: 30
+  agent:
+    id: ""
+----
+
+[discrete]
+[[epr-proxy-setting]]
+== Set the proxy URL of the Elastic Package Registry
+
+{fleet} might be unable to access the Elastic Package Registry because {kib} is
+behind a proxy server.
+
+Also your organization might have network traffic restrictions that prevent {kib}
+from reaching the public Elastic Package Registry endpoints, like
+https://epr.elastic.co/[epr.elastic.co], to download package metadata and
+content. You can route traffic to the public endpoint of EPR through a network
+gateway, then configure proxy settings in the
+{kibana-ref}/fleet-settings-kb.html[{kib} configuration file], `kibana.yml`. For
+example:
+
+[source,yaml]
+----
+xpack.fleet.registryProxyUrl: your-nat-gateway.corp.net
+----
+

docs/en/ingest-management/index.asciidoc

@@ -76,6 +76,8 @@ include::elastic-agent/configuration/env/container-envs.asciidoc[leveloffset=+3]
 
 include::elastic-agent/installation-layout.asciidoc[leveloffset=+2]
 
+include::fleet-agent-proxy-support.asciidoc[leveloffset=+2]
+
 include::elastic-agent/uninstall-elastic-agent.asciidoc[leveloffset=+2]
 
 include::elastic-agent/start-stop-elastic-agent.asciidoc[leveloffset=+2]