From 839bfaf4bffd8bdf445a3bd656e4c83c5ae86f94 Mon Sep 17 00:00:00 2001 From: Pablo Baeyens Date: Thu, 27 Jun 2024 15:35:11 +0200 Subject: [PATCH 1/7] Add blog post on hardening the Collector --- .../blog/2024/hardening-the-collector-one.md | 132 ++++++++++++++++++ 1 file changed, 132 insertions(+) create mode 100644 content/en/blog/2024/hardening-the-collector-one.md diff --git a/content/en/blog/2024/hardening-the-collector-one.md b/content/en/blog/2024/hardening-the-collector-one.md new file mode 100644 index 000000000000..4c6a4f41c26b --- /dev/null +++ b/content/en/blog/2024/hardening-the-collector-one.md @@ -0,0 +1,132 @@ +--- +title: 'Hardening the Collector #1 - A new default bind address' +linkTitle: A new default bind address for the Collector +date: 2024-07-02 +author: '[Pablo Baeyens](https://github.com/mx-psi) (OpenTelemetry, Datadog)' +# prettier-ignore +cSpell:ignore: awsfirehose awsproxy awsxray Baeyens jaegerremotesampling loki remotetap sapm signalfx skywalking splunk +--- + +The OpenTelemetry Collector recently went through a security audit sponsored by +the [CNCF](https://www.cncf.io/). As part of this process we recently published +a security advisory related to a [DoS vulnerability](/blog/2024/cve-2024-36129/) +that was fully addressed in v0.102.1. + +The security audit also motivated us to think about ways to harden official +Collector builds and have a more secure default configuration. We are +[actively][releases-586] [working][core-10469] [on several changes][core-10470] +to achieve this and we will be publishing a series of blog posts to keep the +community informed. + +One of the changes we have recently been working on is changing the default bind +address for Collector servers, such as those exposed by receivers or extensions +that listen for incoming connections. Up to v0.103.0, the default behavior was +to listen on all network interfaces by using the +[unspecified address `0.0.0.0`](https://en.wikipedia.org/wiki/0.0.0.0) on server +addresses. While this is a convenient default for test cases and development +environments, it is +[not the recommended practice for production environments](https://cwe.mitre.org/data/definitions/1327.html), +since it can expose the Collector servers to unnecessary risks. Starting on +v0.104.0 the default bind address becomes `localhost` for all Collector servers. + +It has been a long way to get here. We started discussing changing this in +relation to [CVE-2022-27664](https://github.com/advisories/GHSA-69cg-p879-7622) +on [v0.63.0 (September 2022)][core-6151], when we added a warning and improved +our documentation. On [v0.94.0 (September 2023)][core-8510], we decided to add a +feature gate, `component.UseLocalHostAsDefaultHost` to allow users to opt-in to +the new behavior. Finally, this feature gate was enabled by default on [v0.104.0 +(June 2024)][core-10352] motivated by +[CVE-2024-36129](/blog/2024/cve-2024-36129/). + +## What is changing? + +Starting on v0.104.0, the default bind address of all servers exposed by the +Collector are `localhost` instead of `0.0.0.0`. For example, the OTLP receiver +default endpoints for OTLP/gRPC and OTLP/HTTP are now `localhost:4317` and +`localhost:4318` respectively. The full list of components affected by this +change is: + +- [`otlp` receiver](https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver#otlp-receiver) +- [`awsfirehose` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awsfirehosereceiver#aws-kinesis-data-firehose-receiver) +- [`awsxray` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awsxrayreceiver#aws-x-ray-receiver) +- [`influxdb` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/influxdbreceiver#influxdb-receiver) +- [`jaeger` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/jaegerreceiver#jaeger-receiver) +- [`loki` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/lokireceiver#loki-receiver) +- [`opencensus` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/opencensusreceiver#opencensus-receiver) +- [`sapm` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/sapmreceiver#sapm-receiver) +- [`signalfx` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/signalfxreceiver#signalfx-receiver) +- [`skywalking` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/skywalkingreceiver#skywalking-receiver) +- [`splunk_hec` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/splunkhecreceiver#splunk-hec-receiver) +- [`zipkin` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/zipkinreceiver#zipkin-receiver) +- [`zookeeper` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/zookeeperreceiver#zookeeper-receiver) +- [`awsproxy` extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/awsproxy#aws-proxy) +- [`health_check` extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/healthcheckextension#health-check) +- [`jaegerremotesampling` extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/jaegerremotesampling#jaegers-remote-sampling-extension) +- [`remotetap` processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/remotetapprocessor#remote-tap-processor) + +When in doubt, check the specific components' documentation to see the new +default values. + +## What does it mean to me? + +If you are relying on the default configuration you may need to start explicitly +setting the endpoint on your Collector components. For example, if you are using +the following configuration with the OTLP receiver: + +```yaml +receivers: + otlp: + protocols: + grpc: +``` + +You may now need to explicitly set the `otlp::protocols::grpc::endpoint` +[configuration setting](https://github.com/open-telemetry/opentelemetry-collector/blob/v0.103.0/receiver/otlpreceiver/config.md): + +```yaml +receivers: + otlp: + protocols: + grpc: + endpoint: ${env:HOST_IP}:4317 +``` + +where the `HOST_IP` environment variable should is set to the bind address you +want to use (e.g. `status.podIP` on Kubernetes). + +Note that you are not affected if using the default configuration on the +[OpenTelemetry Collector Helm Chart][helm-chart] or if you use the default +configuration on any of the officially released OpenTelemetry Collector Docker +images. + +## How can I prepare for this change? + +Since v0.63.0 the Collector logs a warning related to this when you have an +endpoint using the 0.0.0.0 address. Before upgrading, you can check for this +warning and address it. From v0.94.0 to v0.103.0 you can also preview the impact +of this change by [enabling][feature-gate] the +`component.UseLocalHostAsDefaultHost` feature gate. + +Addressing this change should be straightforward, however, due to the number of +components that are impacted, starting on v0.104.0 you can temporarily opt out +of this change by disabling the `component.UseLocalHostAsDefaultHost` feature +gate so you can work on addressing this at your own pace. This feature gate will +be marked as stable in a future Collector release, so we recommend addressing +this as soon as possible. + +[helm-chart]: + https://github.com/open-telemetry/opentelemetry-helm-charts?tab=readme-ov-file#opentelemetry-collector +[feature-gate]: + https://github.com/open-telemetry/opentelemetry-collector/tree/v0.103.0/featuregate#controlling-gates +[releases-586]: + https://github.com/open-telemetry/opentelemetry-collector-releases/issues/586 +[core-6151]: + https://github.com/open-telemetry/opentelemetry-collector/issues/6151 +[core-8510]: + https://github.com/open-telemetry/opentelemetry-collector/issues/8510 +[core-10469]: + https://github.com/open-telemetry/opentelemetry-collector/issues/10469 +[core-10470]: + https://github.com/open-telemetry/opentelemetry-collector/issues/10470 +[core-10352]: + https://github.com/open-telemetry/opentelemetry-collector/pull/10352 From 729082fcaff257ad03b5d6b1bcf3a08d9205e4d4 Mon Sep 17 00:00:00 2001 From: Pablo Baeyens Date: Thu, 27 Jun 2024 16:00:18 +0200 Subject: [PATCH 2/7] Update content/en/blog/2024/hardening-the-collector-one.md Co-authored-by: Severin Neumann --- content/en/blog/2024/hardening-the-collector-one.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/en/blog/2024/hardening-the-collector-one.md b/content/en/blog/2024/hardening-the-collector-one.md index 4c6a4f41c26b..b7ae1b6e48fd 100644 --- a/content/en/blog/2024/hardening-the-collector-one.md +++ b/content/en/blog/2024/hardening-the-collector-one.md @@ -5,6 +5,8 @@ date: 2024-07-02 author: '[Pablo Baeyens](https://github.com/mx-psi) (OpenTelemetry, Datadog)' # prettier-ignore cSpell:ignore: awsfirehose awsproxy awsxray Baeyens jaegerremotesampling loki remotetap sapm signalfx skywalking splunk +issue: 4760 +sig: Collector SIG --- The OpenTelemetry Collector recently went through a security audit sponsored by From ee495d2ed50b7380d8ba605e32df92eed22b974d Mon Sep 17 00:00:00 2001 From: Pablo Baeyens Date: Thu, 27 Jun 2024 16:32:17 +0200 Subject: [PATCH 3/7] Apply suggestions from code review Co-authored-by: Fabrizio Ferri-Benedetti --- content/en/blog/2024/hardening-the-collector-one.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/blog/2024/hardening-the-collector-one.md b/content/en/blog/2024/hardening-the-collector-one.md index b7ae1b6e48fd..1c3ce857eb10 100644 --- a/content/en/blog/2024/hardening-the-collector-one.md +++ b/content/en/blog/2024/hardening-the-collector-one.md @@ -1,5 +1,5 @@ --- -title: 'Hardening the Collector #1 - A new default bind address' +title: 'Hardening the Collector Episode 1: A new default bind address' linkTitle: A new default bind address for the Collector date: 2024-07-02 author: '[Pablo Baeyens](https://github.com/mx-psi) (OpenTelemetry, Datadog)' @@ -94,7 +94,7 @@ receivers: ``` where the `HOST_IP` environment variable should is set to the bind address you -want to use (e.g. `status.podIP` on Kubernetes). +want to use (for example, `status.podIP` on Kubernetes). Note that you are not affected if using the default configuration on the [OpenTelemetry Collector Helm Chart][helm-chart] or if you use the default From 0c360aa2b127f2633a9c50de6c506acffc4612c9 Mon Sep 17 00:00:00 2001 From: Pablo Baeyens Date: Thu, 27 Jun 2024 16:37:25 +0200 Subject: [PATCH 4/7] Add link to changelog --- content/en/blog/2024/hardening-the-collector-one.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/en/blog/2024/hardening-the-collector-one.md b/content/en/blog/2024/hardening-the-collector-one.md index 1c3ce857eb10..f7ccf8757c59 100644 --- a/content/en/blog/2024/hardening-the-collector-one.md +++ b/content/en/blog/2024/hardening-the-collector-one.md @@ -12,7 +12,8 @@ sig: Collector SIG The OpenTelemetry Collector recently went through a security audit sponsored by the [CNCF](https://www.cncf.io/). As part of this process we recently published a security advisory related to a [DoS vulnerability](/blog/2024/cve-2024-36129/) -that was fully addressed in v0.102.1. +that was +[fully addressed in v0.102.1](https://github.com/open-telemetry/opentelemetry-collector/releases/tag/v0.102.1). The security audit also motivated us to think about ways to harden official Collector builds and have a more secure default configuration. We are From a33972ccd579dc20111eac28f34f01cf0fc26a09 Mon Sep 17 00:00:00 2001 From: Pablo Baeyens Date: Thu, 27 Jun 2024 17:00:58 +0200 Subject: [PATCH 5/7] Address feedback from review --- .../blog/2024/hardening-the-collector-one.md | 27 +++++++++++++------ 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/content/en/blog/2024/hardening-the-collector-one.md b/content/en/blog/2024/hardening-the-collector-one.md index f7ccf8757c59..bdbd1f869c82 100644 --- a/content/en/blog/2024/hardening-the-collector-one.md +++ b/content/en/blog/2024/hardening-the-collector-one.md @@ -4,22 +4,26 @@ linkTitle: A new default bind address for the Collector date: 2024-07-02 author: '[Pablo Baeyens](https://github.com/mx-psi) (OpenTelemetry, Datadog)' # prettier-ignore -cSpell:ignore: awsfirehose awsproxy awsxray Baeyens jaegerremotesampling loki remotetap sapm signalfx skywalking splunk +cSpell:ignore: awsfirehose awsproxy awsxray Baeyens jaegerremotesampling loki OSTIF remotetap sapm signalfx skywalking splunk issue: 4760 sig: Collector SIG --- The OpenTelemetry Collector recently went through a security audit sponsored by -the [CNCF](https://www.cncf.io/). As part of this process we recently published -a security advisory related to a [DoS vulnerability](/blog/2024/cve-2024-36129/) -that was +the [CNCF](https://www.cncf.io/), facilitated by [OSTIF](https://ostif.org/), +and performed by [7ASecurity](https://7asecurity.com/). As part of this process +we recently published a security advisory related to a +[DoS vulnerability](/blog/2024/cve-2024-36129/) that was [fully addressed in v0.102.1](https://github.com/open-telemetry/opentelemetry-collector/releases/tag/v0.102.1). The security audit also motivated us to think about ways to harden official -Collector builds and have a more secure default configuration. We are -[actively][releases-586] [working][core-10469] [on several changes][core-10470] -to achieve this and we will be publishing a series of blog posts to keep the -community informed. +Collector builds and have a more secure default configuration. We are working on +adopting [several][releases-586] [best][core-10469] [practices][core-10470] that +were recommended in the audit to achieve this and we will be publishing a series +of blog posts to keep the community informed. While we expect the report to be +made public soon, we can already say that we are very satisfied with the +confirmation that the Collector has proven to be very secure, highlighting the +secure coding practices and processes we already have in place. One of the changes we have recently been working on is changing the default bind address for Collector servers, such as those exposed by receivers or extensions @@ -117,6 +121,13 @@ gate so you can work on addressing this at your own pace. This feature gate will be marked as stable in a future Collector release, so we recommend addressing this as soon as possible. +## What's next? + +As we work on adopting the best practices recommended by the security audit, we +will be publishing more blog posts to keep the community informed. This will +include hardening the Collector binaries on macOS and further the +default behavior of Collector servers. Stay tuned! + [helm-chart]: https://github.com/open-telemetry/opentelemetry-helm-charts?tab=readme-ov-file#opentelemetry-collector [feature-gate]: From 4df0bb4759e0f200cab2d010e59a2b61800fd601 Mon Sep 17 00:00:00 2001 From: Pablo Baeyens Date: Thu, 27 Jun 2024 17:06:34 +0200 Subject: [PATCH 6/7] Proofreading --- .../blog/2024/hardening-the-collector-one.md | 28 +++++++++---------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/content/en/blog/2024/hardening-the-collector-one.md b/content/en/blog/2024/hardening-the-collector-one.md index bdbd1f869c82..3ae11d4efd75 100644 --- a/content/en/blog/2024/hardening-the-collector-one.md +++ b/content/en/blog/2024/hardening-the-collector-one.md @@ -12,7 +12,7 @@ sig: Collector SIG The OpenTelemetry Collector recently went through a security audit sponsored by the [CNCF](https://www.cncf.io/), facilitated by [OSTIF](https://ostif.org/), and performed by [7ASecurity](https://7asecurity.com/). As part of this process -we recently published a security advisory related to a +we published a security advisory related to a [DoS vulnerability](/blog/2024/cve-2024-36129/) that was [fully addressed in v0.102.1](https://github.com/open-telemetry/opentelemetry-collector/releases/tag/v0.102.1). @@ -25,10 +25,10 @@ made public soon, we can already say that we are very satisfied with the confirmation that the Collector has proven to be very secure, highlighting the secure coding practices and processes we already have in place. -One of the changes we have recently been working on is changing the default bind -address for Collector servers, such as those exposed by receivers or extensions -that listen for incoming connections. Up to v0.103.0, the default behavior was -to listen on all network interfaces by using the +One of the changes we have been working on is changing the default bind address +for Collector servers, such as those exposed by receivers or extensions that +listen for incoming connections. Up to v0.103.0, the default behavior was to +listen on all network interfaces by using the [unspecified address `0.0.0.0`](https://en.wikipedia.org/wiki/0.0.0.0) on server addresses. While this is a convenient default for test cases and development environments, it is @@ -36,13 +36,13 @@ environments, it is since it can expose the Collector servers to unnecessary risks. Starting on v0.104.0 the default bind address becomes `localhost` for all Collector servers. -It has been a long way to get here. We started discussing changing this in -relation to [CVE-2022-27664](https://github.com/advisories/GHSA-69cg-p879-7622) -on [v0.63.0 (September 2022)][core-6151], when we added a warning and improved -our documentation. On [v0.94.0 (September 2023)][core-8510], we decided to add a +It has been a long way to get here. We started discussing this in relation to +[CVE-2022-27664](https://github.com/advisories/GHSA-69cg-p879-7622) on [v0.63.0 +(September 2022)][core-6151], when we added a warning and improved our +documentation. On [v0.94.0 (September 2023)][core-8510], we decided to add a feature gate, `component.UseLocalHostAsDefaultHost` to allow users to opt-in to the new behavior. Finally, this feature gate was enabled by default on [v0.104.0 -(June 2024)][core-10352] motivated by +(June 2024)][core-10352] motivated by the security audit and [CVE-2024-36129](/blog/2024/cve-2024-36129/). ## What is changing? @@ -98,8 +98,8 @@ receivers: endpoint: ${env:HOST_IP}:4317 ``` -where the `HOST_IP` environment variable should is set to the bind address you -want to use (for example, `status.podIP` on Kubernetes). +where the `HOST_IP` environment variable is set to the bind address you want to +use (for example, `status.podIP` on Kubernetes). Note that you are not affected if using the default configuration on the [OpenTelemetry Collector Helm Chart][helm-chart] or if you use the default @@ -125,8 +125,8 @@ this as soon as possible. As we work on adopting the best practices recommended by the security audit, we will be publishing more blog posts to keep the community informed. This will -include hardening the Collector binaries on macOS and further the -default behavior of Collector servers. Stay tuned! +include hardening the Collector binaries on macOS and further the default +behavior of Collector servers. Stay tuned! [helm-chart]: https://github.com/open-telemetry/opentelemetry-helm-charts?tab=readme-ov-file#opentelemetry-collector From ecb417d354ce4fdf7429db6e1d05c40ddf6bb314 Mon Sep 17 00:00:00 2001 From: Pablo Baeyens Date: Thu, 27 Jun 2024 17:38:06 +0200 Subject: [PATCH 7/7] Reword part about changes in Helm chart --- content/en/blog/2024/hardening-the-collector-one.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/content/en/blog/2024/hardening-the-collector-one.md b/content/en/blog/2024/hardening-the-collector-one.md index 3ae11d4efd75..12d92c0976d3 100644 --- a/content/en/blog/2024/hardening-the-collector-one.md +++ b/content/en/blog/2024/hardening-the-collector-one.md @@ -45,7 +45,7 @@ the new behavior. Finally, this feature gate was enabled by default on [v0.104.0 (June 2024)][core-10352] motivated by the security audit and [CVE-2024-36129](/blog/2024/cve-2024-36129/). -## What is changing? +## What have we changed? Starting on v0.104.0, the default bind address of all servers exposed by the Collector are `localhost` instead of `0.0.0.0`. For example, the OTLP receiver @@ -74,6 +74,11 @@ change is: When in doubt, check the specific components' documentation to see the new default values. +Starting on the [OpenTelemetry Collector Helm Chart][helm-chart] v0.47.1 and on +v0.87.0 of the OpenTelemetry Collector official Docker images we updated the +default configuration for all components to explicitly set the endpoints to an +explicit value. + ## What does it mean to me? If you are relying on the default configuration you may need to start explicitly @@ -101,10 +106,8 @@ receivers: where the `HOST_IP` environment variable is set to the bind address you want to use (for example, `status.podIP` on Kubernetes). -Note that you are not affected if using the default configuration on the -[OpenTelemetry Collector Helm Chart][helm-chart] or if you use the default -configuration on any of the officially released OpenTelemetry Collector Docker -images. +Because of the changes in the Collector Helm Chart and Collector Docker images +you are not affected if using the default configuration on either of these. ## How can I prepare for this change?