Skip to content

Commit

Permalink
Add blog post on hardening the Collector
Browse files Browse the repository at this point in the history
  • Loading branch information
mx-psi committed Jun 27, 2024
1 parent b959bc6 commit 41f057b
Showing 1 changed file with 133 additions and 0 deletions.
133 changes: 133 additions & 0 deletions content/en/blog/2024/hardening-the-collector-one.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
---
title: 'Hardening the Collector #1 - A new default bind address'
linkTitle: A new default bind address for the Collector
date: 2024-07-02
author: '[Pablo Baeyens](https://github.com/mx-psi) (OpenTelemetry, Datadog)'

Check warning on line 5 in content/en/blog/2024/hardening-the-collector-one.md

View workflow job for this annotation

GitHub Actions / SPELLING check

Unknown word (Baeyens)
cSpell:ignore:
Baeyens awsfirehose awsxray loki sapm signalfx skywalking splunk awsproxy

Check warning on line 7 in content/en/blog/2024/hardening-the-collector-one.md

View workflow job for this annotation

GitHub Actions / SPELLING check

Unknown word (Baeyens)

Check warning on line 7 in content/en/blog/2024/hardening-the-collector-one.md

View workflow job for this annotation

GitHub Actions / SPELLING check

Unknown word (awsfirehose)

Check warning on line 7 in content/en/blog/2024/hardening-the-collector-one.md

View workflow job for this annotation

GitHub Actions / SPELLING check

Unknown word (awsxray)

Check warning on line 7 in content/en/blog/2024/hardening-the-collector-one.md

View workflow job for this annotation

GitHub Actions / SPELLING check

Unknown word (loki)

Check warning on line 7 in content/en/blog/2024/hardening-the-collector-one.md

View workflow job for this annotation

GitHub Actions / SPELLING check

Unknown word (sapm)

Check warning on line 7 in content/en/blog/2024/hardening-the-collector-one.md

View workflow job for this annotation

GitHub Actions / SPELLING check

Unknown word (signalfx)

Check warning on line 7 in content/en/blog/2024/hardening-the-collector-one.md

View workflow job for this annotation

GitHub Actions / SPELLING check

Unknown word (skywalking)

Check warning on line 7 in content/en/blog/2024/hardening-the-collector-one.md

View workflow job for this annotation

GitHub Actions / SPELLING check

Unknown word (splunk)

Check warning on line 7 in content/en/blog/2024/hardening-the-collector-one.md

View workflow job for this annotation

GitHub Actions / SPELLING check

Unknown word (awsproxy)
jaegerremotesampling remotetap
---

The OpenTelemetry Collector recently went through a security audit sponsored by
the [CNCF](https://www.cncf.io/). As part of this process we recently published
a security advisory related to a [DoS vulnerability](/blog/2024/cve-2024-36129/)
that was fully addressed in v0.102.1.

The security audit also motivated us to think about ways to harden official
Collector builds and have a more secure default configuration. We are
[actively][releases-586] [working][core-10469] [on several changes][core-10470]
to achieve this and we will be publishing a series of blog posts to keep the
community informed.

One of the changes we have recently been working on is changing the default bind
address for Collector servers, such as those exposed by receivers or extensions
that listen for incoming connections. Up to v0.103.0, the default behavior was
to listen on all network interfaces by using the
[unspecified address `0.0.0.0`](https://en.wikipedia.org/wiki/0.0.0.0) on server
addresses. While this is a convenient default for test cases and development
environments, it is
[not the recommended practice for production environments](https://cwe.mitre.org/data/definitions/1327.html),
since it can expose the Collector servers to unnecessary risks. Starting on
v0.104.0 the default bind address becomes `localhost` for all Collector servers.

It has been a long way to get here. We started discussing changing this in
relation to [CVE-2022-27664](https://github.com/advisories/GHSA-69cg-p879-7622)
on [v0.63.0 (September 2022)][core-6151], when we added a warning and improved
our documentation. On [v0.94.0 (September 2023)][core-8510], we decided to add a
feature gate, `component.UseLocalHostAsDefaultHost` to allow users to opt-in to
the new behavior. Finally, this feature gate was enabled by default on [v0.104.0
(June 2024)][core-10352] motivated by
[CVE-2024-36129](/blog/2024/cve-2024-36129/).

## What is changing?

Starting on v0.104.0, the default bind address of all servers exposed by the
Collector are `localhost` instead of `0.0.0.0`. For example, the OTLP receiver
default endpoints for OTLP/gRPC and OTLP/HTTP are now `localhost:4317` and
`localhost:4318` respectively. The full list of components affected by this
change is:

- [`otlp` receiver](https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver#otlp-receiver)
- [`awsfirehose` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awsfirehosereceiver#aws-kinesis-data-firehose-receiver)
- [`awsxray` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awsxrayreceiver#aws-x-ray-receiver)
- [`influxdb` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/influxdbreceiver#influxdb-receiver)
- [`jaeger` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/jaegerreceiver#jaeger-receiver)
- [`loki` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/lokireceiver#loki-receiver)
- [`opencensus` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/opencensusreceiver#opencensus-receiver)
- [`sapm` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/sapmreceiver#sapm-receiver)
- [`signalfx` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/signalfxreceiver#signalfx-receiver)
- [`skywalking` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/skywalkingreceiver#skywalking-receiver)
- [`splunk_hec` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/splunkhecreceiver#splunk-hec-receiver)
- [`zipkin` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/zipkinreceiver#zipkin-receiver)
- [`zookeeper` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/zookeeperreceiver#zookeeper-receiver)
- [`awsproxy` extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/awsproxy#aws-proxy)
- [`health_check` extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/healthcheckextension#health-check)
- [`jaegerremotesampling` extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/jaegerremotesampling#jaegers-remote-sampling-extension)
- [`remotetap` processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/remotetapprocessor#remote-tap-processor)

When in doubt, check the specific components' documentation to see the new
default values.

## What does it mean to me?

If you are relying on the default configuration you may need to start explicitly
setting the endpoint on your Collector components. For example, if you are using
the following configuration with the OTLP receiver:

```yaml
receivers:
otlp:
protocols:
grpc:
```
You may now need to explicitly set the `otlp::protocols::grpc::endpoint`
[configuration setting](https://github.com/open-telemetry/opentelemetry-collector/blob/v0.103.0/receiver/otlpreceiver/config.md):

```yaml
receivers:
otlp:
protocols:
grpc:
endpoint: ${env:HOST_IP}:4317
```

where the `HOST_IP` environment variable should is set to the bind address you
want to use (e.g. `status.podIP` on Kubernetes).

Note that you are not affected if using the default configuration on the
[OpenTelemetry Collector Helm Chart][helm-chart] or if you use the default
configuration on any of the officially released OpenTelemetry Collector Docker
images.

## How can I prepare for this change?

Since v0.63.0 the Collector logs a warning related to this when you have an
endpoint using the 0.0.0.0 address. Before upgrading, you can check for this
warning and address it. From v0.94.0 to v0.103.0 you can also preview the impact
of this change by [enabling][feature-gate] the
`component.UseLocalHostAsDefaultHost` feature gate.

Addressing this change should be straightforward, however, due to the number of
components that are impacted, starting on v0.104.0 you can temporarily opt out
of this change by disabling the `component.UseLocalHostAsDefaultHost` feature
gate so you can work on addressing this at your own pace. This feature gate will
be marked as stable in a future Collector release, so we recommend addressing
this as soon as possible.

[helm-chart]:
https://github.com/open-telemetry/opentelemetry-helm-charts?tab=readme-ov-file#opentelemetry-collector
[feature-gate]:
https://github.com/open-telemetry/opentelemetry-collector/tree/v0.103.0/featuregate#controlling-gates
[releases-586]:
https://github.com/open-telemetry/opentelemetry-collector-releases/issues/586
[core-6151]:
https://github.com/open-telemetry/opentelemetry-collector/issues/6151
[core-8510]:
https://github.com/open-telemetry/opentelemetry-collector/issues/8510
[core-10469]:
https://github.com/open-telemetry/opentelemetry-collector/issues/10469
[core-10470]:
https://github.com/open-telemetry/opentelemetry-collector/issues/10470
[core-10352]:
https://github.com/open-telemetry/opentelemetry-collector/pull/10352

0 comments on commit 41f057b

Please sign in to comment.