diff --git a/docs/sources/tempo/api_docs/_index.md b/docs/sources/tempo/api_docs/_index.md index 7717f43f6ea..6d41da7ba64 100644 --- a/docs/sources/tempo/api_docs/_index.md +++ b/docs/sources/tempo/api_docs/_index.md @@ -900,4 +900,9 @@ service StreamingQuerier { rpc MetricsQueryRange(QueryRangeRequest) returns (stream QueryRangeResponse) {} } ``` + +{{< admonition type="note" >}} +gRPC compression is disabled by default. +Refer to [gRPC compression configuration](https://grafana.com/docs/tempo//configuration/#grpc-compression) for more information. +{{< /admonition >}} \ No newline at end of file diff --git a/docs/sources/tempo/configuration/_index.md b/docs/sources/tempo/configuration/_index.md index c5f79266a84..6e4c9f6839e 100644 --- a/docs/sources/tempo/configuration/_index.md +++ b/docs/sources/tempo/configuration/_index.md @@ -20,6 +20,7 @@ The Tempo configuration options include: - [Server](#server) - [Distributor](#distributor) - [Set max attribute size to help control out of memory errors](#set-max-attribute-size-to-help-control-out-of-memory-errors) + - [gRPC compression](#grpc-compression) - [Ingester](#ingester) - [Metrics-generator](#metrics-generator) - [Query-frontend](#query-frontend) @@ -267,6 +268,31 @@ Use the `tempo_distributor_attributes_truncated_total` metric to track how many For additional information, refer to [Troubleshoot out-of-memory errors](https://grafana.com/docs/tempo//troubleshooting/out-of-memory-errors/). +### gRPC compression + +If you notice increased network traffic or issues, check the gRPC compression settings. + +Tempo 2.7 disabled gRPC compression in the querier and distributor for performance reasons. ([#4429](https://github.com/grafana/tempo/pull/4429)) +Benchmark testing suggested that without compression, queriers and distributors used less CPU and memory. + +However, you may notice an increase in ingester data and network traffic especially for larger clusters. + +You can configure the gRPC compression in the `querier`, `ingester`, and `metrics_generator` clients of the distributor. +To re-enable the compression, use `snappy` with the following settings: + + ```yaml + ingester_client: + grpc_client_config: + grpc_compression: "snappy" + metrics_generator_client: + grpc_client_config: + grpc_compression: "snappy" + querier: + frontend_worker: + grpc_client_config: + grpc_compression: "snappy" +``` + ## Ingester For more information on configuration options, refer to [this file](https://github.com/grafana/tempo/blob/main/modules/ingester/config.go). diff --git a/docs/sources/tempo/release-notes/v2-7.md b/docs/sources/tempo/release-notes/v2-7.md index 8ec6c451e5b..9097c4a3787 100644 --- a/docs/sources/tempo/release-notes/v2-7.md +++ b/docs/sources/tempo/release-notes/v2-7.md @@ -206,22 +206,12 @@ max_spans_per_span_set ### gRPC compression disabled -Disable gRPC compression in the querier and distributor for performance reasons. ([#4429](https://github.com/grafana/tempo/pull/4429)) Check the gRPC compression settings if you see network issues. -If you would like to re-enable it, we recommend 'snappy'. -Use the following settings: - - ```yaml - ingester_client: - grpc_client_config: - grpc_compression: "snappy" - metrics_generator_client: - grpc_client_config: - grpc_compression: "snappy" - querier: - frontend_worker: - grpc_client_config: - grpc_compression: "snappy" - ``` +This release disables gRPC compression in the querier and distributor for performance reasons. ([#4429](https://github.com/grafana/tempo/pull/4429)) +Our benchmark suggests that without compression, queriers and distributors use less CPU and memory. + +If you notice increased network traffic or issues, check the gRPC compression settings. + +Refer to [gRPC compression configuration](https://grafana.com/docs/tempo//configuration/#grpc-compression) for more information. ### Other upgrade considerations diff --git a/docs/sources/tempo/setup/upgrade.md b/docs/sources/tempo/setup/upgrade.md index c3e87000cf3..0b5732de1fb 100644 --- a/docs/sources/tempo/setup/upgrade.md +++ b/docs/sources/tempo/setup/upgrade.md @@ -98,22 +98,14 @@ For Jaeger exporting, set `OTEL_TRACES_EXPORTER=jaeger`.For more information, re ### gRPC compression disabled -Disable gRPC compression in the querier and distributor for performance reasons. ([#4429](https://github.com/grafana/tempo/pull/4429)) Check the gRPC compression settings if you see network issues. -If you would like to re-enable it, we recommend 'snappy'. -Use the following settings: +This release disables gRPC compression in the querier and distributor for performance reasons. ([#4429](https://github.com/grafana/tempo/pull/4429)) +Our benchmark suggests that without compression, queriers and distributors use less CPU and memory. - ``` - ingester_client: - grpc_client_config: - grpc_compression: "snappy" - metrics_generator_client: - grpc_client_config: - grpc_compression: "snappy" - querier: - frontend_worker: - grpc_client_config: - grpc_compression: "snappy" - ``` +However, you may notice an increase in ingester data and network traffic. + +If you notice increased network traffic or issues, check the gRPC compression settings. + +Refer to [gRPC compression configuration](https://grafana.com/docs/tempo//configuration/#grpc-compression) for more information. ### Added, updated, removed, or renamed configuration parameters