diff --git a/CHANGELOG.md b/CHANGELOG.md index 6eda0f90f5e..c7f70fe0031 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,11 @@ release. ## Unreleased +### Protocol + +- Move OTLP specification to github.com/open-telemetry/opentelemetry-proto. + ([#](https://github.com/open-telemetry/opentelemetry-specification/pull/)) + ### Context ### Traces diff --git a/specification/protocol/README.md b/specification/protocol/README.md index 2d7ecec9449..91289369676 100644 --- a/specification/protocol/README.md +++ b/specification/protocol/README.md @@ -1,8 +1,7 @@ # OpenTelemetry Protocol -This is the specification of new OpenTelemetry protocol (OTLP). +The OpenTelemetry protocol (OTLP) specification has moved to +[github.com/open-telemetry/opentelemetry-proto/specification/README.md](https://github.com/open-telemetry/opentelemetry-proto/blob/main/specification/README.md). -- [Design Goals](design-goals.md). -- [Requirements](requirements.md). -- [Specification](otlp.md). -- [SDK Exporter](exporter.md). +For additional OTLP implementation requirements in the OpenTelemetry SDKs, see +[SDK Exporter](exporter.md). diff --git a/specification/protocol/design-goals.md b/specification/protocol/design-goals.md index e51a4d70edf..0b42df1255d 100644 --- a/specification/protocol/design-goals.md +++ b/specification/protocol/design-goals.md @@ -1,19 +1,4 @@ # Design Goals for OpenTelemetry Wire Protocol -We want to design a telemetry data exchange protocol that has the following characteristics: - -- Be suitable for use between all of the following node types: instrumented applications, telemetry backends, local agents, stand-alone collectors/forwarders. - -- Have high reliability of data delivery and clear visibility when the data cannot be delivered. - -- Have low CPU usage for serialization and deserialization. - -- Impose minimal pressure on memory manager, including pass-through scenarios, where deserialized data is short-lived and must be serialized as-is shortly after and where such short-lived data is created and discarded at high frequency (think telemetry data forwarders). - -- Support ability to efficiently modify deserialized data and serialize again to pass further. This is related but slightly different from the previous requirement. - -- Ensure high throughput (within the available bandwidth) in high latency networks (e.g. scenarios where telemetry source and the backend are separated by high latency network). - -- Allow backpressure signalling. - -- Be load-balancer friendly (do not hinder re-balancing). +This page has moved to +[github.com/open-telemetry/opentelemetry-proto/specification/design-goals.md](https://github.com/open-telemetry/opentelemetry-proto/blob/main/specification/design-goals.md). diff --git a/specification/protocol/exporter.md b/specification/protocol/exporter.md index 4d58a4d3870..23124d5fba6 100644 --- a/specification/protocol/exporter.md +++ b/specification/protocol/exporter.md @@ -158,7 +158,7 @@ The `OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_EXPORTER_OTLP_TRACES_HEADERS`, `OTEL_EXP Transient errors MUST be handled with a retry strategy. This retry strategy MUST implement an exponential back-off with jitter to avoid overwhelming the destination until the network is restored or the destination has recovered. -For OTLP/HTTP, the errors `408 (Request Timeout)` and `5xx (Server Errors)` are defined as transient, detailed information about errors can be found in the [HTTP failures section](otlp.md#failures). For the OTLP/gRPC, the full list of the gRPC retryable status codes can be found in the [gRPC response section](otlp.md#otlpgrpc-response). +For OTLP/HTTP, the errors `408 (Request Timeout)` and `5xx (Server Errors)` are defined as transient, detailed information about errors can be found in the [HTTP failures section](https://github.com/open-telemetry/opentelemetry-proto/blob/main/docs/specification.md#failures). For the OTLP/gRPC, the full list of the gRPC retryable status codes can be found in the [gRPC response section](https://github.com/open-telemetry/opentelemetry-proto/blob/main/docs/specification.md#otlpgrpc-response). ## User Agent @@ -171,5 +171,5 @@ OTel-OTLP-Exporter-Python/1.2.3 The format of the header SHOULD follow [RFC 7231][rfc-7231]. The conventions used for specifying the OpenTelemetry SDK language and version are available in the [Resource semantic conventions][resource-semconv]. [resource-semconv]: ../resource/semantic_conventions/README.md#telemetry-sdk -[otlphttp-req]: otlp.md#otlphttp-request +[otlphttp-req]: https://github.com/open-telemetry/opentelemetry-proto/blob/main/docs/specification.md#otlphttp-request [rfc-7231]: https://datatracker.ietf.org/doc/html/rfc7231#section-5.5.3 diff --git a/specification/protocol/file-exporter.md b/specification/protocol/file-exporter.md index 6342967b0e0..296acb264d1 100644 --- a/specification/protocol/file-exporter.md +++ b/specification/protocol/file-exporter.md @@ -42,7 +42,7 @@ There is no guarantee in particular that timestamps will be monotonically increa This defines the first version of the serialization scheme. The data must be encoded according to the format specified in the -[OTLP JSON Encoding](../../specification/protocol/otlp.md#json-protobuf-encoding). +[OTLP JSON Encoding](https://github.com/open-telemetry/opentelemetry-proto/blob/main/docs/specification.md#json-protobuf-encoding). Only top-level objects, `ExportTraceServiceRequest`, `ExportMetricsServiceRequest`, and `ExportLogsServiceRequest` are supported. diff --git a/specification/protocol/otlp.md b/specification/protocol/otlp.md index 83836ad47a4..96716d3a3d2 100644 --- a/specification/protocol/otlp.md +++ b/specification/protocol/otlp.md @@ -1,705 +1,4 @@ # OpenTelemetry Protocol Specification -**Status**: [Stable](../document-status.md) - -The OpenTelemetry Protocol (OTLP) specification describes the encoding, transport, -and delivery mechanism of telemetry data between telemetry sources, intermediate -nodes such as collectors and telemetry backends. - -
-Table of Contents - - - -- [Protocol Details](#protocol-details) - * [OTLP/gRPC](#otlpgrpc) - + [OTLP/gRPC Concurrent Requests](#otlpgrpc-concurrent-requests) - + [OTLP/gRPC Response](#otlpgrpc-response) - - [Full Success](#full-success) - - [Partial Success](#partial-success) - - [Failures](#failures) - + [OTLP/gRPC Throttling](#otlpgrpc-throttling) - + [OTLP/gRPC Service and Protobuf Definitions](#otlpgrpc-service-and-protobuf-definitions) - + [OTLP/gRPC Default Port](#otlpgrpc-default-port) - * [OTLP/HTTP](#otlphttp) - + [Binary Protobuf Encoding](#binary-protobuf-encoding) - + [JSON Protobuf Encoding](#json-protobuf-encoding) - + [OTLP/HTTP Request](#otlphttp-request) - + [OTLP/HTTP Response](#otlphttp-response) - - [Full Success](#full-success-1) - - [Partial Success](#partial-success-1) - - [Failures](#failures-1) - - [Bad Data](#bad-data) - - [OTLP/HTTP Throttling](#otlphttp-throttling) - - [All Other Responses](#all-other-responses) - + [OTLP/HTTP Connection](#otlphttp-connection) - + [OTLP/HTTP Concurrent Requests](#otlphttp-concurrent-requests) - + [OTLP/HTTP Default Port](#otlphttp-default-port) -- [Implementation Recommendations](#implementation-recommendations) - * [Multi-Destination Exporting](#multi-destination-exporting) -- [Known Limitations](#known-limitations) - * [Request Acknowledgements](#request-acknowledgements) - + [Duplicate Data](#duplicate-data) -- [Future Versions and Interoperability](#future-versions-and-interoperability) -- [Glossary](#glossary) -- [References](#references) - - - -
- -OTLP is a general-purpose telemetry data delivery protocol designed in the scope -of the OpenTelemetry project. - -## Protocol Details - -OTLP defines the encoding of telemetry data and the protocol used to exchange -data between the client and the server. - -This specification defines how OTLP is implemented over -[gRPC](https://grpc.io/) and HTTP 1.1 transports and specifies -[Protocol Buffers schema](https://developers.google.com/protocol-buffers/docs/overview) -that is used for the payloads. - -OTLP is a request/response style protocol: the clients send requests, and the -server replies with corresponding responses. This document defines one request -and response type: `Export`. - -All server components MUST support the following transport compression options: - -* No compression, denoted by `none`. -* Gzip compression, denoted by `gzip`. - -### OTLP/gRPC - -After establishing the underlying gRPC transport, the client starts sending -telemetry data using unary requests using -[Export*ServiceRequest](https://github.com/open-telemetry/opentelemetry-proto) -messages ([ExportLogsServiceRequest](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/collector/logs/v1/logs_service.proto) for logs, -[ExportMetricsServiceRequest](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/collector/metrics/v1/metrics_service.proto) for metrics, -[ExportTraceServiceRequest](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/collector/trace/v1/trace_service.proto) for traces). -The client continuously sends a sequence of requests to the server and expects -to receive a response to each request: - -![Request-Response](img/otlp-request-response.png) - -_Note: this protocol is concerned with the reliability of delivery between one -pair of client/server nodes and aims to ensure that no data is lost in transit -between the client and the server. Many telemetry collection systems have -intermediary nodes that the data must travel across until reaching the final -destination (e.g. application -> agent -> collector -> backend). End-to-end -delivery guarantees in such systems is outside of the scope of OTLP. The -acknowledgements described in this protocol happen between a single -client/server pair and do not span intermediary nodes in multi-hop delivery -paths._ - -#### OTLP/gRPC Concurrent Requests - -After sending the request the client MAY wait until the response is received -from the server. In that case there will be at most only one request in flight -that is not yet acknowledged by the server. - -![Unary](img/otlp-sequential.png) - -Sequential operation is recommended when simplicity of implementation is -desirable, and when the client and the server are connected via very low-latency -network, such as when the client is an instrumented application and -the server is an OpenTelemetry Collector running as a local daemon (agent). - -The implementations that need to achieve high throughput SHOULD support -concurrent Unary calls to achieve higher throughput. The client SHOULD send new -requests without waiting for the response to the earlier sent requests, -essentially creating a pipeline of requests that are currently in flight that -are not acknowledged. - -![Concurrent](img/otlp-concurrent.png) - -The number of concurrent requests SHOULD be configurable. - -The maximum achievable throughput is -`max_concurrent_requests * max_request_size / (network_latency + server_response_time)`. -For example, if the request can contain at most 100 spans, network roundtrip -latency is 200ms, and server response time is 300 ms, then the maximum achievable -throughput with one concurrent request is `100 spans / (200ms+300ms)` or 200 -spans per second. It is easy to see that in high latency networks or when the -server response time is high to achieve good throughput, the requests need to be -very big or a lot concurrent requests must be done. - -If the client is shutting down (e.g. when the containing process wants to exit) -the client will optionally wait until all pending acknowledgements are received -or until an implementation-specific timeout expires. This ensures the reliable -delivery of telemetry data. The client implementation SHOULD expose an option to -turn on and off the waiting during a shutdown. - -If the client is unable to deliver a certain request (e.g. a timer expired while -waiting for acknowledgements) the client SHOULD record the fact that the data -was not delivered. - -#### OTLP/gRPC Response - -The response MUST be the appropriate message (see below for -the specific message to use in the [Full Success](#full-success), -[Partial Success](#partial-success) and [Failure](#failures) cases). - -##### Full Success - -The success response indicates telemetry data is successfully accepted by the -server. - -If the server receives an empty request (a request that does not carry -any telemetry data) the server SHOULD respond with success. - -On success, the server response MUST be a -[ExportServiceResponse](https://github.com/open-telemetry/opentelemetry-proto/tree/main/opentelemetry/proto/collector) -message (`ExportTraceServiceResponse` for traces, -`ExportMetricsServiceResponse` for metrics and -`ExportLogsServiceResponse` for logs). - -The server MUST leave the `partial_success` field unset -in case of a successful response. - -##### Partial Success - -If the request is only partially accepted -(i.e. when the server accepts only parts of the data and rejects the rest), the -server response MUST be the same -[ExportServiceResponse](https://github.com/open-telemetry/opentelemetry-proto/tree/main/opentelemetry/proto/collector) -message as in the [Full Success](#full-success) case. - -Additionally, the server MUST initialize the `partial_success` field -(`ExportTracePartialSuccess` message for traces, -`ExportMetricsPartialSuccess` message for metrics and -`ExportLogsPartialSuccess` message for logs), and it MUST set the respective -`rejected_spans`, `rejected_data_points` or `rejected_log_records` field with -the number of spans/data points/log records it rejected. - -The server SHOULD populate the `error_message` field with a human-readable -error message in English. The message should explain why the -server rejected parts of the data and might offer guidance on how users -can address the issues. -The protocol does not attempt to define the structure of the error message. - -Servers MAY also use the `partial_success` field to convey -warnings/suggestions to clients even when the server fully accepts the request. -In such cases, the `rejected_` field MUST have a value of `0`, and -the `error_message` field MUST be non-empty. - -The client MUST NOT retry the request when it receives a partial success -response where the `partial_success` is populated. - -##### Failures - -When the server returns an error, it falls into 2 broad categories: -retryable and not-retryable: - -- Retryable errors indicate that telemetry data processing failed, and the - client SHOULD record the error and may retry exporting the same data. - For example, this can happen when the server is temporarily unable to - process the data. - -- Not-retryable errors indicate that telemetry data processing failed, and the - client MUST NOT retry sending the same telemetry data. The client MUST drop - the telemetry data. - For example, this can happen, when the request contains bad data - and cannot be deserialized or processed by the server. The client - SHOULD maintain a counter of such dropped data. - -The server MUST indicate retryable errors using code -[Unavailable](https://godoc.org/google.golang.org/grpc/codes) and MAY supply -additional -[details via status](https://godoc.org/google.golang.org/grpc/status#Status.WithDetails) -using -[RetryInfo](https://github.com/googleapis/googleapis/blob/6a8c7914d1b79bd832b5157a09a9332e8cbd16d4/google/rpc/error_details.proto#L40) -containing 0 value of RetryDelay. Here is a sample Go code to illustrate: - -```go - // Do this on server side. - st, err := status.New(codes.Unavailable, "Server is unavailable"). - WithDetails(&errdetails.RetryInfo{RetryDelay: &duration.Duration{Seconds: 0}}) - if err != nil { - log.Fatal(err) - } - - return st.Err() -``` - -To indicate not-retryable errors, the server is recommended to use code -[InvalidArgument](https://godoc.org/google.golang.org/grpc/codes) and MAY supply -additional -[details via status](https://godoc.org/google.golang.org/grpc/status#Status.WithDetails) -using -[BadRequest](https://github.com/googleapis/googleapis/blob/6a8c7914d1b79bd832b5157a09a9332e8cbd16d4/google/rpc/error_details.proto#L119). -If more appropriate, another gRPC status code may be used. Here is a -snippet of sample Go code to illustrate: - -```go - // Do this on the server side. - st, err := status.New(codes.InvalidArgument, "Invalid Argument"). - WithDetails(&errdetails.BadRequest{}) - if err != nil { - log.Fatal(err) - } - - return st.Err() -``` - -The server MAY use other gRPC codes to indicate retryable and not-retryable -errors if those other gRPC codes are more appropriate for a particular erroneous -situation. The client SHOULD interpret gRPC status codes as retryable or -not-retryable according to the following table: - -|gRPC Code|Retryable?| -|---------|----------| -|CANCELLED|Yes| -|UNKNOWN|No| -|INVALID_ARGUMENT|No| -|DEADLINE_EXCEEDED|Yes| -|NOT_FOUND|No| -|ALREADY_EXISTS|No| -|PERMISSION_DENIED|No| -|UNAUTHENTICATED|No| -|RESOURCE_EXHAUSTED|Only if the server can recover (see below)| -|FAILED_PRECONDITION|No| -|ABORTED|Yes| -|OUT_OF_RANGE|Yes| -|UNIMPLEMENTED|No| -|INTERNAL|No| -|UNAVAILABLE|Yes| -|DATA_LOSS|Yes| - -When retrying, the client SHOULD implement an exponential backoff strategy. An -exception to this is the Throttling case explained below, which provides -explicit instructions about retrying interval. - -The client SHOULD interpret `RESOURCE_EXHAUSTED` code as retryable only if the -server signals that the recovery from resource exhaustion is possible. -This is signaled by the server by returning -[a status](https://godoc.org/google.golang.org/grpc/status#Status.WithDetails) containing -[RetryInfo](https://github.com/googleapis/googleapis/blob/6a8c7914d1b79bd832b5157a09a9332e8cbd16d4/google/rpc/error_details.proto#L40). -In this case the behavior of the server and the client is exactly as described in -[OTLP/gRPC Throttling](#otlpgrpc-throttling) section. If no such status is returned, -then the `RESOURCE_EXHAUSTED` code SHOULD be treated as non-retryable. - -#### OTLP/gRPC Throttling - -OTLP allows backpressure signaling. - -If the server is unable to keep up with the pace of data it receives from the -client then it SHOULD signal that fact to the client. The client MUST then -throttle itself to avoid overwhelming the server. - -To signal backpressure when using gRPC transport, the server MUST return an -error with code [Unavailable](https://godoc.org/google.golang.org/grpc/codes) -and MAY supply additional -[details via status](https://godoc.org/google.golang.org/grpc/status#Status.WithDetails) -using -[RetryInfo](https://github.com/googleapis/googleapis/blob/6a8c7914d1b79bd832b5157a09a9332e8cbd16d4/google/rpc/error_details.proto#L40). -Here is a snippet of sample Go code to illustrate: - -```go - // Do this on the server side. - st, err := status.New(codes.Unavailable, "Server is unavailable"). - WithDetails(&errdetails.RetryInfo{RetryDelay: &duration.Duration{Seconds: 30}}) - if err != nil { - log.Fatal(err) - } - - return st.Err() - - ... - - // Do this on the client side. - st := status.Convert(err) - for _, detail := range st.Details() { - switch t := detail.(type) { - case *errdetails.RetryInfo: - if t.RetryDelay.Seconds > 0 || t.RetryDelay.Nanos > 0 { - // Wait before retrying. - } - } - } -``` - -When the client receives this signal, it SHOULD follow the recommendations -outlined in documentation for -[RetryInfo](https://github.com/googleapis/googleapis/blob/6a8c7914d1b79bd832b5157a09a9332e8cbd16d4/google/rpc/error_details.proto#L40): - -``` -// Describes when the clients can retry a failed request. Clients could ignore -// the recommendation here or retry when this information is missing from the error -// responses. -// -// It's always recommended that clients should use exponential backoff when -// retrying. -// -// Clients should wait until `retry_delay` amount of time has passed since -// receiving the error response before retrying. If retrying requests also -// fail, clients should use an exponential backoff scheme to increase gradually -// the delay between retries based on `retry_delay` until either a maximum -// number of retries has been reached, or a maximum retry delay cap has been -// reached. -``` - -The value of `retry_delay` is determined by the server and is implementation -dependant. The server SHOULD choose a `retry_delay` value that is big enough to -give the server time to recover yet is not too big to cause the client to drop -data while being throttled. - -#### OTLP/gRPC Service and Protobuf Definitions - -gRPC service definitions -[are here](https://github.com/open-telemetry/opentelemetry-proto/tree/master/opentelemetry/proto/collector). - -Protobuf definitions for requests and responses -[are here](https://github.com/open-telemetry/opentelemetry-proto/tree/master/opentelemetry/proto). - -Please make sure to check the proto version and -[maturity level](https://github.com/open-telemetry/opentelemetry-proto/#maturity-level). -Schemas for different signals may be at different maturity level - some stable, -some in beta. - -#### OTLP/gRPC Default Port - -The default network port for OTLP/gRPC is 4317. - -### OTLP/HTTP - -OTLP/HTTP uses Protobuf payloads encoded either in -[binary format](#binary-protobuf-encoding) or in [JSON format](#json-protobuf-encoding). -Regardless of the encoding the Protobuf schema of the messages is the same for -OTLP/HTTP and OTLP/gRPC as -[defined here](https://github.com/open-telemetry/opentelemetry-proto/tree/master/opentelemetry/proto). - -OTLP/HTTP uses HTTP POST requests to send telemetry data from clients to -servers. Implementations MAY use HTTP/1.1 or HTTP/2 transports. Implementations -that use HTTP/2 transport SHOULD fallback to HTTP/1.1 transport if HTTP/2 -connection cannot be established. - -#### Binary Protobuf Encoding - -Binary Protobuf encoded payloads use proto3 -[encoding standard](https://developers.google.com/protocol-buffers/docs/encoding). - -The client and the server MUST set "Content-Type: application/x-protobuf" -request and response headers when sending binary Protobuf encoded payload. - -#### JSON Protobuf Encoding - -JSON Protobuf encoded payloads use proto3 standard defined -[JSON Mapping](https://developers.google.com/protocol-buffers/docs/proto3#json) -for mapping between Protobuf and JSON, with the following deviations from that mapping: - -- The `trace_id` and `span_id` byte arrays are represented as - [case-insensitive hex-encoded strings](https://tools.ietf.org/html/rfc4648#section-8); - they are not base64-encoded as is defined in the standard - [Protobuf JSON Mapping](https://developers.google.com/protocol-buffers/docs/proto3#json). - Hex encoding is used for `trace_id` and `span_id` fields in all OTLP - Protobuf messages, e.g., the `Span`, `Link`, `LogRecord`, etc. messages. - For example, the `trace_id` field in a Span can be represented like this: - { "trace_id": "5B8EFFF798038103D269B633813FC60C", ... } - -- Values of enum fields MUST be encoded as integer values. Unlike the standard - [Protobuf JSON Mapping](https://developers.google.com/protocol-buffers/docs/proto3#json), - which allows values of enum fields to be encoded as either integer values or - as enum name strings, only integer enum values are allowed in OTLP JSON - Protobuf Encoding; the enum name strings MUST NOT be used. For example, the - `kind` field with a value of SPAN_KIND_SERVER in a Span can be represented - like this: { "kind": 2, ... } - -- OTLP/JSON receivers MUST ignore message fields with unknown names and MUST - unmarshal the message as if the unknown field was not present in the payload. - This aligns with the behavior of the Binary Protobuf unmarshaler and ensures - that adding new fields to OTLP messages does not break existing receivers. - -- The keys of JSON objects are field names converted to lowerCamelCase. Original - field names are not valid to use as keys for JSON objects. - For example, this is a valid JSON representation of a Resource: - `{ "attributes": {...}, "droppedAttributesCount": 123 }`, and this is NOT a valid - representation: - `{ "attributes": {...}, "dropped_attributes_count": 123 }`. - -Note that according to [Protobuf specs]( -https://developers.google.com/protocol-buffers/docs/proto3#json) 64-bit integer -numbers in JSON-encoded payloads are encoded as decimal strings, and either -numbers or strings are accepted when decoding. - -The client and the server MUST set "Content-Type: application/json" request and -response headers when sending JSON Protobuf encoded payload. - -#### OTLP/HTTP Request - -Telemetry data is sent via HTTP POST request. The body of the POST request is a -payload either in binary-encoded Protobuf format or in JSON-encoded Protobuf -format. - -The default URL path for requests that carry trace data is `/v1/traces` (for -example the full URL when connecting to "example.com" server will be -`https://example.com/v1/traces`). The request body is a Protobuf-encoded -`ExportTraceServiceRequest` message. - -The default URL path for requests that carry metric data is `/v1/metrics` and -the request body is a Protobuf-encoded `ExportMetricsServiceRequest` message. - -The default URL path for requests that carry log data is `/v1/logs` and the -request body is a Protobuf-encoded `ExportLogsServiceRequest` message. - -The client MAY gzip the content and in that case MUST include -"Content-Encoding: gzip" request header. The client MAY include -"Accept-Encoding: gzip" request header if it can receive gzip-encoded responses. - -Non-default URL paths for requests MAY be configured on the client and server -sides. - -#### OTLP/HTTP Response - -The response body MUST be the appropriate serialized Protobuf message (see -below for the specific message to use in the [Full Success](#full-success-1), -[Partial Success](#partial-success-1) and [Failure](#failures-1) cases). - -The server MUST set "Content-Type: application/x-protobuf" header if the -response body is binary-encoded Protobuf payload. The server MUST set -"Content-Type: application/json" if the response is JSON-encoded Protobuf -payload. The server MUST use the same "Content-Type" in the response as it -received in the request. - -If the request header "Accept-Encoding: gzip" is present in the request the -server MAY gzip-encode the response and set "Content-Encoding: gzip" response -header. - -##### Full Success - -The success response indicates telemetry data is successfully accepted by the -server. - -If the server receives an empty request (a request that does not carry -any telemetry data) the server SHOULD respond with success. - -On success, the server MUST respond with `HTTP 200 OK`. The response body MUST be -a Protobuf-encoded -[ExportServiceResponse](https://github.com/open-telemetry/opentelemetry-proto/tree/main/opentelemetry/proto/collector) -message (`ExportTraceServiceResponse` for traces, -`ExportMetricsServiceResponse` for metrics and -`ExportLogsServiceResponse` for logs). - -The server MUST leave the `partial_success` field unset -in case of a successful response. - -##### Partial Success - -If the request is only partially accepted -(i.e. when the server accepts only parts of the data and rejects the rest), the -server MUST respond with `HTTP 200 OK`. The response body MUST be the same -[ExportServiceResponse](https://github.com/open-telemetry/opentelemetry-proto/tree/main/opentelemetry/proto/collector) -message as in the [Full Success](#full-success-1) case. - -Additionally, the server MUST initialize the `partial_success` field -(`ExportTracePartialSuccess` message for traces, -`ExportMetricsPartialSuccess` message for metrics and -`ExportLogsPartialSuccess` message for logs), and it MUST set the respective -`rejected_spans`, `rejected_data_points` or `rejected_log_records` field with -the number of spans/data points/log records it rejected. - -The server SHOULD populate the `error_message` field with a human-readable -error message in English. The message should explain why the -server rejected parts of the data and might offer guidance on how users -can address the issues. -The protocol does not attempt to define the structure of the error message. - -Servers MAY also use the `partial_success` field to convey -warnings/suggestions to clients even when it fully accepts the request. -In such cases, the `rejected_` field MUST have a value of `0`, and -the `error_message` field MUST be non-empty. - -The client MUST NOT retry the request when it receives a partial success -response where the `partial_success` is populated. - -##### Failures - -If the processing of the request fails, the server MUST respond with appropriate -`HTTP 4xx` or `HTTP 5xx` status code. See the sections below for more details about -specific failure cases and HTTP status codes that should be used. - -The response body for all `HTTP 4xx` and `HTTP 5xx` responses MUST be a -Protobuf-encoded -[Status](https://godoc.org/google.golang.org/genproto/googleapis/rpc/status#Status) -message that describes the problem. - -This specification does not use `Status.code` field and the server MAY omit -`Status.code` field. The clients are not expected to alter their behavior based -on `Status.code` field but MAY record it for troubleshooting purposes. - -The `Status.message` field SHOULD contain a developer-facing error message as -defined in `Status` message schema. - -The server MAY include `Status.details` field with additional details. Read -below about what this field can contain in each specific failure case. - -The server SHOULD use HTTP response status codes to indicate -retryable and not-retryable errors for a particular erroneous situation. The -client SHOULD honour HTTP response status codes as retryable or not-retryable. -The requests that receive a response status code listed in following table SHOULD -be retried. -All other `4xx` or `5xx` response status codes MUST NOT be retried. - -|HTTP response status code| -|---------| -|429 Too Many Requests| -|502 Bad Gateway| -|503 Service Unavailable| -|504 Gateway Timeout| - -##### Bad Data - -If the processing of the request fails because the request contains data that -cannot be decoded or is otherwise invalid and such failure is permanent, then the -server MUST respond with `HTTP 400 Bad Request`. The `Status.details` field in -the response SHOULD contain a -[BadRequest](https://github.com/googleapis/googleapis/blob/d14bf59a446c14ef16e9931ebfc8e63ab549bf07/google/rpc/error_details.proto#L166) -that describes the bad data. - -The client MUST NOT retry the request when it receives `HTTP 400 Bad Request` -response. - -##### OTLP/HTTP Throttling - -If the server receives more requests than the client is allowed or the server is -overloaded, the server SHOULD respond with `HTTP 429 Too Many Requests` or -`HTTP 503 Service Unavailable` and MAY include -["Retry-After"](https://tools.ietf.org/html/rfc7231#section-7.1.3) header with a -recommended time interval in seconds to wait before retrying. - -The client SHOULD honour the waiting interval specified in the "Retry-After" -header if it is present. If the client receives an `HTTP 429` or an `HTTP 503` -response and the "Retry-After" header is not present in the response, then the -client SHOULD implement an exponential backoff strategy between retries. - -##### All Other Responses - -All other HTTP responses that are not explicitly listed in this document should -be treated according to HTTP specifications. - -If the server disconnects without returning a response, the client SHOULD retry -and send the same request. The client SHOULD implement an exponential backoff -strategy between retries to avoid overwhelming the server. - -#### OTLP/HTTP Connection - -If the client cannot connect to the server, the client SHOULD retry the -connection using an exponential backoff strategy between retries. The interval -between retries must have a random jitter. - -The client SHOULD keep the connection alive between requests. - -Server implementations SHOULD accept OTLP/HTTP with binary-encoded Protobuf -payload and OTLP/HTTP with JSON-encoded Protobuf payload requests on the same -port and multiplex the requests to the corresponding payload decoder based on -the "Content-Type" request header. - -Server implementations MAY accept OTLP/gRPC and OTLP/HTTP requests on the same -port and multiplex the connections to the corresponding transport handler based -on the "Content-Type" request header. - -#### OTLP/HTTP Concurrent Requests - -To achieve higher total throughput, the client MAY send requests using several -parallel HTTP connections. In that case, the maximum number of parallel -connections SHOULD be configurable. - -#### OTLP/HTTP Default Port - -The default network port for OTLP/HTTP is 4318. - -## Implementation Recommendations - -### Multi-Destination Exporting - -An additional complication must be accounted for when one client must send -telemetry data to more than one destination server. When one of the servers -acknowledges the data and the other server does not (yet), the client needs -to decide how to move forward. - -In such a situation, the client SHOULD implement queuing, acknowledgment -handling, and retrying logic per destination. This ensures that servers do not -block each other. The queues SHOULD reference shared, immutable data to be sent, -thus minimizing the memory overhead caused by having multiple queues. - -![Multi-Destination Exporting](img/otlp-multi-destination.png) - -This ensures that all destination servers receive the data regardless of their -speed of reception (within the available limits imposed by the size of the -client-side queue). - -## Known Limitations - -### Request Acknowledgements - -#### Duplicate Data - -In edge cases (e.g. on reconnections, network interruptions, etc) the client has -no way of knowing if recently sent data was delivered if no acknowledgement was -received yet. The client will typically choose to re-send such data to guarantee -delivery, which may result in duplicate data on the server side. This is a -deliberate choice and is considered to be the right tradeoff for telemetry data. - -## Future Versions and Interoperability - -OTLP will evolve and change over time. Future versions of OTLP must be designed -and implemented in a way that ensures that clients and servers that implement -different versions of OTLP can interoperate and exchange telemetry data. Old -clients must be able to talk to new servers and vice versa. Suppose new versions -of OTLP introduce new functionality that cannot be understood and supported by -nodes implementing the old versions of OTLP. In that case, the protocol must -regress to the lowest common denominator from a functional perspective. - -When possible, the interoperability MUST be ensured between all versions of -OTLP that are not declared obsolete. - -OTLP does not use explicit protocol version numbering. OTLP's interoperability -of clients and servers of different versions is based on the following concepts: - -1. OTLP (current and future versions) defines a set of capabilities, some of - which are mandatory, while others are optional. Clients and servers must implement - mandatory capabilities and can choose to implement only a subset of optional - capabilities. - -2. For minor changes to the protocol, future versions and extensions of OTLP are - encouraged to use the Protobuf's ability to evolve the message schema in - a backward-compatible manner. Newer versions of OTLP may add new fields to - messages that will be ignored by clients and servers that do not understand - these fields. In many cases, careful design of such schema changes and correct - choice of default values for new fields is enough to ensure interoperability - of different versions without nodes explicitly detecting that their peer node - has different capabilities. - -3. More significant changes must be explicitly defined as new optional - capabilities in future OTEPs. Such capabilities SHOULD be discovered by - client and server implementations after establishing the underlying - transport. The exact discovery mechanism SHOULD be described in future OTEPs, - which define the new capabilities and typically can be implemented by making - a discovery request/response message exchange from the client to server. The - mandatory capabilities defined by this specification are implied and do not - require discovery. The implementation which supports a new, optional - capability MUST adjust its behavior to match the expectation of a peer that - does not support a particular capability. - -## Glossary - -There are 2 parties involved in telemetry data exchange. In this document the -party that is the source of telemetry data is called the `Client`, the party -that is the destination of telemetry data is called the `Server`. - -![Client-Server](img/otlp-client-server.png) - -Examples of a Client are instrumented applications or sending side of telemetry -collectors, examples of Servers are telemetry backends or receiving side of -telemetry collectors (so a Collector is typically both a Client and a Server -depending on which side you look from). - -Both the Client and the Server are also a `Node`. This term is used in the -document when referring to either one. - -## References - -- [OTEP 0035](https://github.com/open-telemetry/oteps/blob/main/text/0035-opentelemetry-protocol.md) OpenTelemetry Protocol Specification -- [OTEP 0099](https://github.com/open-telemetry/oteps/blob/main/text/0099-otlp-http.md) OTLP/HTTP: HTTP Transport Extension for OTLP -- [OTEP 0122](https://github.com/open-telemetry/oteps/blob/main/text/0122-otlp-http-json.md) OTLP: JSON Encoding for OTLP/HTTP +This page has moved to +[github.com/open-telemetry/opentelemetry-proto/specification/otlp.md](https://github.com/open-telemetry/opentelemetry-proto/blob/main/specification/otlp.md). diff --git a/specification/protocol/requirements.md b/specification/protocol/requirements.md index 5d1f1cb6695..6dfb354c079 100644 --- a/specification/protocol/requirements.md +++ b/specification/protocol/requirements.md @@ -1,99 +1,4 @@ # OpenTelemetry Protocol Requirements -This document will drive OpenTelemetry Protocol design and RFC. - -## Goals - -See the goals of OpenTelemetry Protocol design [here](design-goals.md). - -## Vocabulary - -There are 2 parties involved in telemetry data exchange. In this document the party that is the source of telemetry data is called the Client, the party that is the destination of telemetry data is called the Server. - -Examples of a Client are instrumented applications or sending side of telemetry collectors, examples of Servers are telemetry backends or receiving side of telemetry collectors (so a Collector is typically both a Client and a Server depending on which side you look from). - -## Known Issues with Existing Protocols - -Our experience with OpenCensus and other protocols has been that many of them have one or more of the following drawbacks: - -- High CPU consumption for serialization and especially deserialization of received telemetry data. -- High and frequent CPU consumption by Garbage Collector. -- Lack of delivery guarantees for certain protocols (e.g. stream-based gRPC OpenCensus protocol) which makes troubleshooting of telemetry pipelines difficult. -- Not aware / not cooperating with load balancers resulting in potentially large imbalances in horizontally scaled backends. -- Support either traces or metrics but not both. - -Our goal is to avoid or mitigate these known issues in the new protocol. - -## Requirements - -The following are OpenTelemetry protocol requirements. - -### Supported Node Types - -The protocol must be suitable for use between all of the following node types: instrumented applications, telemetry backends, telemetry agents running as local daemons, stand-alone collector/forwarder services. - -### Supported Data Types - -The protocol must support traces and metrics as data types. - -### Reliability of Delivery - -The protocol must ensure reliable data delivery and clear visibility when the data cannot be delivered. This should be achieved by sending data acknowledgements from the Server to the Client. - -Note that acknowledgements alone are not sufficient to guarantee that: a) no data will be lost and b) no data will be duplicated. Acknowledgements can help to guarantee a) but not b). Guaranteeing both at the same is difficult. Because it is usually preferable for telemetry data to be duplicated than to lose it, we choose to guarantee that there are no data losses while potentially allowing duplicate data. - -Duplicates can typically happen in edge cases (e.g. on reconnections, network interruptions, etc) when the client has no way of knowing if last sent data was delivered. In these cases the client will usually choose to re-send the data to guarantee the delivery which in turn may result in duplicate data on the server side. - -_To avoid having duplicates the client and the server could track sent and delivered items using uniquely identifying ids. The exact mechanism for tracking the ids and performing data de-duplication may be defined at the layer above the protocol layer and is outside the scope of this document._ - -For this reason we have slightly relaxed requirements and consider duplicate data acceptable in rare cases. - -Note: this protocol is concerned with reliability of delivery between one pair of client/server nodes and aims to ensure that no data is lost in-transit between the client and the server. Many telemetry collection systems have multiple nodes that the data must travel across until reaching the final destination (e.g. application -> agent -> collector -> backend). End-to-end delivery guarantees in such systems is outside of the scope for this document. The acknowledgements described in this protocol happen between a single client/server pair and do not span multiple nodes in multi-hop delivery paths. - -### Throughput - -The protocol must ensure high throughput in high latency networks when the client and the server are not in the same data center. - -This requirement may rule out half-duplex protocols. The throughput of half-duplex protocols is highly dependent on network roundtrip time and request size. To achieve good throughput request sizes may be too large to be practical. - -### Compression - -The protocol must achieve high compression ratios for telemetry data. The protocol design must consider batching of telemetry data and grouping of similar data (both can help to achieve better compression using common compression algorithms). - -### Encryption - -Industry standard encryption (e.g. TLS/HTTPS) must be supported. - -### Backpressure Signalling and Throttling - -The protocol must allow backpressure signalling. - -If the server is unable to keep up with the pace of data it receives from the client then it must be able to signal that fact to the client. The client may then throttle itself to avoid overwhelming the server. - -If the underlying transport is a stream that has its own flow control mechanism then the backpressure could be applied by delaying the reading of data from the server’s endpoint which could then be signalled to the client via underlying flow-control. However this approach makes it difficult for the client to distinguish server overloading from network delays (due to e.g. network losses). Such distinction is important for [observability reasons](https://github.com/open-telemetry/opentelemetry-service/pull/188). Because of this it is required for the protocol to allow to explicitly and clearly signal backpressure from the server to the client without relying on implicit signalling using underlying flow-control mechanisms. - -The backpressure signal should include a hint to the client about desirable reduced rate of data. - -### Serialization Performance - -The protocol must have fast data serialization and deserialization characteristics. - -Ideally it must also support very fast pass-through mode (when no modifications to the data are needed), fast “augmenting” or “tagging” of data and partial inspection of data (e.g. check for presence of specific tag). These requirements help to create fast Agents and Collectors. - -### Memory Usage Profile - -The protocol must impose minimal pressure on memory manager, including pass-through scenarios, when deserialized data is short-lived and must be serialized as-is shortly after and when such short-lived data is created and discarded at high frequency (think telemetry data forwarders). - -The implementation of telemetry protocol must aim to minimize the number of memory allocations and dealocations performed during serialization and deserialization and aim to minimize the pressure on Garbage Collection (for GC languages). - -### Level 7 Load Balancer Friendly - -The protocol must allow Level 7 load balancers such as Envoy to re-balance the traffic for each batch of telemetry data. The traffic should not get pinned by a load balancer to one server for the entire duration of telemetry data sending, thus potentially leading to imbalanced load of servers located behind the load balancer. - -### Backwards Compatibility - -The protocol should be possible to evolve over time. It should be possible for nodes that implement different versions of OpenTelemetry protocol to interoperate (while possibly regressing to the lowest common denominator from functional perspective). - -### General Requirements - -The protocol must use well-known, mature encoding and transport mechanisms with ubiquitous availability of implementations in wide selection of languages that are supported by OpenTelemetry. +This page has moved to +[github.com/open-telemetry/opentelemetry-proto/specification/requirements.md](https://github.com/open-telemetry/opentelemetry-proto/blob/main/specification/requirements.md).