Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP client span clarification and establishing HTTP connection spec #3234

Closed
wants to merge 2 commits into from
Closed

HTTP client span clarification and establishing HTTP connection spec #3234

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cd72ae4
HTTP client span clarification and establishing HTTP connection spec
Feb 20, 2023
869bc5e
Apply suggestions from code review
Feb 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions semantic_conventions/trace/http.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,26 @@ groups:
When [request target](https://www.rfc-editor.org/rfc/rfc9110.html#target.resource) is absolute URI, `net.peer.name` MUST match
URI port identifier, otherwise it MUST match `Host` header port identifier.

- id: trace.http.client_connect
type: span
span_kind: client
brief: 'Semantic Convention for establishing an HTTP connection'
attributes:
- ref: net.peer.name
sampling_relevant: true
requirement_level: required
brief: >
Host identifier of the ["URI origin"](https://www.rfc-editor.org/rfc/rfc9110.html#name-uri-origin) HTTP request is sent to.
- ref: net.peer.port
sampling_relevant: true
brief: >
Port identifier of the ["URI origin"](https://www.rfc-editor.org/rfc/rfc9110.html#name-uri-origin) HTTP request is sent to.
- ref: net.sock.peer.addr
- ref: net.sock.peer.port
- ref: net.sock.peer.name
- ref: net.sock.family
examples: ['inet', 'inet6']

- id: trace.http.server
prefix: http
type: span
Expand Down
118 changes: 109 additions & 9 deletions specification/trace/semantic_conventions/http.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ and various HTTP versions like 1.1, 2 and SPDY.
- [Common Attributes](#common-attributes)
* [HTTP request and response headers](#http-request-and-response-headers)
- [HTTP client](#http-client)
* [Establishing an HTTP connection](#establishing-an-http-connection)
* [HTTP request retries and redirects](#http-request-retries-and-redirects)
- [HTTP server](#http-server)
* [HTTP server definitions](#http-server-definitions)
Expand All @@ -24,6 +25,8 @@ and various HTTP versions like 1.1, 2 and SPDY.
* [HTTP client retries examples](#http-client-retries-examples)
* [HTTP client authorization retry examples](#http-client-authorization-retry-examples)
* [HTTP client redirects examples](#http-client-redirects-examples)
* [HTTP client establishing connection examples](#http-client-establishing-connection-examples)
* [HTTP client establishing connection error examples](#http-client-establishing-connection-error-examples)

<!-- tocstop -->

Expand Down Expand Up @@ -129,12 +132,20 @@ Users MAY explicitly configure instrumentations to capture them even though it i

## HTTP client

This span type represents an outbound HTTP request.
This span type represents an outbound HTTP request. There are two ways this can be achieved in an instrumentation:

For an HTTP client span, `SpanKind` MUST be `Client`.
1. Instrumentations SHOULD create an HTTP span for each attempt to send an HTTP request over the wire.
In case the request is resent, the resend attempts MUST follow the [HTTP resend spec](#http-request-retries-and-redirects).
Instrumentations MUST NOT emit a "logical" (encompassing) HTTP span.
Since establishing an HTTP connection happens before the client attempts to send an HTTP requests,
instrumentations SHOULD emit a [CONNECT span](#establishing-an-http-connection) whenever a new HTTP connection is made.
Copy link
Contributor

@lmolkova lmolkova Feb 23, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that connect span does not cover all cases. If we follow this approach, to catch other problems, we'd need DNS span and read-stream span and whatnot. I think they can be useful, but verbose and don't solve all problems.

Moreover, in scope of an encompassing span, there could be operations that don't involve any physical HTTP, TCP or TLS operation. E.g. when circuit-breaking or client-side throttling is used.

I suggest an alternative approach:

  1. if instrumentation can create HTTP spans for each attempt, it SHOULD. If it can also instrument logical encompassing operation, it MAY create it, but it's not covered in HTTP semconv v1 (and may be covered in v1.1+). It's a different span, it does not have HTTP client semantics

If instrumented HTTP client includes DNS and establishing connection into the first try, outer operation does not help much. When these things happen before the instrumentation point, instrumentation SHOULD add outer span too.

  1. if it can't, it SHOULD create HTTP span for topmost operation (as in the p2 there)

With this approach, we can also add CONNECT, DNS, and any other related events and exceptions into the outer operation.
I think the presence of the outer span should be configurable. E.g. if I use a well-known retry library or client SDK, they should provide outer operation instead of HTTP client instrumentation.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Example (with a couple of other possible error points)

image

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that connect span does not cover all cases. If we follow this approach, to catch other problems, we'd need DNS span and read-stream span and whatnot. I think they can be useful, but verbose and don't solve all problems.

Yeah, that's true.

  1. if instrumentation can create HTTP spans for each attempt, it SHOULD. If it can also instrument logical encompassing operation, it MAY create it, but it's not covered in HTTP semconv v1 (and may be covered in v1.1+). It's a different span, it does not have HTTP client semantics

I like the idea of the encompassing span that serves as a sort of catch-all net. It worries me that we're not defining it at all though -- if I were to instrument over-the-wire HTTP requests (and the spec says I SHOULD if it's doable) then I would possibly limit the observability of our instrumentations (e.g. all the network/implementation related events/errors); and since the encompassing span is not really well defined I probably couldn't implement it in a stable instrumentation.

WDYT about emitting both the top-most HTTP span and the over-the-wire HTTP spans? Perhaps with an attribute/annotation explaining that it's a "logical" HTTP span.

  1. if it can't, it SHOULD create HTTP span for topmost operation (as in the p2 there)

👍

With this approach, we can also add CONNECT, DNS, and any other related events and exceptions into the outer operation.

👍

Copy link
Contributor

@lmolkova lmolkova Feb 23, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that we should somewhat define the outer span, but I don't know if it has to follow any semantic conventions and if it needs to be spec-ed out.

E.g. we can say something along these lines:

  • (when applicable...) instrumentation SHOULD create outer logical INTERNAL(?) span and record connection errors and events that happen before, after, or in between physical HTTP attempts. If no additional information about this logical operation is available, this span SHOULD have "Logical HTTP {VERB}" (?) name or the name of http client class and method called by user such as HttpClient.send.

If this span has HTTP semantics, it will appear in the results of all queries that match span name or attributes and would skew the results of such queries.
It's a common problem for any other network instrumentation, and essentially users may want to create such spans themselves to track the overall success/duration of logical operations, so my guess is that we don't need it to have any specific semantics beyond what general otel spec defines.

For example, if user puts @WithSpan around their function that prepares request, sends and retries it, and parses the response, it's good enough. And even though we might want to add more stuff on this span, we don't have to define it right away. Correct?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • (when applicable...) instrumentation SHOULD create outer logical INTERNAL(?) span and record connection errors and events that happen before, after, or in between physical HTTP attempts. If no additional information about this logical operation is available, this span SHOULD have "Logical HTTP {VERB}" (?) name or the name of http client class and method called by user such as HttpClient.send.

Hmm, I suppose we could use the code semconv for that outer span. At least they would provide some attributes for sampling.

If this span has HTTP semantics, it will appear in the results of all queries that match span name or attributes and would skew the results of such queries.

👍

For example, if user puts @WithSpan around their function that prepares request, sends and retries it, and parses the response, it's good enough. And even though we might want to add more stuff on this span, we don't have to define it right away. Correct?

Yeah, that sounds correct.

So, to sum up: "low-level" HTTP instrumentations SHOULD/MAY emit outer INTERNAL span with code semantics, enabled by default, users can opt-out per instrumentation lib.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this name is confusing, since there is also the HTTP verb "CONNECT", which presumably is unrelated.


2. If for some reason it is not possible to emit a span for each send attempt (because e.g. the instrumented library does not expose hooks that would allow this),
instrumentations MAY create an HTTP span for the top-most operation of the HTTP client.
In this case, the `http.url` MUST be the originally requested URL, before any HTTP-redirects that may happen when executing the request.
The instrumentation MUST NOT set the `http.resend_count` attribute.

If set, `http.url` must be the originally requested URL,
before any HTTP-redirects that may happen when executing the request.
For an HTTP client span, `SpanKind` MUST be `Client`.

<!-- semconv trace.http.client(full) -->
| Attribute | Type | Description | Examples | Requirement Level |
Expand Down Expand Up @@ -169,18 +180,63 @@ Following attributes MUST be provided **at span creation time** (when provided a

Note that in some cases host and port identifiers in the `Host` header might be different from the `net.peer.name` and `net.peer.port`, in this case instrumentation MAY populate `Host` header on `http.request.header.host` attribute even if it's not enabled by user.

### Establishing an HTTP connection

If an instrumented HTTP client library allows capturing connection level information, instrumentations SHOULD emit a CONNECT span whenever a new HTTP connection is made.
A CONNECT span contains information that might not get captured by an HTTP client span.
In particular, if the connection was not established due to some network failure, the HTTP client might return with an error before any HTTP client span is emitted.

Spans emitted when an HTTP connection is established should simply be named `CONNECT`.
CONNECT spans are not HTTP spans; they do not represent an outgoing HTTP requests and SHOULD NOT contain any `http.*` attributes.
Copy link
Member

@Oberon00 Oberon00 Feb 24, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CONNECT spans are not HTTP spans

Then, shouldn't we define such a span in some more general network semantic conventions?


For a CONNECT span, `SpanKind` MUST be `Client`.

<!-- semconv trace.http.client_connect(full) -->
| Attribute | Type | Description | Examples | Requirement Level |
|---|---|---|---|---|
| [`net.peer.name`](span-general.md) | string | Host identifier of the ["URI origin"](https://www.rfc-editor.org/rfc/rfc9110.html#name-uri-origin) HTTP request is sent to. [1] | `example.com` | Required |
| [`net.peer.port`](span-general.md) | int | Port identifier of the ["URI origin"](https://www.rfc-editor.org/rfc/rfc9110.html#name-uri-origin) HTTP request is sent to. | `80`; `8080`; `443` | Recommended |
| [`net.sock.family`](span-general.md) | string | Protocol [address family](https://man7.org/linux/man-pages/man7/address_families.7.html) which is used for communication. | `inet`; `inet6` | Conditionally Required: [2] |
| [`net.sock.peer.addr`](span-general.md) | string | Remote socket peer address: IPv4 or IPv6 for internet protocols, path for local communication, [etc](https://man7.org/linux/man-pages/man7/address_families.7.html). | `127.0.0.1`; `/tmp/mysql.sock` | Recommended |
| [`net.sock.peer.name`](span-general.md) | string | Remote socket peer name. | `proxy.example.com` | Recommended: [3] |
| [`net.sock.peer.port`](span-general.md) | int | Remote socket peer port. | `16456` | Recommended: [4] |

**[1]:** `net.peer.name` SHOULD NOT be set if capturing it would require an extra DNS lookup.

**[2]:** If different than `inet` and if any of `net.sock.peer.addr` or `net.sock.host.addr` are set. Consumers of telemetry SHOULD accept both IPv4 and IPv6 formats for the address in `net.sock.peer.addr` if `net.sock.family` is not set. This is to support instrumentations that follow previous versions of this document.

**[3]:** If available and different from `net.peer.name` and if `net.sock.peer.addr` is set.

**[4]:** If defined for the address family and if different than `net.peer.port` and if `net.sock.peer.addr` is set.

Following attributes MUST be provided **at span creation time** (when provided at all), so they can be considered for sampling decisions:

* [`net.peer.name`](span-general.md)
* [`net.peer.port`](span-general.md)

`net.sock.family` has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

| Value | Description |
|---|---|
| `inet` | IPv4 address |
| `inet6` | IPv6 address |
| `unix` | Unix domain socket path |
<!-- endsemconv -->

See the examples of a [successful CONNECT span and the subsequent HTTP request](#http-client-establishing-connection-examples) and a [failure when establishing a connection](#http-client-establishing-connection-error-examples).

### HTTP request retries and redirects

Retries and redirects cause more than one physical HTTP request to be sent.
A request is resent when an HTTP client library sends more than one HTTP request to satisfy the same API call.
This may happen due to following redirects, authorization challenges, 503 Server Unavailable, network issues, or any other.
A CLIENT span SHOULD be created for each one of these physical requests.
No span is created corresponding to the "logical" (encompassing) request.

Each time an HTTP request is resent, the `http.resend_count` attribute SHOULD be added to each repeated span
and set to the ordinal number of the request resend attempt.
Each time an HTTP request is resent, the `http.resend_count` attribute MUST be added to each repeated span and set to the ordinal number of the request resend attempt.

See [examples](#http-client-retries-examples) for more details.
See the examples for more details about:
* [retrying a server error](#http-client-retries-examples),
* [redirects](#http-client-redirects-examples),
* [authorization](#http-client-authorization-retry-examples).

## HTTP server

Expand Down Expand Up @@ -428,3 +484,47 @@ GET /hello - 200 (CLIENT, trace=t2, span=s1, http.resend_count=1)
|
--- server (SERVER, trace=t2, span=s2)
```

### HTTP client establishing connection examples

Example of establishing an HTTP connection in the presence of a trace started by an inbound request:

```
request (SERVER, trace=t1, span=s1)
|
-- CONNECT www.example.com:80 (CLIENT, trace=t1, span=s2)
|
-- GET /hello - 200 (CLIENT, trace=t1, span=s3)
|
--- server (SERVER, trace=t1, span=s4)
```

Example of establishing an HTTP connection with no trace started upfront:

```
CONNECT www.example.com:80 (CLIENT, trace=t1, span=s2)

GET /hello - 200 (CLIENT, trace=t1, span=s3)
|
--- server (SERVER, trace=t1, span=s4)
mateuszrzeszutek marked this conversation as resolved.
Show resolved Hide resolved
```

### HTTP client establishing connection error examples

Example of an error occurring during establishing an HTTP connection in the presence of a trace started by an inbound request:

```
request (SERVER, trace=t1, span=s1)
|
-- CONNECT www.example.com:80 (CLIENT, trace=t1, span=s2) Couldn't connect to server
```

Example of an error occurring during establishing an HTTP connection with no trace started upfront:

```
CONNECT www.example.com:80 (CLIENT, trace=t1, span=s2)

GET /hello - 200 (CLIENT, trace=t1, span=s3)
|
--- server (SERVER, trace=t1, span=s4)
mateuszrzeszutek marked this conversation as resolved.
Show resolved Hide resolved
```