Metrics prototype scenario

[reyang]

Follow up on the 02/11/2021 #108 Metrics API/SDK SIG Mtg, I've created this OTEP which has described two scenarios for the metrics prototyping work.

The actual prototype will be submitted as PR(s) to the language client repo, for example:

• Create Grocery Example for Metrics API opentelemetry-dotnet#1831
• Metrics examples for API SIG discusssion opentelemetry-java#2818

[cijothomas]

Not sure if this is something we want in initial stage, but i'd like to add a requirement of same metric being reported with different interval, potentially with diff. dimensions.
For example, the app owner wants to see HTTP Server Duration metric exported every 1 second with only HttpStatusCode dimension, and HTTP Server Duration metric exported every 30 seconds with dimensions {hostname, HTTP Method, Host, Status Code, Client Type}. The former is typically used for near-real-time dashboards, and the latter for more permanent storage.

[reyang]

In your example - I guess normally people would only report the 1 second one from the SDK pre-aggregation, and rely on the metrics backend to aggregate the 30 seconds one (and daily/weekly/monthly summary if there is a need).

[noahfalk]

Do we think this type of offline historical reporting is a good primary use case for the metrics API? Although I can envision a metrics API doing it I'd guess it is a better fit for a standard transaction database where there are stronger guarantees about data consistency and richer data, but potentially worse performance/availability. I think of metrics being focused on high availability and low latency which is more oriented towards diagnostics/live monitoring/alerting where the grocery would be looking for signs like:

1. Is there an unexpected change in rate of sales suggesting an unknown incident may be occuring at the store?
2. Is inventory getting unexpectedly low so we need to dispatch an urgent delivery from the warehouse?
3. Is there a sudden spike in demand for a product so we need to consider rationing or price changes?

Of course if I am looking at it with too narrow a lense then this example might be accomplishing exactly what it intends, expanding my understanding of what scenarios a metrics API is intended to support.

[jmacd]

I see (1) and (2) as good use-cases for monitoring using a metrics API, but maybe not (3). Although the example feels like it fell out of a textbook, you could re-imagine the store as a Message-Queue consumer processing orders in a horizontally scalable store. Can we ask another form of query: "how many stores were in operation at a given time?"

[jmacd]

I like how this example asks for three counters, because it seems possible to achieve with two instruments: a count of received requests and a histogram of response durations (i.e., seems to call for either a view or a 3rd instrument).

[reyang]

And it might affect the semantic convention open-telemetry/opentelemetry-specification#1378 (comment).

[jmacd]

(Reply: open-telemetry/opentelemetry-specification#1378 (comment))

[jmacd]

I see (1) and (2) as good use-cases for monitoring using a metrics API, but maybe not (3). Although the example feels like it fell out of a textbook, you could re-imagine the store as a Message-Queue consumer processing orders in a horizontally scalable store. Can we ask another form of query: "how many stores were in operation at a given time?"

[victlu]

AFAIK, we need to have multiple "pipelines" for different configurations. This may include...

• Reporting period which is related to Collection rate, Export rate, etc...
• Selection / grouping of desired metrics for this pipeline

[jmacd]

These are all SDK configuration topics, to me. The use-cases in this document are about how code is instrumented, I think, and what the API looks like.

OpenCensus had a programmatic API for configuring the kinds of variables you mentioned. I'm not sure if a programmatic API or a configuration file is what user's want, but I'd argue to keep this setup out of the instrumentation API.

[jsuereth]

My main comment, thinking deeper on this, is whether or not "external annotation" needs to be called out specifically as a non-goal.

Specifically, should we call out a use case where an DevOps wants to add metric labels POST DEVELOPMENT to the API. I think this is an SDK concern and should be called out there (e.g. how you can add annotations to Resource via ENV, which would then influence metrics), but I think the scenarios listed here are sufficient for API discussions.

[noahfalk]

Looks great, thanks @reyang!

[jmacd]

I reviewed the open comments and believe that @reyang has addressed them all. Thank you!