Start zPages Experimental Spec #767
Conversation
experimental/zpages.md
Outdated
| ------------- | ||
| ## Future Possibilities | ||
| ### Out-process | ||
| zPages that compatible across different languages, with processing of information happening outside of applications |
There was a problem hiding this comment.
Is this related to the zpage extension on the collector? I guess the otlp receiver should already have zpages similar to what the SDK provides?
There was a problem hiding this comment.
I didn't realize the collector had zPages, so I'm not sure how/if the collector zPages differs from the individual OTel language implementations of zPages. I will update you and change things once I look into it more, thanks for the information!
There was a problem hiding this comment.
Re-posting my previous comments from the email thread, I don't see them particularly well reflected here:
Many frameworks already provide similar functionality, for example Uber has an internal plugin that renders pages like below (only showing the sidebar menu). I think the following are important for debug pages design:
- pages framework should not be coupled with OTel SDK, it can be a standalone library, so that SDK can supply providers into it, but other tools can also supply providers
- OTel providers should be implemented as separate components for data vs. rendering, so that, for example, data providers could be also integrated into other rendering frameworks (like Uber's debug pages below).
- open question: should rendering be done in-app or should the app only provide data and rendering done via a sidecar? Uber's framework, as well as many OSS projects (e.g. Flink Job Master, Spark jobs, etc), opt for including rendering in-app, simplifying the use of debug pages.
experimental/zpages.md
Outdated
| @@ -0,0 +1,112 @@ | |||
| # zPages | |||
There was a problem hiding this comment.
I would suggest dropping the "z" theme, which afaik is an internal Google speak. We can call them "debug pages", similar to /debug/vars used in Go, which is way more descriptive than "z-pages".
There was a problem hiding this comment.
I like z, because it doesn't have any connotation like debug. Debug is sometimes associated with something slow and not for production.
Spring calls it "Production Ready Endpoints": https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-features.html#production-ready-endpoints Perhaps we can call it the same.
As for endpoint names, dropping z is OK with me.
There was a problem hiding this comment.
@jajanet can you call the page "Production Ready Endpoints"? @yurishkuro @anuraaga @arminru ware you advocating for simply dropping z from the names of endpoints?
There was a problem hiding this comment.
I don't consider "debug" to mean "slow". Debug means investigating performance of a process by looking deeper into the signals that are not available through normal monitoring. Changing logging level is also an example of debugging.
"Production ready" sounds much more narrow, like a healthcheck endpoint.
Another possible name is "introspection endpoints", however that is also more narrow than what the framework should provide.
There was a problem hiding this comment.
@SergeyKanzhelev I would drop the z from both the endpoints and names, I agree with @yurishkuro we can have a semantic name that is self-explanatory, and URLs with z's are not as clean as with normal English IMO (z suffixes are from an era when these pages were exposed to the public internet).
debug seems fine to me, I might name a similar concept just "internal pages", or the other name suggestions seem fine to me since they're still self explanatory.
There was a problem hiding this comment.
Would everybody be OK with "Introspection endpoints"? We might need more opinions, but I feel very strongly that "debug" will be considered as something not acceptable for production by customers.
There was a problem hiding this comment.
From the conversation on naming conventions, my understanding is:
- rename all references of zPages to Introspection Endpoints
- drop the z from TraceZ, RPCz, etc
Is this correct? Once this is clarified and decided, I'll update the file
There was a problem hiding this comment.
@jajanet Sounds good to me!
We could also add an Internal or OpenTelemetry SDK there to further disambiguate it from telemetry about user/app code.
There was a problem hiding this comment.
I am late to the party, but perhaps "Diagnostic Pages" or "Diagnostic Endpoints"? I feel it conveys both the debugging and introspection meanings.
OpenCensus also uses the term for describing what it really is:
They are useful to for in-process diagnostics
experimental/zpages.md
Outdated
| While zPages are uniquely useful in being more lightweight and quicker compared to installing external exporters like Jaeger and Zipkin, they still offer many useful ways to debug and gain insight into applications. The uses depend on the type of zPage, which is detailed below. | ||
|
|
||
| ## Types of zPages | ||
| ### TraceZ |
There was a problem hiding this comment.
There is no need for Z suffixes
experimental/zpages.md
Outdated
|
|
||
| You can read about TraceConfigZ more [here](https://opencensus.io/zpages/java/#traceconfigz). | ||
|
|
||
| ### RPCz |
There was a problem hiding this comment.
This seems like a projection of the overall metrics view, does it need to be pulled out? If it is pulled out to the top level, what conventions is it going to use for detecting relevant metrics?
There was a problem hiding this comment.
From my understanding, metrics and collecting RPC call information is separate? I'm not exactly sure, I think @SergeyKanzhelev could provide better insight
experimental/zpages.md
Outdated
|
|
||
| You can read about RPCz more [here](https://docs.google.com/document/d/1RWNyUIaKTYK12tck_rQjki4jTyHFfkD8sk54mjwRwso/edit#) and [here](https://opencensus.io/zpages/java/#rpcz). | ||
|
|
||
| ### StatsZ |
There was a problem hiding this comment.
We're not using "stats" in OTel, we use "metrics".
There was a problem hiding this comment.
That makes sense! So everywhere with stats, I should replace with metrics? Also thanks for the detailed information and explanations!
experimental/zpages.md
Outdated
|
|
||
| The idea of "zPages" originates from one of OpenTelemetry's predecessors, [OpenCensus](https://opencensus.io/). You can read more about zPages from the OpenCensus docs [here](https://opencensus.io/zpages) or the OTEP [here](https://github.com/open-telemetry/oteps/blob/master/text/0110-z-pages.md). OpenCensus has different zPage implementations in [Java](https://opencensus.io/zpages/java/), [Go](https://opencensus.io/zpages/go/), and [Node](https://opencensus.io/zpages/node/) and there has been similar internal solutions developed at companies like Uber. Within OpenTelemetry, zPages are also either developed or being developed in [C#](https://github.com/open-telemetry/opentelemetry-dotnet/tree/master/src/OpenTelemetry.Exporter.ZPages), Java, and C++. | ||
|
|
||
| While zPages are uniquely useful in being more lightweight and quicker compared to installing external exporters like Jaeger and Zipkin, they still offer many useful ways to debug and gain insight into applications. The uses depend on the type of zPage, which is detailed below. |
There was a problem hiding this comment.
Please mention that "For high throughput applications, external exporters are typically configured to send subset of telemetry for reach analysis to save costs, while zPages, being in-memory, can analyze more telemetry with the limited set of supported scenarios."
There was a problem hiding this comment.
Added to this line (26) as:
For high throughput applications, zPages can also analyze more telemetry with the limited set of supported scenarios than external exporters; this is because zPages are in-memory while external exporters are typically configured to send subset of telemetry for reach analysis to save costs.
Let me know if I should change it!
experimental/zpages.md
Outdated
|
|
||
| ### TraceConfigZ | ||
|
|
||
| TraceConfigZ is closely related to TraceZ, allowing the user to modify how spans are sampled or how much data to keep in TraceZ by updating the TraceZ components accordingly. |
There was a problem hiding this comment.
Are TraceConfig pages control sampling for spans that would be exported by regular exporter? I don't think TraceConfigZ is related to TraceZ.
There was a problem hiding this comment.
I'm not sure; my understanding was that TraceConfigz is for TraceZ for OpenCensus, but I will look into it more and edit things accordingly. If it makes more sense, the decision here also doesn't need involve OpenCensus' zPages
There was a problem hiding this comment.
I checked with someone who implemented TraceConfigz and they said TraceConfigz would effect all sampling for zPages and external backends. I updated the spec accordingly, and I'm wondering if that's that what we want in the spec?
experimental/zpages.md
Outdated
|
|
||
| For OpenTelemetry, a custom `span processor` can be made to interface with the `Tracer` API to collect spans. This span processor collects references to running spans and exports completed spans to its own memory or directly to an aggregator. An alternative to a span processor is using some sort of profiler. | ||
|
|
||
| A `data aggregator` keeps a track of counts for running, error, and latency buckets for spans grouped by their name. It also samples some spans to provide users with more information. To prevent memory overload, only some spans are sampled for each bucket for each span name; for example, if that sampled span max number is set to 5, then only up to 55 pieces of span data can be kept for each span name in the aggregator (sampled_max * number of buckets = 5 * [running + error + 9 latency buckets] = 5 * 11 = 55). |
There was a problem hiding this comment.
tracez is also designed to keep samples for every bucket of latency. This is a big benefit comparing to regular exporter. zPages will store unique spans that falls into specific latency bucket or having an error - those, that would likely be sampled out for the regular exporter
There was a problem hiding this comment.
Reclarified under line 30 (explanations/overviews on the types of zPages) as:
In addition to these counts, TraceZ also keeps a set number of samples for error, running, and latency (including within each duration bucket) spans for each span name to allow users to look closer at span fields. This is particularly useful compared to external exporters that would otherwise likely sample them out.
Does that sound okay? I meant for this later section for outline more architecture details, but could restructure things as needed
There was a problem hiding this comment.
Naming is one change that we need to close on. If no comments from Yuri and others, we can keep the current naming.
I think it's critical to explain that tracez works BEFORE sampling and will catch unique samples of Spans that otherwise wouldn't be collected with the regular exporter. See my comments
Co-authored-by: Sergey Kanzhelev <S.Kanzhelev@live.com>
carlosalberto
added
area:sdk
There was a problem hiding this comment.
I did not see any mention of security. zPages available via an embedded HTTP server increase the attack surface of the application. It can potentially allow a malicious actor to:
- Read sensitive data contained in telemetry.
- Perform a DOS attack on the HTTP server.
- Initiate a telemetry storm by reconfiguring the telemetry collection (e.g. by changing the sampling factor cause significant increase of exporter telemetry).
I would like to understand what is our stance on this. Should zPages be enabled by default or we disable them by default and they need to be explicitly enabled by the end user
experimental/zpages.md
Outdated
| @@ -0,0 +1,112 @@ | |||
| # zPages | |||
There was a problem hiding this comment.
I am late to the party, but perhaps "Diagnostic Pages" or "Diagnostic Endpoints"? I feel it conveys both the debugging and introspection meanings.
OpenCensus also uses the term for describing what it really is:
They are useful to for in-process diagnostics
experimental/zpages.md
Outdated
|
|
||
| A `data aggregator` keeps track of counts for running, error, and latency buckets for spans grouped by their name. It also samples some spans to provide users with more information. To prevent memory overload, only some spans are sampled for each bucket for each span name; for example, if that sampled span max number is set to 5, then only up to 55 pieces of span data can be kept for each span name in the aggregator (sampled_max * number of buckets = 5 * [running + error + 9 latency buckets] = 5 * 11 = 55). | ||
|
|
||
| When the user visits the TraceZ endpoint, likely something similar to `host:port/tracez`, then the distribution of latencies for span names will be rendered. When clicking on buckets counts for a span name, additional details on individual sampled spans for that bucket would be shown. These details would include trace ID, parent ID, span ID, start time, attributes, and more depending on the type of bucket (running, error, or latency) and what's implemented/recorded in the other components. See [HTTP Server](#http-server) for more information on implementation. |
There was a problem hiding this comment.
Do we standardize the port number? It will make the discovery easier.
There was a problem hiding this comment.
not at the moment. I think it's not critical for the experimental stage unless we will get some usability results. Perhaps a separate research will be needed what port other things like springboot endpoints standardize on...
experimental/zpages.md
Outdated
|
|
||
| Instead of directly translating native data structures to HTML strings based on the stored information, the data layer would do 2 things depending on the webpage endpoint accessed: 1. Serve the static HTML, JS, and CSS files, which are consistent, not server generated, and not data dependent and 2. Act like a web/HTTP API by translating stored data to JSON strings. Whether the data layer does one or the other depends on which URL endpoint is accessed; the former is intended for the initial zPages load, and latter for user interactions. | ||
|
|
||
| The UI/frontend/rendering layer is the HTML, CSS, and Javascript itself, in contrast to the logic to serve those files. This frontend uses the data layer's API on the client-side within the browser with Javascript by accessing certain endpoints depending on the user's actions. The data returned interacts with the Javascript, which determines and executes the logic necessary to render updates to the HTML DOM. Modifying the HTML DOM means there are no unnecessary requesting and re-rendering static files, and only parts of the webpage are changed. This makes subsequent data queries quicker and requires no knowledge of client-side rendering for the zPages developer. |
There was a problem hiding this comment.
Does this imply that rendered zPages will be ONLY available if the client is JS-enabled? In troubleshooting scenarios I may have no access to a full-blown browser and just fire a wget against the zPage.
I agree that the separation to the data collection and rendering components is useful. I think it is also very useful to expose the zPage data purely as data, e.g. in JSON format, available via a REST API. However, I think there is also a value in simple human-readable HTML/CSS output that is rendered on server side and is available without requiring JS-enabled web clients.
I think a good approach would be to render on the server-side but for each endpoint also support JSON-encoded responses if the client requests it (via a request parameter or via "Accept" HTTP header).
There was a problem hiding this comment.
this is how OTEP is formulated and experimental code is implemented. Perhaps those endpoints are important to document. @jajanet can you please add a TODO in the document saying that those data endpoints needs to be documented as well?
There was a problem hiding this comment.
Yes, that's a good point and I'll do that! I'll also add points on security, as I didn't think about that before
|
Hey everybody! Based on the discussion, let's have a poll on the new zPages name to help finalize this PR. Probably gonna have this up for at least a couple of days, and/or when there's a clear winner. In any case, the name discussion can be an open one and may be postponed to a later PR depending on the results. Vote using emojis! Prefix: Suffix: @SergeyKanzhelev @anuraaga @yurishkuro @tigrannajaryan @arminru |
|
this is experimental feature with the lower bar for approvals. Merging for quicker iterations... |
|
Naming thing should be addressed separately. |
|
@jajanet @SergeyKanzhelev This PR broke the build, can you fix lint errors: |
* Fix links that weren't working * Start zPages spec with basic details * Formatting, todo details * Fix link * Fix design link * Fix some typos, add otep link * OT->OTel, other detail cleanup * Define zpages deadlock more accuractely Co-authored-by: Sergey Kanzhelev <S.Kanzhelev@live.com> * Add suggestions and clarifications * Fix typos and grammar errors * Update tracez sentence flow * Fix grammar issues, mention OTel collector zpages * Add security concerns, data url endpoints TODO * Fix typo, uppercasing * Reclarify TraceConfigz * Nest zPages.md in trace folder, clarify ui/data separation * More cleanup and typo fixes Co-authored-by: Sergey Kanzhelev <S.Kanzhelev@live.com>
This PR begins providing more insight into what zPages are, their background, different types utilized in the past (including uses and other details), and other avenues to explore for zPages.
Implementation insights are also provided for TraceZ based off the findings from the OpenTelemetry C++ TraceZ project. Some links are provided, and more will be added in the future along with recommendations on interfacing more specifically using the OpenTelemetry API/SDK.
Related OTEP
@SergeyKanzhelev