-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fixes #4368 #4421
Closed
Closed
Fixes #4368 #4421
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
6845a99
add document on
413385d
revert incorrect / accidental change
9baa07b
Merge branch 'main' into michael2893-patch
966e685
move information to existing gateway document.
4aa1670
Merge branch 'main' into michael2893-patch
b734701
Slight tweaks to formatting
michael2893 f6de9d2
Update gateway.md once more, retrigger easy CLA
michael2893 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
57 changes: 57 additions & 0 deletions
57
content/en/docs/collector/deployment/single-writer-principle.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -0,0 +1,57 @@ | ||||||
--- | ||||||
title: The Single Writer Principle | ||||||
description: How to handle the single writer principle in Collector deployments | ||||||
weight: 1 | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Alter the weight of the page so that it shows below "Gateway".
Suggested change
|
||||||
--- | ||||||
|
||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As the doc is likely not self-contained, consider adding links from other topics. |
||||||
The Single Writer Principle is a design principle that refers to maintaining a single logical writer specific data types | ||||||
of events or data. The intention is to prevent concurrent write attempts from multiple writers | ||||||
to the same data source, potentially leading to data corruption or data loss | ||||||
|
||||||
## Context in OTLP | ||||||
Collector deployments can involve multiple instance across different nodes in a distributed systems. | ||||||
It is possible in this case for these instances to receive and process data from the same resources. | ||||||
When multiple collectors write data to the same backend (such as Prometheus), the issues that arise are | ||||||
a result of violating the single writer principle. | ||||||
|
||||||
Duplicate samples for the same metric can be submitted. | ||||||
|
||||||
|
||||||
## Potential Problems caused by multiple writers | ||||||
|
||||||
An example scenario might be where multiple collectors are submitting the same samples, as | ||||||
they are receiving traces from same resources. If each collector is scraped, it's possible to | ||||||
observe inconsistent values for the same series due to data being collected by different collectors | ||||||
|
||||||
Any inconsistency in the metric could be explained by a single series referencing each individaul collector, which | ||||||
would have varying values due to inconsistent time intervals. | ||||||
|
||||||
For this reason uniqueness is a crucial concept in most metric systems. | ||||||
|
||||||
## Detection | ||||||
|
||||||
One way to identify this is by inspecting the series visually. | ||||||
A series with significant gaps or jumps may be a clue that multiple collectors are submitting the same samples. | ||||||
Finding a pattern of plateaus that seem to oscillate may indicate that the data is coming from two sources. | ||||||
|
||||||
## Prevention | ||||||
Implement a globally unique identifier for all metric data streams. | ||||||
|
||||||
A potential approach is to use a unique identifier, such as a dynamic hostname in your configuration. | ||||||
|
||||||
```yaml | ||||||
connectors: | ||||||
spanmetrics: | ||||||
dimensions: | ||||||
- name: http.method | ||||||
- name: http.status_code | ||||||
- name: k8s.namespace.name | ||||||
- name: gateway_host: "{{ .hostname }}" | ||||||
|
||||||
`` | ||||||
|
||||||
|
||||||
|
||||||
|
||||||
|
||||||
|
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How can we better contextualize what this doc is about when it comes to Collector deployments? Think of a nav title that would fit the progression in the current section. Why should someone know about this principle?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, could this be a section in an existing doc rather than its own doc?