Skip to content

Commit

Permalink
add document on
Browse files Browse the repository at this point in the history
  • Loading branch information
michael2893 committed May 3, 2024
1 parent e05f8bf commit 6845a99
Show file tree
Hide file tree
Showing 2 changed files with 58 additions and 1 deletion.
57 changes: 57 additions & 0 deletions content/en/docs/collector/deployment/single-writer-principle.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
---
title: The Single Writer Principle
description: How to handle the single writer principle in Collector deployments
weight: 1
---

The Single Writer Principle is a design principle that refers to maintaining a single logical writer specific data types
of events or data. The intention is to prevent concurrent write attempts from multiple writers
to the same data source, potentially leading to data corruption or data loss

## Context in OTLP
Collector deployments can involve multiple instance across different nodes in a distributed systems.
It is possible in this case for these instances to receive and process data from the same resources.
When multiple collectors write data to the same backend (such as Prometheus), the issues that arise are
a result of violating the single writer principle.

Duplicate samples for the same metric can be submitted.


## Potential Problems caused by multiple writers

An example scenario might be where multiple collectors are submitting the same samples, as
they are receiving traces from same resources. If each collector is scraped, it's possible to
observe inconsistent values for the same series due to data being collected by different collectors

Any inconsistency in the metric could be explained by a single series referencing each individaul collector, which
would have varying values due to inconsistent time intervals.

For this reason uniqueness is a crucial concept in most metric systems.

## Detection

One way to identify this is by inspecting the series visually.
A series with significant gaps or jumps may be a clue that multiple collectors are submitting the same samples.
Finding a pattern of plateaus that seem to oscillate may indicate that the data is coming from two sources.

## Prevention
Implement a globally unique identifier for all metric data streams.

A potential approach is to use a unique identifier, such as a dynamic hostname in your configuration.

```yaml
connectors:
spanmetrics:
dimensions:
- name: http.method
- name: http.status_code
- name: k8s.namespace.name
- name: gateway_host: "{{ .hostname }}"

``






2 changes: 1 addition & 1 deletion content/en/docs/collector/img/otel-agent-sdk.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 6845a99

Please sign in to comment.