[exporter/elasticsearch] route based on data stream attributes #33755
Conversation
| logs := newLogsWithAttributeAndResourceMap( | ||
| map[string]string{ | ||
| dataStreamDataset: "record.dataset", | ||
| }, | ||
| map[string]string{ | ||
| dataStreamDataset: "resource.dataset", | ||
| dataStreamNamespace: "resource.namespace", | ||
| }, | ||
| ) |
There was a problem hiding this comment.
Test is only covering record and resource, not scope
| Data point attributes take precedence over scope attributes, which take precedence over resource attributes. | ||
| - `prefix_suffix` - uses resource, scope or data point attributes `elasticsearch.index.prefix` and `elasticsearch.index.suffix` | ||
| to dynamically construct index name in the form `${elasticsearch.index.prefix}${metrics_index}${elasticsearch.index.suffix}`. | ||
| Data point attributes take precedence over scope attributes, which take precedence over resource attributes. |
There was a problem hiding this comment.
metrics prefix_suffix precedence is different to logs and traces. Is this deliberate?
There was a problem hiding this comment.
Yes, I created #33725 because I think the current setting for prefix_suffix should be changed.
When introducing metrics, my thinking was to introduce the prefix_suffix mode for metrics using the correct precedence, to prevent a breaking change at least in metrics behavior.
I do see value in having those consistent between signals though. If you think we should rather keep prefix_suffix mode precedence consistent with other signals, I can change that.
@felixbarny I have added filling in missing data stream attributes in cb88ebc, can you take a look? |
| @@ -213,6 +213,43 @@ func TestExporterLogs(t *testing.T) { | |||
| rec.WaitItems(1) | |||
| }) | |||
|
|
|||
| t.Run("publish with dynamic index data_stream mode", func(t *testing.T) { | |||
There was a problem hiding this comment.
I think there are some parts missing from the tests
- scope attributes (mentioned in another comment)
- fields in doc
- default is used when no data_stream attributes apply
- data_stream attributes are present
- doc action
- default is used when no data_stream attributes apply
| dataType := ensureAttribute(dataStreamType, defaultDataStreamTypeLogs, record.Attributes(), scope.Attributes(), resource.Attributes()) | ||
| return fmt.Sprintf("%s-%s-%s", dataType, dataSet, namespace) |
There was a problem hiding this comment.
Not blocking this PR but the type should be fixed based on the record type. Similar story for metrics and spans.
| dataType := ensureAttribute(dataStreamType, defaultDataStreamTypeLogs, record.Attributes(), scope.Attributes(), resource.Attributes()) | |
| return fmt.Sprintf("%s-%s-%s", dataType, dataSet, namespace) | |
| recordAttributes.PutStr(dataStreamType, "logs") | |
| return fmt.Sprintf("logs-%s-%s", dataSet, namespace) |
|
Superseded by #33794 |
Description:
This adds a feature to the Elasticsearch exporter to route telemetry based on data stream attributes, as described in An introduction to the Elastic data stream naming scheme.
Link to tracking Issue:
Testing:
Added unit tests.
Documentation:
Added documentation in exporter's README.