Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[chore][pkg/stanza/fileconsumer] emit logs in batches #3

Closed
wants to merge 5 commits into from
Closed

[chore][pkg/stanza/fileconsumer] emit logs in batches #3

wants to merge 5 commits into from

Conversation

andrzej-stencel
Copy link
Owner

Description

Modifies the File consumer to emit logs in batches as opposed to sending each log individually through the Stanza pipeline and on to the Log Emitter.

This was achieved via the following incremental changes, with the goal of making code reviews easier (multiple smaller changesets):

  • a584e0a collect scanned tokens into batches in Reader::ReadToEnd method in File consumer, but still call emit function for each token individually
  • 6391f55 change emit.Callback function signature to accept a slice of tokens and emit tokens in batches from the Reader. At this point, the batches are still split into individual tokens inside the emit function, because the Stanza operators can only process one entry at a time.
  • 187b345 add ProcessBatch method to Stanza operators and use it in the emit function. At this point, the batch of tokens is translated to a batch of entries and passed to Log Emitter as a whole. The batch is still split in the Log Emitter, which calls consumeFunc for each entry in a loop.
  • 8e3197b do not split batches in Log Emitter, call consumeFunc on the whole batch of entries

Note that this is currently a draft, requesting initial feedback. I haven't yet implemented the ProcessBatch method for all Stanza operators, as I'd like to first get feedback its definition. Specifically, should the function accept a []entry.Entry or []*entry.Entry?

Link to tracking issue

Testing

No changes in tests. The goal is for the functionality to not change and for performance to not decrease.

Documentation

These are internal changes, no user documentation needs changing.

@andrzej-stencel
refactor: introduce batching in Reader, but call emit function for ea…
a584e0a 
…ch entry still

Next step is to change the emit function to accept a batch.
@andrzej-stencel
change emit.Callback signature to accept a slice of tokens
6391f55 
But still send each entry one by one to the next consumer in the file input.
Next step is to change Stanza operators to accept batches.
@andrzej-stencel
update changelog entry
8780b9a
@andrzej-stencel
add WriterOperator::WriteBatch and Operator::ProcessBatch methods
187b345 
The File input's `emitBatch` function now calls `ProcessBatch` instead of `Process`.

The added `ProcessBatch` method will make each Stanza operator
capable of accepting a batch of entries.
@andrzej-stencel
do not split batches in Log Emitter
8e3197b 
This changes the Log Emitter to run the `consumerFunc`
on the whole batch, instead of splitting the batch into individual entries
and calling `consumerFunc` on each of them.

This doesn't change much while the Log Emitter has its own `batch` buffer,
but if we remove the `batch` buffer (see open-telemetry#35456),
this should prevent the performance drop described in open-telemetry#35454.
@andrzej-stencel andrzej-stencel changed the title Emit multiple [chore][pkg/stanza/fileconsumer] emit logs in batches Nov 8, 2024
andrzej-stencel pushed a commit that referenced this pull request Jan 7, 2025
@shivanthzen @mx-psi
Add histogram to telemetrygen (open-telemetry#36322)
9cb09d5 
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
Generates simple histograms using telemetrygen
<!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes

<!--Describe what testing was performed and which tests were added.-->
#### Testing
Test with a local otel collector with debug output

```
bin/telemetrygen metrics --metrics 5  --otlp-http --otlp-endpoint "localhost:4318"    --metric-type Histogram --otlp-insecure 
```
Output from debug Exporter: 
```
Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0
ScopeMetrics #0
ScopeMetrics SchemaURL:
InstrumentationScope
Metric #0
Descriptor:
     -> Name: gen
     -> Description:
     -> Unit:
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
StartTimestamp: 2024-11-13 16:22:50.633365 +0000 UTC
Timestamp: 2024-11-13 16:22:51.633367 +0000 UTC
Count: 0
Sum: 0.000000
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 1.000000
ExplicitBounds #2: 2.000000
ExplicitBounds #3: 3.000000
ExplicitBounds open-telemetry#4: 4.000000
Buckets #0, Count: 0
Buckets #1, Count: 0
Buckets #2, Count: 0
Buckets #3, Count: 0
Buckets open-telemetry#4, Count: 0
ResourceMetrics #1
Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0
ScopeMetrics #0
ScopeMetrics SchemaURL:
InstrumentationScope
Metric #0
Descriptor:
     -> Name: gen
     -> Description:
     -> Unit:
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
StartTimestamp: 2024-11-13 16:22:50.639942 +0000 UTC
Timestamp: 2024-11-13 16:22:51.639942 +0000 UTC
Count: 1
Sum: 1.000000
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 1.000000
ExplicitBounds #2: 2.000000
ExplicitBounds #3: 3.000000
ExplicitBounds open-telemetry#4: 4.000000
Buckets #0, Count: 0
Buckets #1, Count: 1
Buckets #2, Count: 0
Buckets #3, Count: 0
Buckets open-telemetry#4, Count: 0
ResourceMetrics #2
Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0
ScopeMetrics #0
ScopeMetrics SchemaURL:
InstrumentationScope
Metric #0
Descriptor:
     -> Name: gen
     -> Description:
     -> Unit:
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
StartTimestamp: 2024-11-13 16:22:50.6404 +0000 UTC
Timestamp: 2024-11-13 16:22:51.640401 +0000 UTC
Count: 2
Sum: 4.000000
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 1.000000
ExplicitBounds #2: 2.000000
ExplicitBounds #3: 3.000000
ExplicitBounds open-telemetry#4: 4.000000
Buckets #0, Count: 0
Buckets #1, Count: 1
Buckets #2, Count: 0
Buckets #3, Count: 1
Buckets open-telemetry#4, Count: 0
ResourceMetrics #3
Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0
ScopeMetrics #0
ScopeMetrics SchemaURL:
InstrumentationScope
Metric #0
Descriptor:
     -> Name: gen
     -> Description:
     -> Unit:
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
StartTimestamp: 2024-11-13 16:22:50.640729 +0000 UTC
Timestamp: 2024-11-13 16:22:51.640729 +0000 UTC
Count: 3
Sum: 3.000000
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 1.000000
ExplicitBounds #2: 2.000000
ExplicitBounds #3: 3.000000
ExplicitBounds open-telemetry#4: 4.000000
Buckets #0, Count: 1
Buckets #1, Count: 1
Buckets #2, Count: 1
Buckets #3, Count: 0
Buckets open-telemetry#4, Count: 0
ResourceMetrics open-telemetry#4
Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0
ScopeMetrics #0
ScopeMetrics SchemaURL:
InstrumentationScope
Metric #0
Descriptor:
     -> Name: gen
     -> Description:
     -> Unit:
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
StartTimestamp: 2024-11-13 16:22:50.641073 +0000 UTC
Timestamp: 2024-11-13 16:22:51.641073 +0000 UTC
Count: 4
Sum: 12.000000
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 1.000000
ExplicitBounds #2: 2.000000
ExplicitBounds #3: 3.000000
ExplicitBounds open-telemetry#4: 4.000000
Buckets #0, Count: 0
Buckets #1, Count: 0
Buckets #2, Count: 1
Buckets #3, Count: 2
Buckets open-telemetry#4, Count: 1
        {"kind": "exporter", "data_type": "metrics", "name": "debug"}
```
<!--Describe the documentation added.-->
#### Documentation

<!--Please delete paragraphs that you did not use before submitting.-->

---------

Co-authored-by: Pablo Baeyens <pbaeyens31+github@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[pkg/stanza/fileconsumer] Emit logs in batches
1 participant
@andrzej-stencel