[receiver/filelog] Not respecting order of log messages per file

[SamerJ]

Component(s)

receiver/filelog

What happened?

Description

We have a log file with 100s of lines. OTEL is configured with a receiver/filelog that will read files from the beginning (start_at: 'beginning'). When OTEL is started, it will read the already existing lines and output the data. The problem we are having is that the order of the log messages isn't preserved per file.
At this point I am not sure what is guaranteed or what configuration can be utilized to preserve the order of logs assuming a sunny scenario.

Steps to Reproduce

Create a log file with 100s of lines. Run OTEL with the configuration below. Check the exported data. Sometimes the order of the log lines for the file are mixed up. It is flaky, so you might need to run it multiple times.

Expected Result

Assuming no issues (no failures or retries), the log messages for a given file should be exported in the right order
i.e line 10 of file A should appear before line 11 of file A.

Actual Result

For a given file, the order of the log messages isn't guaranteed.

Collector version

0.75.0

Environment information

Environment (Shouldn't matter)

OS: Ubuntu 22.04.2 LTS

OpenTelemetry Collector configuration

receivers:
  filelog:
    include: [ logs/**/*.* ]
    start_at: 'beginning'
    # max_concurrent_files: 2
    # max_batches: 1
    include_file_name: false
    include_file_path: true
exporters:
  file/logs:
    path: "./out.txt"
service:
  extensions: []
  pipelines:
    logs:
      receivers: [filelog]
      processors: []
      exporters: [file/logs]

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/filelog: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

Arguably the receiver does not need to preserve order because log timestamps should be sufficient to establish ordering. However, I think it's an intuitive and reasonable expectation that it should be able to do this.

I believe this may be due to the fact that we are using a worker pool to convert logs into the collector's internal pdata format. This functionality has been called into question for other reasons so I'll add this to the list of reasons why we should consider removing it.

This needs to be looked at closer but I'm hoping to find time for this in the near future.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/filelog: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[SamerJ]

Hello @djaglowski,

Any chance we can know if the something is planned that will indirectly fix this "issue"? Whether we will remove batching from the stanza adapter or allow configuring it.
If indeed any of the above is planned, any chance we can have visibility on when we can expect it?

Thanks in Advance

[djaglowski]

Hi @SamerJ, I haven't identified a way to fix this issue while maintaining performance. I may have more time to look into this in the near future but there is no concrete plan to make changes relating to this concern.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/filelog: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.

[Amoolaa]

Does there exist a workaround for this? Has this been resolved?

[djaglowski]

Not yet, but hopefully with changes proposed in #35453, we will naturally preserve ordering.