filelog: Multiline merging mixes up logs from different files

[rockb1017]

this works fine.

operators:
    - default: clean-up-log-record
      routes:
      - expr: ($$resource["k8s.namespace.name"]) matches ".*" && ($$resource["k8s.container.name"])
          == "loggen"
        output: .*_loggen
      - expr: ($$resource["k8s.container.name"]) == "loggen2"
        output: loggen2
      type: router
    - combine_field: log
      id: .*_loggen
      is_first_entry: '($.log) matches "num: \\d+0\\s"'
      output: clean-up-log-record
      type: recombine
    - combine_field: log
      id: loggen2
      is_first_entry: '($$.log) matches "num: \\d+0\\s"'
      output: clean-up-log-record
      type: recombine
    - id: clean-up-log-record
      ops:
      - move:
          from: log
          to: $
      type: restructure

because loggen container and loggen2 container is separately routed to its own recombine operator.
but with configurations like this one

operators:
    - default: clean-up-log-record
      routes:
      - expr: ($$resource["k8s.namespace.name"]) matches ".*" && ($$resource["k8s.container.name"])
          == "loggen.*"
        output: .*_loggen.*
      type: router
    - combine_field: log
      id: .*_loggen.*
      is_first_entry: '($.log) matches "num: \\d+0\\s"'
      output: clean-up-log-record
      type: recombine
    - id: clean-up-log-record
      ops:
      - move:
          from: log
          to: $
      type: restructure

they get mixed up

[jsirianni]

Do both containers log to the same file? Can you provide input and output samples?

[rockb1017]

sorry for late reply. it is logging to a separate files. I can't reproduce it so i think it was a misconfiguration. Thank you.

[rockb1017]

reopening this. similar issue is reported. signalfx/splunk-otel-collector-chart#309

[rockb1017]

I also see it now.

num: 100061 | loggen-tjh79 | 2021-11-19T21:23:21.643933 | rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr this is from log generator. i have it merging at every 10 events.
but 2021/11/19 21:23:22 EPS: 0 this like is from another pod

[rockb1017]

updating issue name and description with more information i found.

[rockb1017]

I think in the recombine operator, everything is store in batch. it is just an array.

I think we need to make it into a map. make(map[string][]*entry.Entry)
Add separatorField parameter that this recombine operator will be applied separated. (good default would be filename)
and get string of separatorField and use it as s key field for batch Map.

[rockb1017]

@djaglowski Any suggestions?

[djaglowski]

To summarize my understanding of the problem and proposed solution:

The problem is that the recombine operator only works for "partial" entries coming from a single log source (e.g. a single file).

The general solution is to make the operator "source-aware", so that it can keep partial entries separate from each other based on the log source. The general idea is then is to have what I would tentatively describe as a "log source identifier". This is used to properly group partial entries together. I think this all makes sense.

A couple things we need to consider:

1. A filename seems like a reasonable "log source identifier" for any log files, but I don't think we can assume these partial entries are coming from a file. To make this work, I think we need to allow the user to specify which value using field syntax. Probably the default can be $attributes.file.name though.
2. We need a strategy for ensuring the map does not grow indefinitely.

[rockb1017]

Thanks for putting your thoughts into it. I can take a stab at it if no one yet started.