[exporter/kafka] Send individual LogRecords

[nanorobocop]

Is your feature request related to a problem? Please describe.
Current implementation of of Kafka exporter sends complete ResourceLogs, which consists of ScopeLogs, which consists of LogRecords.
So instead of sending individual LogRecords, it sents structure of 3 nested levels.
It requires storage behind Kafka to store such complex structure.

Describe the solution you'd like
This proposal is to send individual LogRecords.
One LogRecord per sarama.ProducerMessage.

Describe alternatives you've considered

Additional context

[github-actions]

Pinging code owners: @pavolloffay @MovieStoreGuy

[nanorobocop]

As far as I can see, other exporters also extract individual LogRecords and send it: clickhouseexporter, elasticsearchexporter, lokiexporter.
So this seems more common.

[bogdandrutu]

An individual "log record" is incorrect to be sent, since lots of informations about that record are in the Resource and Scope.

/cc @tigrannajaryan

[tigrannajaryan]

One way sending individual log records would be acceptable is if somehow the Resource and Scope information was merged into the LogRecord. Essentially have a flat record which contains all the information. That however would be a totally different encoding, different from the otlp_proto and otlp_json supported encodings.

other exporters also extract individual LogRecords and send it
The elastic exporter for example does exactly what I described above: it flattens the Resource attributes:

opentelemetry-collector-contrib/exporter/elasticsearchexporter/model.go

Line 41 in c364eb9

func (m *encodeModel) encodeLog(resource pcommon.Resource, record plog.LogRecord) ([]byte, error) {

This is a choice vendor specific exporters are free to make. Kafka exporter can't make such a choice for OTLP encodings since that would mean loosing data.

One possible option would be to add support for other encodings to kafka exporter. However, the encoding needs to be well defined and we need to have strong arguments in favour of adding such encoding to Kafka exporter.

Another alternate option is to use batch processor with send_batch_max_size = 1 to ensure every emitted Kafka record contains only one LogRecord.

[nanorobocop]

An individual "log record" is incorrect to be sent, since lots of informations about that record are in the Resource and Scope.

Resource and Scope attributes could be merged into LogRecord attributes (in attribute processor, for example).
Individual LogRecords are usually what is expected by backend storage.

[nanorobocop]

One way sending individual log records would be acceptable is if somehow the Resource and Scope information was merged into the LogRecord. Essentially have a flat record which contains all the information. That however would be a totally different encoding, different from the otlp_proto and otlp_json supported encodings.

Right, that is what I'm proposing - to have different encodings: logRecordJSON encoding in PR #12657 and logRecordAttributesJSON encoding in PR #12658

[nanorobocop]

Another alternate option is to use batch processor with send_batch_max_size = 1 to ensure every emitted Kafka record contains only one LogRecord.

That could be an option, but I think problem is nested structure with arrays []ResourceLog:[]ScopeLog:[]LogRecords, which is unnecessary in this case.
Backend would need to go 3 levels deep to get actual LogRecord.

As noticed above, many exporters implement sending individual LogRecords and have such 3 'for' loops pattern.
This seems common and deserves it's own flattened encoding maybe? Even some common encoding, not only for Kafka exporter.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.