Logs derived metrics processor

[yeya24]

Is your feature request related to a problem? Please describe.

I'd like to have the ability to derive metrics from incoming logs so that the calculated metrics can be exported using metrics exporters.

This ability is very similar to spanmetricsprocessor and Promtail metrics stage https://grafana.com/docs/loki/latest/clients/promtail/stages/metrics/.

Describe the solution you'd like

Have a new processor to create metrics from logs based on the configuration.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Why not use Promtail directly?

Promtail works well for this usecase but it is bound to Loki and it is AGPL.

[TylerHelmuth]

@yeya24 can you open a new issue that uses the New Component template for this ask?

[yeya24]

@TylerHelmuth Sure, I'd love to. This issue is mainly to gather some feedbacks first as I am not sure if someone from the Otel team would love to sponsor it.

[evan-bradley]

@yeya24 If you open a new issue with the New Component template, it would make it easier for potential sponsors to review your proposal and determine who might be able to sponsor it.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[jpkrohling]

Was a new issue created for this? I have talked about this with different people, like @djaglowski, @kovrus and @mar4uk, and I think this would be great to have once the connectors feature is implemented.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[wbh1]

I'm unable to find a new issue created for this, but it is still a useful idea that could replace usage of mtail in my experience.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[wbh1]

This is still something that I think would be useful. Maybe metricsgenerationprocessor could be expanded to operate on logs, too?

[jpkrohling]

We have a span metrics connector, which would likely be a better source of inspiration for this. Would you like to give it a try?

[yeya24]

Indeed it looks good to reuse the connector pattern for this one

[atoulme]

This connector probably fits the bill: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/countconnector

[bdschaap]

I think the countconnector may satisfy some use cases for converting logs to metrics. However, if I have logs that are essentially measurement values with name/value pairs in them then I'd like to convert those to metrics and export them to a metrics exporter. For example, if I have a measurement that contains an elapsed duration then I would want to use that value for the metric (rather than a count of those measurements).

[cwegener]

A 'New Component' issue was actually created at some point, but not linked back to this discussion. Here it is: #18269

[djaglowski]

Since this issue overlaps with #18269, and we eventually need a formal "New Component" issue for the idea, I'm closing this on in favor of the other. Please continue discussion over there.