New component: Queue Processor

[akats7]

The purpose and use-cases of the new component

The purpose of a queue processor would be to decouple the queuing mechanism from the exporterhelper util. This stems from a discussion in #33007, where the ask was for the failover connector to support queueing directly in the component. As the failover connector relies on backwards propagated errors to trigger failover but with queueing enabled on the exporters will only failover once the queue is full, leading to delayed failover and without a properly configured queue dropped data. For that reason certain users have resorted to disabling the queue.

Instead of enabling a queue mechanism directly in the failover connector, it was discussed if it might be beneficial to develop a queue processor that could be re-usable for similar uses cases in the future.

The queue processor would be a component that sits directly in front of the exporters(last in the processor chain) in the pipeline and would be responsible for providing an async export queue to the nextConsumer exporters to consume from without enabling the queue on the exporter itself.

I'd be happy to contribute this component if we are able to get it sponsored.

cc. @djaglowski @sinkingpoint

Example configuration for the component

receivers:
  otlp:

processors:
  batch:
  queue:

exporters:
  otlp:


service:
  pipelines:
    traces:
       receivers: [otlp]
       processors: [batch, queue]
       exporters: [otlp]

Telemetry data types supported

traces->traces
metrics->metrics
logs->logs

Is this a vendor-specific component?

• This is a vendor-specific component
• If this is a vendor-specific component, I am a member of the OpenTelemetry organization.
• If this is a vendor-specific component, I am proposing to contribute and support it as a representative of the vendor.

Code Owner(s)

akats7

Sponsor (optional)

No response

Additional context

No response

[atoulme]

Please see the guidelines for new components here: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md

Who's the sponsor for your component. A sponsor is an approver or maintainer who will be the official reviewer of the code and a code owner for the component. You will need to find a sponsor for the component in order for it to be accepted.

To secure a sponsor, please communicate on the CNCF slack and/or attend a SIG meeting to make the case for your component.

Please also consider hosting your component in a separate repository if it suits you.