Queue messages are acknowledged immediately when they first should be successfully emitted

[danielnelson]

Bug report

This is a known problem but I don't think we have an issue opened explicitly for it.

Telegraf immediately acks incoming queue messages in the various queue consumers. It does this because there is no mechanism to learn when the metrics has been successfully outputted.

Metrics can be copied, modified, and can cause new metrics to be produced. A received metric should not be treated as being emitted successfully until all produced metrics are emitted, or there should be some options for configuring this behavior. The most routine time this occurs is when you have multiple outputs.

When this issue is resolved, it will be possible to use any queue and fully gain the durability properties of the queue. For example you can run a queue, potentially even locally, with Telegraf writing new metrics to the queue and reading from the queue and sending items read from the queue to the final output.

Example config:

[[inputs.cpu]]

[[outputs.nats]]
  [inputs.nats.tagdrop]
    telegraf_output = ["influxdb"]

[[inputs.nats_consumer]]
  [inputs.nats.tags]
    telegraf_output = "influxdb"

[[outputs.influxdb]]
  tagexclude = ["telegraf_output"]
  [inputs.nats.tagpass]
    telegraf_output = ["influxdb"]

This has been discussed in:

• Outputs block inputs when batch size is reached #2919
• [Feature Discussion] Dealing with congestion by adding internal buffer handling options #2905

System info:

<= Telegraf 1.6.0

Steps to reproduce:

1. Enable a queue consumer plugin
2. Start Telegraf
3. kill -9 Telegraf

Expected behavior:

No unemitted metrics are acknowledged

Actual behavior:

Unemitted metrics are acknowledged