Feature: some way to limit size of inputs/outputs

[nfirvine]

Sometimes a misbehaving client will post a huge measurement to a Telegraf input. It would be nice to have a feature to drop those inputs.

Use case:

Johnny Appdeveloper writes an app that posts a measurement to Telegraf with a tag generated from the code. The tag happens to be 3 MB long, by accident.

Nick in Ops is running the metrics pipeline, and decides a 3 MB Telegraf measurement is never on purpose and can hurt other users of the service, so will drop the measurement, log it, and count it.

[danielnelson]

I agree there should be a limit. Not sure if it needs to be user defined or not. Exactly where to place this check is also debatable, some inputs already have limits (http_listener).

[danielnelson]

In InfluxDB, this key must be less than 32768 bytes long: <measurement>,<tag1=value1>,<tagN=valueN>#!~#<field>.

[nfirvine]

Which key? I'm not sure exactly what you mean... The series?

…

On Fri, Mar 31, 2017, 14:07 Daniel Nelson ***@***.***> wrote: In InfluxDB, this key must be less than 32768 bytes long: <measurement>,<tag1=value1>,<tagN=valueN>#!~#<field>. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2606 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAdmrdWMf9BvuCO5j_V7m6zr-ilajnBks5rrWsngaJpZM4MvEKA> .

[danielnelson]

Sorry, didn't add enough context, the key is that string. This is another limit we have in the TICK stack, and it would probably makes sense to limit Telegraf metrics to this size at most.

[nfirvine]

Is <field> the field key or <fieldkey>=<fieldvalue>?

BTW, feel free to just give me a line number. Having trouble grepping for the info you're' giving me.

[danielnelson]

Just the field name. https://github.com/influxdata/influxdb/blob/master/tsdb/engine/tsm1/writer.go#L98-L99

[danielnelson]

I can think of several ways this could be implemented, but they all have some drawbacks. In some sense it should be the job of the input and output plugins to handle limits, but I see the draw of having a system wide limit.

The most straightforward place to limit metrics system wide would be when added to the accumulator, but we would want to ensure that we aren't creating a significant performance hit.

[nfirvine]

To scratch my particular itch, all I'd need it for is the kafka_consumer input. I took a quick peek at the code, and it seems like it could be pretty simple if len(buf) > MAX_LEN then drop with error. Not sure if this is the right approach though. If you think that's fine, I could probably find the time for a PR in the next couple of weeks.

[danielnelson]

I think that would be fine, what do you think MAX_LEN should be?

[danielnelson]

64 * 1024 bytes would probably be good if there is nothing more fitting for kafka.

[nfirvine]

I actually have very little data on how "wide" our metrics end up being, besides the ones that are very big and clearly pathological. May need to measure this. But yes, 64k's a good start.