Get incoming message size from MQTT

[Inovum]

Feature Request

Opening a feature request kicks off a discussion.

Proposal:

Similar to how the Net plugin can collect metrics from the local machine such as bytes sent/received from different network interfaces, would like to be able to integrate this with the MQTT Consumer plugin in order to get the size of the incoming data from MQTT into InfluxDB. In other words, get the total size of the incoming data for a certain interval (i.e. MB/Min)

Current behavior:

The MQTT plugin only collects the data itself, not info about the data such as message size or byte count. Similarly, the Net plugin doesn't allow us to monitor MQTT traffic, just local traffic.

Desired behavior:

Be able to get the total message size of incoming data from MQTT and write this message size into influxDB.

Use case:

We want to be able to see the datarate of incoming traffic from MQTT into the broker, and write this rate/size to influxDB in order to see how much data is being sent back and forth.

[reimda]

Telegraf has a separate plugin called the internal plugin that plugins can use to report information like this.

https://github.com/influxdata/telegraf/blob/master/plugins/inputs/internal/README.md

The MQTT input plugin doesn't report internal stats but it might be possible to extend it and add the stats you're interested in.

Here is an example of an input plugin that does report similar stats:

telegraf/plugins/inputs/influxdb_v2_listener/influxdb_v2_listener.go

Line 65 in 6113bba

bytesRecv selfstat.Stat

Are you able to implement this and submit a pull request? I think it would be a useful feature and something that would be straightforward to review and merge. Thanks!

[Inovum]

This is my first time working with telegraf and its plugins so I'm not sure that it's doing exactly what I'd like it to do just quite yet. It did pass the tests ran with make but is there a way to test how it performs in Influx without actually merging it into the main code in case it needs to be changed more?

[mrro253]

This should do it I believe, first it gets the size of each component of msg and then the size of msg itself, sums it up, then adds it as a tag. If you're wanting to get throughput or the rate of the data (bytes/min or bytes/sec etc.) you could pass the output from this plugin to basicstats and use rate/sum/sd depending on needs.

[Hipska]

Do you want to see the actual bytes (this is a bit harder) or is only the number of messages also a good metric?

[mrro253]

I was thinking getting the actual number of bytes/size of message in bytes, which I would then be able to pass into an aggregator such as basicstats to get the number of bytes being sent per minute or per a defined interval.

[Hipska]

Okay you can try it. But please also add a stat counter for the number of messages, as it seems there can be more than 1 metric per message retrieved, so it might also be a good measurement for some users..

[mrro253]

I've added the message counter that will increment the selfstat object each time a message comes in. What I'm missing now and can't figure out is being able to successfully read the "collect_bytes_received" option from the configuration from the mqtt_consumer.go file to only collect the data when the field is set.