You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
With the merge of vectordotdev/vector#19637, we thought of using that decoder instead of a custom lua one (which takes a lot of cpu), but it seems that we have malformed line protocol messages and using the decoding.codec=influxdb option in sources does allow to route the malformed messages somewhere else, as the data is being dropped.
The alternative we see is to use a remap transform with a parse_influxdb function and drop_on_abort=true, so we can route the malformed data (<component_id>.dropped) to our custom lua decoder which is less strict.
Doubt: The source decoder option is decoding.codec=influxdb and not decoding.codec=line_protocol, shoud we call this function parse_influxdb in order to be consistent with the vector option? or should we change the vector config spec to use decoding.codec=line_protocol?
The text was updated successfully, but these errors were encountered:
Agreed, most of our codecs have analogues in VRL for cases where people want more control. We've also previously discussed having sources be able to route events that fail codec parsing to another output, which I think would also help here, but that is a bigger change and I think we'd want this VRL function still anyway.
I think we should call this parse_influxdb to match the codec name.
The existing parse_* functions can be used as an example if you are anyone else wants to take a shot at this.
We've also previously discussed having sources be able to route events that fail codec parsing to another output,
Yes! that is what I was initially looking for, and I think it would be a very useful feature as we wouldn't have to use this additional remap transform.
The existing parse_* functions can be used as an example if you are anyone else wants to take a shot at this.
We are very interested in this feature, so I could address it by myself soon. I took a look to the others parse_* functions and it does not seem too complicated to glue the influxdb_line_protocol crate in it
Thanks!!
jorgehermo9
changed the title
Add parse_line_protocol function
Add parse_influxdb function
Aug 6, 2024
With the merge of vectordotdev/vector#19637, we thought of using that decoder instead of a custom lua one (which takes a lot of cpu), but it seems that we have malformed line protocol messages and using the
decoding.codec=influxdb
option in sources does allow to route the malformed messages somewhere else, as the data is being dropped.The alternative we see is to use a remap transform with a
parse_influxdb
function anddrop_on_abort=true
, so we can route the malformed data (<component_id>.dropped
) to our custom lua decoder which is less strict.In order to handle this failing line protocol messages, we think that a
parse_influxdb
vrl function is needed. The implementation should be very similar to https://github.com/vectordotdev/vector/blob/210ff0925d391213556f07bf6ce621967f0368ca/lib/codecs/src/decoding/format/influxdb.rs#L97Doubt: The source decoder option is
decoding.codec=influxdb
and notdecoding.codec=line_protocol
, shoud we call this functionparse_influxdb
in order to be consistent with the vector option? or should we change the vector config spec to usedecoding.codec=line_protocol
?The text was updated successfully, but these errors were encountered: