Fluent Bit stream processor returning empty Keys after processing a message

[xenbyte]

Bug Report

Description:
Before updating to version 2.1.2, we were using Fluent Bit version 1.7, and the stream processor worked perfectly. We were obtaining satisfactory results from the stream processor using SQL syntax.

However, after updating to version 2.1.2, we encountered issues with the stream processor. Upon thorough analysis and testing, we discovered that the keys used for GROUP BY operations were no longer present in the resulting logs. Nevertheless, the corresponding values were correct.

The syntax SQL we used for applying stream processing with window tumbling is this:

CREATE STREAM agg.logs WITH (tag='agg.logs') AS SELECT server, host, SUM(size_sent), SUM(body_size), COUNT(*)  FROM TAG:'logs.*' WINDOW TUMBLING (15 SECOND) GROUP BY server, host;

To Reproduce

• Upgrade Fluent Bit from version 1.7 to version 2.1.
• Configure and execute the stream processor using the following SQL syntax for window tumbling:

[STREAM_TASK]
    Name   agg_stream
    Exec   CREATE STREAM agg.logs WITH (tag='agg.logs') AS SELECT server, host, SUM(size_sent), SUM(body_size), COUNT(*)  FROM TAG:'logs.*' WINDOW TUMBLING (15 SECOND) GROUP BY server, host;

• Observe the resulting logs.

Expected behavior
The resulting logs should retain the keys used for GROUP BY operations, while ensuring that the values remain accurate, even when applying window tumbling:

{"server"=>"stage01", "host"=>"www.example.com", "SUM(size_sent)"=>14124, "SUM(body_size)"=>0, "COUNT(*)"=>67}

Actual Behavior:
The keys used for GROUP BY operations are missing in the resulting logs when applying window tumbling, although the values are correct:

{""=>"stage01", ""=>"www.example.com", "SUM(size_sent)"=>14124, "SUM(body_size)"=>0, "COUNT(*)"=>67}

Environment:

• Version used: v2.1.2
• Configuration:

[SERVICE]
    flush        1
    daemon       off
    log_level    warning
    HTTP_Server  On
    HTTP_Listen  127.0.0.1
    HTTP_PORT    2020
    storage.metrics on
    Log_File /var/log/fluentbit.log
    Streams_File /etc/fluentbit/script/stream.conf

[INPUT]
    Name      syslog
    tag       logs.acc
    Path      /var/log/logs_acc.sock
    Unix_Perm 777
    Mem_Buf_Limit 15M
    Buffer_Chunk_Size 1M

[OUTPUT]
    Name stdout
    Match agg.logs

streams.Conf:

[STREAM_TASK]
    Name   agg_stream
    Exec   CREATE STREAM agg.logs WITH (tag='agg.logs') AS SELECT server, host, SUM(size_sent), SUM(body_size), COUNT(*)  FROM TAG:'logs.*' WINDOW TUMBLING (15 SECOND) GROUP BY server, host;

Additional Information:

Fluent Bit version: 2.1.2
Operating System: Debian GNU/Linux 11 (bullseye)
Reproducibility: Yes, the issue is consistently reproducible.
Deployment: Container Based

Please let me know if there is any further information or clarification needed

[xenbyte]

Any updates on this?

[canob]

This bug happen to me too, but I found a workaround for that, which is use the "AS same_name_field" for every field.
So, for your example, I rewrite the query like this:

CREATE STREAM agg.logs WITH (tag='agg.logs') AS SELECT server AS server, host AS host, SUM(size_sent) AS SUM_size_sent, SUM(body_size) AS SUM_body_size, COUNT(*) AS COUNT_total FROM TAG:'logs.*' WINDOW TUMBLING (15 SECOND) GROUP BY server, host;

Regards!

[koleini]

@xenbyte going to look into this in the next couple of days.