a general question regarding performance

[liujoey]

Hi,

I have a very general question regarding saving the message list in the kafka table.

I have a topic, the interval between each message is about 5 seconds. Now I want to compute mean of that topic for a window size of 4 weeks. According to your implementation, it means the table needs to be updated for every 5 seconds and there will be 4 weeks worth of data sitting in a single window.

Now my concern is about the following three lines of code:

messages = table["{{ source_topic_name }}"].value()
messages.append(message)
table["{{ source_topic_name }}"] = messages

Since messages in my case will be a huge list (483840 messages for the whole 4 weeks, and this is only for 1 user, I need to handle 1 million users, so...), my concern is that the table_change_log in Kafka will become huge as well.
My question, is Faust been optimized such that the append operation in above example ONLY append one message to the change log, or otherwise the change log actually needs to exchange 4 weeks of data points in and out for every update?

Thank you.