How to read entire log?

[CoenraadS]

Hello

In the documentation, there is a hard coded message count of 100, at which point it stops reading.

I have a situation where I want to read the log from start until the end, then stop listening.

I could implement the documentation example, replacing the hardcoded count with a check if messageContext.Offset is the latest known offset. However it is unclear how I would get the latest offset. I could specify OffsetTypeOffset(ulong.MaxValue), read 1 message which should be the latest, then restart the stream from the beginning until I reach that message again. (I don't mind if I miss any messages that get appended during this time)

Would this be the correct approach? It wouldn't work if the stream was empty. I don't really want to rely on timeouts to figure out if I'm at the end of the stream either.

I think ideally I would like to read the stream metedata somehow to get the latest offset that way without actually having to try to consume the stream.

[Gsantomaggio]

Hi @CoenraadS

I have a situation where I want to read the log from start until the end, then stop listening.

There is no way to understand the last_offset. The only way to solve your use case is to create a consumer and implement a timeout.

If you don't have messages for some time, you can consider it the end of the stream.

The last_offset concept is a temporary value in common use cases. After a bit, the value can change.

We will think about providing stats about that. ( Not sure we will give it)

[CoenraadS]

I have a centralized point where I write messages. If I keep track of the latest message publishing id, is that equivalent to offset? I can then make that available to my application instead.

[Gsantomaggio]

No, the publishing ID is different from the offset.
If you know the last message in your stream during the write, you can mark it somehow. You can use application properties with a specific field. does it make sense?

[acogoluegnes]

Note the "last offset" of a stream is a moving target, unless the stream is read-only (no application publishing to it).

There are different ways to read the whole stream:

• Use the StreamStats command and use the committedChunkId value. This is a good enough approximation of the end of the stream at a given time. The value - how precise it is - is stale as soon as you receive it if messages are published to the stream anyway.
• Create a consumer with the LAST offset specification and calculate the last offset with the chunk information (use the chunk ID and the number of messages in the chunk). This will require to use the low-level API of the client library. Then you can create a consumer and stop consuming when you reach the "last" offset.
• Create a consumer with the FIRST offset specification, process messages, and use a timeout mechanism to stop the consumer after no messages have been received in x seconds (10, 20 seconds should be appropriate values).

[CoenraadS]

Thanks for the ideas, I'm aware it's a moving target, unfortunately it's a requirement I have to fulfill. Basically that a client can download all messages at time of connection, over a potential bad network (hence trying to avoid timeouts) I will try idea 2, it sounds doable

…

On Tue, May 28, 2024, 6:13 PM Arnaud Cogoluègnes ***@***.***> wrote: Note the "last offset" of a stream is a moving target, unless the stream is read-only (no application publishing to it). There are different ways to read the whole stream: - Use the StreamStats command and use the committedChunkId value. This is a good enough approximation of the end of the stream at a given time. The value - how precise it is - is stale as soon as you receive it if messages are published to the stream anyway. - Create a consumer with the LAST offset specification and calculate the last offset with the chunk information (use the chunk ID and the number of messages in the chunk). This will require to use the low-level API of the client library. Then you can create a consumer and stop consuming when you reach the "last" offset. - Create a consumer with the FIRST offset specification, process messages, and use a timeout mechanism to stop the consumer after no messages have been received in x seconds (10, 20 seconds should be appropriate values). — Reply to this email directly, view it on GitHub <#378 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKVVYPR7W5CS4FL7TGHQLTZEQ4C3AVCNFSM6AAAAABICXYQWOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TKNZXHAYDA> . You are receiving this because you were mentioned.Message ID: <rabbitmq/rabbitmq-stream-dotnet-client/repo-discussions/378/comments/9577800 @github.com>