Fix a bug that causes sources to stall

[dominiklohmann]

📔 Description

To reproduce the bug, run the following commands without this fix:

vast start

while true; do
  sleep 30
  cat integration/data/suricat/eve.json
done | vast import suricata

PR to be taken over by @tobim, who as per our call has an idea for another improvement.

📝 Checklist

• All user-facing changes have changelog entries.
• The changes are reflected on docs.tenzir.com/vast, if necessary.
• The PR description contains instructions for the reviewer, if necessary.

🎯 Review Instructions

n/t

[tobim]

It took me a while to understand the failed test. What happens is the reader returns with stalled after reading the initial lines. It is expected to return normally afterwards because the batch-timeout should trigger. With the changes in the first commit, that does not happen any more, because the subsequent rounds don't produce any data.

The original change was made to fix the blocking of the source actor. It was achieved by making read re-entrant. But while the time-point for last_batch_sent_ was moved to the readers state, the same wasn't done for the produced counter.

I believe the implementation of the readers can be simplified if read() gets a time-point at which it must exit from it's loop, and this time-point is managed in the source.

[dominiklohmann]

I can't review this as I've opened the PR, but here's some feedback on your changes @tobim:

I'd like to see better documentation on ec::stalled and ec::timeout. I did not immediately grok the difference when reading the comments currently above their definition.

Generally, this works aside from the unit tests as you mentioned.

For the proposed reader change, can you show the desired API first?

[mavam]

For the proposed reader change, can you show the desired API first?

I would like to avoid spending more cycles here right now. We have the input load-balancing epic in the backlog and we'll revisit the source and reader design at that point.

[tobim]

I implemented a workaround that fixes the immediate problems at the cost of a bit extra complexity. We plan to change the interaction between the source and reader to make it more straightforward soon, I hope this is acceptable in the meantime.

[dominiklohmann]

I tested this locally. @tobim feel free to approve. Changelog entry lgtm.