fixed: flush remaining rows on termination #67
Conversation
| s.lastSeenCursor = cursor | ||
| s.lastSeenFinalBlock = data.FinalBlockHeight |
There was a problem hiding this comment.
Should we simply add those to flush directly instead? That would avoid maintaining some state before calling the function.
It also makes me wondering what happen if flush is called through OnTerminating but last block cursor/final block is the one before. It seems that it's a possibility here.
There was a problem hiding this comment.
Should we simply add those to flush directly instead? That would avoid maintaining some state before calling the function.
That doesn't work because we don't know the last seen cursor and final block when executing flush from OnTerminating
It also makes me wondering what happen if flush is called through OnTerminating but last block cursor/final block is the one before. It seems that it's a possibility here.
True, I guess we can just lock a mutex in HandleBlockScopedData to ensure that we always have a block fully processed before anything is flushed.
There was a problem hiding this comment.
added flushMutex that ensures each block is fully processed and the lastSeen stats are saved before any flush can happen
fschoell
force-pushed
the
fix/flush_on_shutdown
branch
from
ff20eaa to
2c78f13
Compare
this fixes a potential case of missing block data in case the stop block number doesn't match the flush interval. So if you were processing the range of 0:1100 with a flush interval of 1000 you'd be missing all blocks > 1000 as flushes only happen on
%1000block numbers.The sql-sink will automatically stop at 1100, not flush the block range of 1000 to 1100 and also not produce any error letting the user know that there are blocks within their requested ranges that haven't been flushed.