Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix deadlock when a sequence of long-ish strings appear in the input (bug #101) #102

Merged
merged 3 commits into from
Mar 1, 2023

Conversation

kosak
Copy link
Contributor

@kosak kosak commented Mar 1, 2023

Background: we have FIFO queues between the thread that parses the input into cells, and the threads for each column that parse those cells into data.

Each FIFO queue is in turn composed of a "control queue" and a "data queue". The control queue indicates what kind of data cell was pushed onto the data queue (there are two choices: either a short-ish string, packed with other strings into a block, or a longer string in its own byte[] array).

Importantly, because our flow control is based on limiting the number of data queue blocks outstanding (per DenseStorageConstants.MAX_UNOBSERVED_BLOCKS), we want to flush the control queue every time we fill a block on the data queue, so the consumer has a chance to consume the data. If we did not do this, in some cases the data queue would run too far ahead, the flow control be invoked to block the writer, but the reader would also be blocked because it is still waiting on control queue notifications, which haven't arrived because the latest control queue block isn't full and hasn't been flushed yet.

See #101

@devinrsmith devinrsmith merged commit 769bfb0 into deephaven:main Mar 1, 2023
@kosak kosak deleted the kosak_bug-101 branch March 1, 2023 15:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants