Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DataShard: read iterators quota bug and unexpected stream resets #7674

Closed
snaury opened this issue Aug 12, 2024 · 1 comment · Fixed by #7697
Closed

DataShard: read iterators quota bug and unexpected stream resets #7674

snaury opened this issue Aug 12, 2024 · 1 comment · Fixed by #7697
Assignees
Labels
area/datashard Issues related to datashard tablets (relational table partitions) bug Something isn't working prio:high

Comments

@snaury
Copy link
Member

snaury commented Aug 12, 2024

Datashard doesn't handle decreasing quotas very well. When client specifies lower quotas in TEvReadAck while TEvReadContinue or TTxReadContinue are inflight, the read could fail by the ShouldStop() condition too early, before LastProcessedKey is initialized, which causes query to unexpectedly restart later. It's unclear whether this specific problem causes duplicate and out-of-order rows for some clients.

@snaury snaury added bug Something isn't working area/datashard Issues related to datashard tablets (relational table partitions) labels Aug 12, 2024
@snaury snaury self-assigned this Aug 12, 2024
@snaury
Copy link
Member Author

snaury commented Aug 13, 2024

Possible race condition scenario:

  • KQP read actor starts a read with both limit and quota set to 1000 rows.
  • DataShard produces two chunks of 450 rows each, remaining quota is 100 rows.
  • DataShard's TEvReadContinue/TTxReadContinue is pending and hasn't completed yet.
  • KQP read actor receives both chunks and sets ReceivedRowCount = 900, notifying compute actor.
  • KQP read actor processes the first chunk and sends TEvReadAck for SeqNo=1 with row quota = 1000 - 900 = 100 rows (this is imho another subtle bug and may lead to read inefficiency)
  • DataShard processes ack for SeqNo=1 with quota of 100 rows, unacknowledged queue has SeqNo=2 with 450 rows, so the remaining quota becomes -350 (0) iterator is marked as exhausted quota.
  • TTxReadContinue starts/resumes and attempts to iterate. At the very start of the loop its ShouldStop() method returns true, because RowsRead (0) is greater-or-equal to Quota.Rows (0).
  • Read stops and produces a "useful" result that signals iterator is out-of-quota.
  • Since IterateRange was never actually called LastProcessedKey is left uninitialized and empty.
  • When UpdateState is called it overwrites State.LastProcessedKey with an empty value, losing current iterator position.
  • KQP read actor processes the second chunk and sends TEvReadAck for SeqNo=2 with row quota = 100 rows.
  • DataShard receives this ack and since it no longer has any rows inflight remaining quota is set to 100 rows, iterator is unblocked and TEvReadContinue is rescheduled.
  • TTxReadContinue starts iteration, but since State.LastProcessedKey is empty the query effectively restarts from the beginning of the range, and DataShard sends duplicate rows to KQP read actor.

In practice both TEvReadAck messages are sent in quick succession, but there's still a very narrow window where TTxReadContinue::Execute may run between processing those two messages, triggering the bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/datashard Issues related to datashard tablets (relational table partitions) bug Something isn't working prio:high
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant