Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: add lock to flow control #899

Merged
merged 3 commits into from
Dec 12, 2023
Merged

Conversation

daniel-sanche
Copy link
Contributor

The MutationBatcher's _FlowControl class currently can experience a race condition, where the release of resources can be clobbered by a different thread adding resources to the flow. This can result in deadlock, where the flow is empty, but remains blocked waiting on mutations that are already complete

This PR adds a mutex lock, ensuring that only one thread has access to the flow control state at a time.

@daniel-sanche daniel-sanche requested review from a team as code owners December 12, 2023 21:42
Copy link

conventional-commit-lint-gcf bot commented Dec 12, 2023

🤖 I detect that the PR title and the commit message differ and there's only one commit. To use the PR title for the commit history, you can use Github's automerge feature with squashing, or use automerge label. Good luck human!

-- conventional-commit-lint bot
https://conventionalcommits.org/

@product-auto-label product-auto-label bot added size: s Pull request size is small. api: bigtable Issues related to the googleapis/python-bigtable API. labels Dec 12, 2023
self.inflight_size += batch_info.mutations_size
with self._lock:
self.inflight_mutations += batch_info.mutations_count
self.inflight_size += batch_info.mutations_size
self.set_flow_control_status()
Copy link
Contributor Author

@daniel-sanche daniel-sanche Dec 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we can keep set_flow_control_status out of the lock. Even if it occasionally gets clobbered by later changes, every add and release ends with another call to set_flow_control_status, so it should be eventually correct.

@daniel-sanche daniel-sanche merged commit e4e63c7 into main Dec 12, 2023
24 of 30 checks passed
@daniel-sanche daniel-sanche deleted the fix_flow_control_threads2 branch December 12, 2023 22:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigtable Issues related to the googleapis/python-bigtable API. size: s Pull request size is small.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants