Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit to kafka-python's Queue size. #187

Closed
JDShu opened this issue Aug 4, 2014 · 2 comments
Closed

Limit to kafka-python's Queue size. #187

JDShu opened this issue Aug 4, 2014 · 2 comments
Labels

Comments

@JDShu
Copy link

JDShu commented Aug 4, 2014

I saw this is brought up in #137. When using kafka-python I've encountered cases where my application's memory usage increases uncontrollably because the internal kafka queue is filling up faster than the items can be shipped to Kafka. In my application code I hacked in some logic to check the producer.queue.queuesize before calling send_messages, but maybe it would be better and generally useful to others to set the Queue's maxsize in kafka-python itself and pass in a parameter to Producer?

@dpkp
Copy link
Owner

dpkp commented Aug 18, 2014

sounds good -- can you submit a PR ?

@dpkp dpkp added the producer label Aug 22, 2014
@dpkp
Copy link
Owner

dpkp commented Jun 9, 2015

max queue size can now be configured via producer kwargs async_queue_maxsize

@dpkp dpkp closed this as completed Jun 9, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants