-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Promtail: Add configuration to drop batches when rate limited by Loki #7973
Promtail: Add configuration to drop batches when rate limited by Loki #7973
Conversation
ee9cd84
to
7be2f5a
Compare
I realized when looking at the metrics with this change that I was not able to distinguish between drops due to rate limiting versus other reasons like stream limiting or exhausting all retries. Use of a |
2abbef9
to
8333b68
Compare
8333b68
to
f088e7d
Compare
3812654
to
23bb42a
Compare
./tools/diff_coverage.sh ../loki-target-branch/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell. + ingester 0%
+ distributor 0%
+ querier 0%
+ querier/queryrange 0%
+ iter 0%
+ storage 0%
+ chunkenc 0%
+ logql 0%
+ loki 0% |
Enabling this option has eliminated drops we were seeing with Loki canary when using it with a shared Promtail alongside a heavily rate-limited tenant. cc @MichelHollands @chaudum @DylanGuedes for review/comment. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks for the contribution!
What this PR does / why we need it:
This change will allow for more deterministic behavior by multitenant Promtail instances where one or more tenants are being heavily throttled due to limits exceeded within Loki. Specifically, it will reduce the performance impact of having exponential backoff cause HOL blocking for rate limit response codes while other tenants are ready to send batches.
Which issue(s) this PR fixes:
Fixes #7972
Special notes for your reviewer:
Checklist
CONTRIBUTING.md
guideCHANGELOG.md
updateddocs/sources/upgrading/_index.md