Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Whitelist for labels #8239

Closed
the-voidl opened this issue Jan 23, 2023 · 2 comments
Closed

Whitelist for labels #8239

the-voidl opened this issue Jan 23, 2023 · 2 comments

Comments

@the-voidl
Copy link

Is your feature request related to a problem? Please describe.
We're currently in the migration process of moving our log system to loki and promtail. We already have a high log volume in the dev-cluster and play around with promtail scrape config.
While experimenting it can happen that one pushes a wrong promtail config that creates too many labels.

Example: By accident I create a label out of some unique-per-line value, because I wanted to preserve it using the promtail pack stage but forgot including the label at pack.labels.

That makes promtail creating unique label values for each log line which can lead to a denial of service in insanely short time. Only visible by the log entry: err="rpc error: code = Code(429) desc = Maximum active stream limit exceeded, reduce the number of active streams (reduce labels or reduce label values), or contact your Loki administrator to see if the limit can be increased"

Describe the solution you'd like
We would love to be able to whiletlist only the few labels we really need. We already evaluated which labels to use and that won't change in the foreseeable future.

Promtail or Loki should output an error or even not start when a client tries to push values to a not-whitlisted label.

Describe alternatives you've considered
I thought of deleting log entries matching the wrong label using compactor's API. Since the problem only exists as long as you don't notice the denial of service and remove that label from the scrape config. But the precedence of mitigating DoS in the future seems higher to me than fixing old chunks.

@MichelHollands
Copy link
Contributor

@the-voidl Promtail has a labelallow stage. This might work in your case.
Please reopen this if you have more questions.

@the-voidl
Copy link
Author

Thanks, that seems to be working as intended.

Isn't in more intuitive to allow labels on Loki's side? We have multiple promtails and of course can add a labelallow on every one but this way it seems more prune to errors. I would be more satisfied to "whitelist" on one single place in the stack.
In the end it's Loki that is handling the chunks and "too many streams".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants