You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
We're currently in the migration process of moving our log system to loki and promtail. We already have a high log volume in the dev-cluster and play around with promtail scrape config.
While experimenting it can happen that one pushes a wrong promtail config that creates too many labels.
Example: By accident I create a label out of some unique-per-line value, because I wanted to preserve it using the promtail pack stage but forgot including the label at pack.labels.
That makes promtail creating unique label values for each log line which can lead to a denial of service in insanely short time. Only visible by the log entry: err="rpc error: code = Code(429) desc = Maximum active stream limit exceeded, reduce the number of active streams (reduce labels or reduce label values), or contact your Loki administrator to see if the limit can be increased"
Describe the solution you'd like
We would love to be able to whiletlist only the few labels we really need. We already evaluated which labels to use and that won't change in the foreseeable future.
Promtail or Loki should output an error or even not start when a client tries to push values to a not-whitlisted label.
Describe alternatives you've considered
I thought of deleting log entries matching the wrong label using compactor's API. Since the problem only exists as long as you don't notice the denial of service and remove that label from the scrape config. But the precedence of mitigating DoS in the future seems higher to me than fixing old chunks.
The text was updated successfully, but these errors were encountered:
Isn't in more intuitive to allow labels on Loki's side? We have multiple promtails and of course can add a labelallow on every one but this way it seems more prune to errors. I would be more satisfied to "whitelist" on one single place in the stack.
In the end it's Loki that is handling the chunks and "too many streams".
Is your feature request related to a problem? Please describe.
We're currently in the migration process of moving our log system to loki and promtail. We already have a high log volume in the dev-cluster and play around with promtail scrape config.
While experimenting it can happen that one pushes a wrong promtail config that creates too many labels.
Example: By accident I create a label out of some unique-per-line value, because I wanted to preserve it using the promtail
pack
stage but forgot including the label at pack.labels.That makes promtail creating unique label values for each log line which can lead to a denial of service in insanely short time. Only visible by the log entry:
err="rpc error: code = Code(429) desc = Maximum active stream limit exceeded, reduce the number of active streams (reduce labels or reduce label values), or contact your Loki administrator to see if the limit can be increased"
Describe the solution you'd like
We would love to be able to whiletlist only the few labels we really need. We already evaluated which labels to use and that won't change in the foreseeable future.
Promtail or Loki should output an error or even not start when a client tries to push values to a not-whitlisted label.
Describe alternatives you've considered
I thought of deleting log entries matching the wrong label using compactor's API. Since the problem only exists as long as you don't notice the denial of service and remove that label from the scrape config. But the precedence of mitigating DoS in the future seems higher to me than fixing old chunks.
The text was updated successfully, but these errors were encountered: