-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prevent Alfred Telemetry Prometheus endpoint threads from stacking up #70
Comments
Use our own threadpool with a configurable Also worth reading with regards to the thread pool: |
Or wrapping in a Future with a timeout: We just have to check if that timeout would also stop the query. |
My main concern is: how are we going to define this timeout? This needs to match exactly with the Prometheus poll interval? If it's less, Prometheus requests might be aborted. If it's more, threads might still stack up? |
You are right. A timeout could be an "extra". Controlling concurrency is probably the more structural fix. We can use a lock: There is a tryLock() method that returns false immediately if another thread holds the lock. In that case Telemetry can return an error code. This way you can limit Telemetry to only have one active thread at the same time. Would 'one' be acceptable? |
#70 Manage maximum number of Prometheus requests with a Semaphore
A client had an issue where fetching some metrics was getting very slow, slower then the scraping frequency of Prometheus. After a while the http threads where being exausted.
Possible solution: Wrap the the fetching of metrics in a separate thread that gets killed when taking too long, returning an error message.
The text was updated successfully, but these errors were encountered: