-
-
Notifications
You must be signed in to change notification settings - Fork 92
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Using excessive file descriptors #798
Comments
Hm that's interesting. I'll have to dig into this as I'm not sure what is causing this - Sorry for the inconvenience! |
Cheers Tom! I've started to take a look today and I'll probably try and spend some time on it next week too, but you'll obviously be more familiar with the code. |
Yes but unfortunately not have the bandwidth to fix this so soon - Sorry :/ Happy to go through a PR if you come up with something! But hopefully upgrading to .NET Core 3.1 fixes this, but not there yet (#718) Would be good to learn about the culprit indeed. |
This might be due to a bad HttpClient not being re-used: https://aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/ Have to take a closer look though. |
@ResDiaryLewis was the error due to an OS-level issue or did Promitor throw an exception? If so, do you still have the exception & stack trace? |
@tomkerkhove it was an error emitted by several other applications which happened to be scheduled to the same Nodes as Promitor in our cluster.
As a stop-gap solution, we've added a CronJob to our cluster now which periodically kills Promitor. EDIT:
Some other Pods including Tiller, nginx-ingress, cert-manager, and a TeamCity agent were crashing at the same time. We determined that Promitor was the culprit using the aforementioned shell commands. |
Thanks for the additional information and sorry for that! Will have a look but since we don't use |
Might have found something in #844 @ResDiaryLewis. Looks like Azure SDK create a ton of HttpClients. |
Nice find @tomkerkhove 👍 |
Are you still seeing this? |
Promitor uses too many Linux file descriptors, mainly sockets. We're running Promitor on Kubernetes and this has resulted in the Node running out of file descriptors a few times, where other Pods will crash with
too many open files
error.Expected Behavior
Promitor should re-use sockets or release them after use.
Actual Behavior
Promitor opens sockets and doesn't seem to release them in a timely fashion.
Steps to Reproduce the Problem
810243
.Configuration
Provide insights in the configuration that you are using:
*/1 * * * *
Used scraping configuration
This is our staging configuration, which scrapes less resources but still encounters the same issue.
Specifications
The text was updated successfully, but these errors were encountered: