Track down Memory Leak in Teraslice v2.9.2 #3945
Assignees
Labels
Comments
|
Recall, the versions with the leak were:
We are going to make two release candidates:
|
|
We're getting this tracked down, we see this in the logs: This PR added the Abort controller, which appears to be leaking event handlers: |
godber
pushed a commit
that referenced
this issue
Feb 5, 2025
This PR makes the following changes: - Create an `abortController` in the core client `handleSendResponse` function if `sendAbortSignal` is set to true. This is only the case for when the execution-controller client sends the `worker:slice:complete` event. On a `server:shutdown` event the `abortController` will abort awaiting a server response so it can shutdown properly. - Remove the `abortController` from the execution-controller client - bump teraslice-messaging from 1.10.3 to 1.10.4 ref: #3945
|
This leak should be resolved in Teraslice |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
About a month ago we rolled out Teraslice
v2.9.2into our production environment. After about 3 or 4 weeks we started having Teraslice workers OOM, due to a newly introduced but very slow memory leak.The things that changed in the jobs were:
We've been tracking this internally so there's more information that in this issue. I did some internal testing and rolled back the kafka asset to 5.1.0 and it didn't change the situation, the leak still existed (it takes about 18-24 hours to conclusively see the leak).
Here are the relevant versions from the latest releases:
The text was updated successfully, but these errors were encountered: