Resource leak in validator client

[dapplion]

Describe the bug

Something is leaking resources in the validator client, both in memory and CPU. This has been observed in the past month too but a much lower scale. This capture is from Ropsten nodes, post-genesis pre-merge

[dapplion]

Managed to capture two CPU profiles at start and at a higher value:

• Profile 1: 1653974851, Tue May 31 2022 05:27:31 GMT+0000 CPU-hetzner-ropsten-3-validator-1653974851.zip CPU usage rate = 2%
• Profile 2: 1654183940, Thu Jun 02 2022 15:32:20 GMT+0000 CPU-hetzner-ropsten-3-validator-1654183940.zip CPU usage rate = 6%

Profile 1 Heavy (bottom-up) (CPU 2%)

Profile 2 Heavy (bottom-up) (CPU 6%)

Heap-snapshot 1 (CPU 2%)

Heap-snapshot 2 (CPU 6%)

Summary

So this most likely looks like a leak and improper handling of signals in the lodestar-api client request function.

• CPU increase caused by EventTarget adding and removing listeners
• Memory increase caused by many EventTarget and AbortController (78,000).

[dapplion]

Looks like the leak is caused here, where the callback being un-subscribed is not the same as the one being subscribed

Adds () => controller.abort(), removes controller.abort

lodestar/packages/api/src/client/utils/httpClient.ts

Line 85 in 86b95a7

signalGlobal.addEventListener("abort", () => controller.abort());

lodestar/packages/api/src/client/utils/httpClient.ts

Line 133 in 86b95a7

signalGlobal.removeEventListener("abort", controller.abort);