Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix light client plumbing in beacon processor #6993

Merged

Conversation

michaelsproul
Copy link
Member

Issue Addressed

Our Holesky nodes running with the light client enabled were logging messages about full queues:

Feb 12 22:09:28.949 ERRO Work queue is full queue: unknown_light_client_optimistic_update, queue_len: 128, msg: the system has insufficient resources for load, service: bproc

I thought this might be genuine overload, but it turns out this queue was never being read from!

Proposed Changes

  • Rename light-client related queues in the beacon processor for clarity.
  • Ensure all light-client related queues are being popped from.

Additional Info

I noticed that the sampling_result_queue is never being read from either, but I've added a TODO(das) to address that in future.

@michaelsproul michaelsproul added bug Something isn't working light-client v7.0.0-beta.0 New release c. Q1 2025 labels Feb 12, 2025
Copy link
Member

@jimmygchen jimmygchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, LGTM!

@jimmygchen jimmygchen added the ready-for-merge This PR is ready to merge. label Feb 13, 2025
@michaelsproul
Copy link
Member Author

For the record: the impact of the bug isn't that we never processing light client updates from gossip. It's that we never process them once queued. It takes a while for the queue to build up, because if the processor is not at capacity, we will still schedule and run processing immediately here:

match work {
_ if can_spawn => self.spawn_worker(work, idle_tx),

@michaelsproul
Copy link
Member Author

@mergify queue

Copy link

mergify bot commented Feb 13, 2025

queue

✅ The pull request has been merged automatically

The pull request has been merged automatically at 25f804a

@mergify mergify bot merged commit 25f804a into sigp:release-v7.0.0-beta.0 Feb 13, 2025
30 checks passed
@michaelsproul michaelsproul deleted the light-client-plumbing branch February 13, 2025 01:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working light-client ready-for-merge This PR is ready to merge. v7.0.0-beta.0 New release c. Q1 2025
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants