Skip to content
This repository has been archived by the owner on Jan 13, 2025. It is now read-only.

test_logs_subscribe occasionally times out #27480

Closed
bw-solana opened this issue Aug 30, 2022 · 1 comment · Fixed by #27481
Closed

test_logs_subscribe occasionally times out #27480

bw-solana opened this issue Aug 30, 2022 · 1 comment · Fixed by #27481

Comments

@bw-solana
Copy link
Contributor

bw-solana commented Aug 30, 2022

Problem

test_logs_subscribe fails sporadically in CI like so:
thread 'rpc_subscriptions::tests::test_logs_subscribe' panicked at 'broadcast receiver error: TestBroadcastReceiver: no data, timeout reached', rpc/src/rpc_pubsub_service.rs:197:25

We wait forever for the broadcast receiver to receive the notification and eventually time out and panic.

This appears to be caused by a race condition between completing setting up RpcSubscriptions::new_with_config and sending notification. More specifically, the subscriptions of "Alice" and "All" must be completed before the bank transaction is completely processed to ensure the log configuration is updated such that we will notify these subscribers later on.

The problem is rarely if ever seen on my laptop, and I suspect this is because the higher CPU count for CI machines results in more notification threads being spawned and widening the race condition.

There appear to be 30+ tests that use this path to setup RPC subscription service.

Proposed Solution

Ensure that RPC subscription is setup before sending notifications

@bw-solana
Copy link
Contributor Author

I've been seeing test_check_account_subscribe fail with essentially this same signature. The changes from #27481 are ensuring the setup is complete, but there is still a race condition with actually processing the Subscribed notification before the bank transaction is completely processed. Adding a short delay in fn process_notifications can force this test to timeout

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant