You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Jan 13, 2025. It is now read-only.
test_logs_subscribe fails sporadically in CI like so: thread 'rpc_subscriptions::tests::test_logs_subscribe' panicked at 'broadcast receiver error: TestBroadcastReceiver: no data, timeout reached', rpc/src/rpc_pubsub_service.rs:197:25
We wait forever for the broadcast receiver to receive the notification and eventually time out and panic.
This appears to be caused by a race condition between completing setting up RpcSubscriptions::new_with_config and sending notification. More specifically, the subscriptions of "Alice" and "All" must be completed before the bank transaction is completely processed to ensure the log configuration is updated such that we will notify these subscribers later on.
The problem is rarely if ever seen on my laptop, and I suspect this is because the higher CPU count for CI machines results in more notification threads being spawned and widening the race condition.
There appear to be 30+ tests that use this path to setup RPC subscription service.
Proposed Solution
Ensure that RPC subscription is setup before sending notifications
The text was updated successfully, but these errors were encountered:
I've been seeing test_check_account_subscribe fail with essentially this same signature. The changes from #27481 are ensuring the setup is complete, but there is still a race condition with actually processing the Subscribed notification before the bank transaction is completely processed. Adding a short delay in fn process_notifications can force this test to timeout
Problem
test_logs_subscribe fails sporadically in CI like so:
thread 'rpc_subscriptions::tests::test_logs_subscribe' panicked at 'broadcast receiver error: TestBroadcastReceiver: no data, timeout reached', rpc/src/rpc_pubsub_service.rs:197:25
We wait forever for the broadcast receiver to receive the notification and eventually time out and panic.
This appears to be caused by a race condition between completing setting up
RpcSubscriptions::new_with_config
and sending notification. More specifically, the subscriptions of "Alice" and "All" must be completed before the bank transaction is completely processed to ensure the log configuration is updated such that we will notify these subscribers later on.The problem is rarely if ever seen on my laptop, and I suspect this is because the higher CPU count for CI machines results in more notification threads being spawned and widening the race condition.
There appear to be 30+ tests that use this path to setup RPC subscription service.
Proposed Solution
Ensure that RPC subscription is setup before sending notifications
The text was updated successfully, but these errors were encountered: