This repository has been archived by the owner on Oct 23, 2022. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 163
Flaky multi-node tests #134
Comments
For some reason the tests have started getting better. This might be because of libp2p 0.19 or just random luck with the other builds running on gha. |
Currently the |
I've found a semi-reliable way of reproducing the issue; I'm suspecting insufficient node cleanup to be the root cause. |
Ok, so it might require a bit of a broader solution, as even though my fix seems to be doing the trick, it sometimes causes some |
This was referenced Jul 28, 2020
Merged
bors bot
added a commit
that referenced
this issue
Jul 30, 2020
271: Adjust subscription locking r=ljedrz a=koivunej Fixes the number of critical sections in SubscriptionFuture from 2 to 1, helping out with the hangs in exchange_block which have been recorded in #134. Also backports the `related_subs` cleanup from #264. Co-authored-by: Joonas Koivunen <joonas@equilibrium.co>
These have not been flaky at all recently, nor sure is that because of GHA architecture or #307 for example. Lets reopen we get more of these. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Running the pubsub tests repeatedly like:
(Your executable name will vary, use
cargo test -- publish_between_two_nodes
to find out)With all logging at level at
trace
this does produce at least two kinds of failures. Seen errors:[2020-04-02T09:41:35Z DEBUG libp2p_tcp] Dropped TCP connection to undeterminate peer
when the 10s timeout started[2020-04-02T09:41:08Z DEBUG libp2p_secio] error during secio handshake IoError(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })
even after sending pubsub rpc's back and forth but before the 10s timeout startedIn a highly unscientific "lets run these in multiple shells until they fail, then inspect reason" both seem to happen as often. I think I'll just reference this comment in a tracking issue, this might already be better handled with the just released 0.17 of libp2p. Being resilient on disconnects in the test is one possibility but I am used to running tests in Linux and Windows over loopback without much of an issue. Then again, my systems have probably never been as loaded as any public CI server.
By running other tests than the
pubsub_between_two_nodes
they can also fail but I didn't recover any information on those as 5 tests were executed in parallel and only one of them turned logging on so let it be a small data point.Originally posted by @koivunej in #133 (comment)
The text was updated successfully, but these errors were encountered: