Use the initial peer success count for the fanout limit #2182
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I've put this PR in draft until #2181 merges to main. |
2 tasks
teor2345
force-pushed
the
initial-handshake-task
branch
from
c1aa7ba to
344d118
Compare
2 tasks
teor2345
force-pushed
the
initial-handshake-task
branch
from
344d118 to
52850e1
Compare
teor2345
force-pushed
the
initial-handshake-success-count
branch
from
9838e62 to
7511d05
Compare
teor2345
force-pushed
the
initial-handshake-task
branch
from
52850e1 to
9328a88
Compare
Previously, Zebra was using the total number of seed peers, even if many of those peers had failed. This change also makes sure that the crawler starts after the first successful peer. (Or after all the initial handshake attempts have finished.)
teor2345
force-pushed
the
initial-handshake-success-count
branch
from
7511d05 to
4c49a2e
Compare
teor2345
commented
May 21, 2021
Comment on lines
+174
to
+180
| info!("waiting for a successful initial peer connection"); | ||
| // it doesn't matter if the sender has been dropped | ||
| let _ = initial_success_count_rx.changed().await; | ||
| info!(initial_successes = ?*initial_success_count_rx.borrow(), | ||
| "asking initial peers for new peers"); | ||
| let _ = candidates | ||
| .update_initial(initial_peer_count_rx.await.expect("value sent before drop")) | ||
| .update_initial(*initial_success_count_rx.borrow()) |
There was a problem hiding this comment.
Use the initial peer success count for the fanout limit, Part 2
- Wait for the first success count update
- Start the crawler using that success count
teor2345
commented
May 21, 2021
Comment on lines
+299
to
+301
| // if the receiver has been dropped, we still want to process | ||
| // the handshakes | ||
| let _ = success_count_tx.send(success_count); |
There was a problem hiding this comment.
Use the initial peer success count for the fanout limit, Part 1a
- Send the success count whenever we have a success
- This also starts the crawler
teor2345
commented
May 21, 2021
jvff
approved these changes
May 21, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Zebra is using the total number of seed peers for the fanout limit.
The limit will be too high if any of the initial peer connections fail.
(Typically, lots of those connections fail.)
Solution
The code in this pull request has:
Review
Whoever reviews #2181 should also review this task. (@dconnolly or @jvff)
Related Issues
Part of #2163
Based on PR #2181
Split off PR #2160
Follow Up Work
Rest of #2160