Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Frequent network timeouts on slow connection #6284

Closed
ehuss opened this issue Nov 8, 2018 · 0 comments · Fixed by #6285
Closed

Frequent network timeouts on slow connection #6284

ehuss opened this issue Nov 8, 2018 · 0 comments · Fixed by #6285

Comments

@ehuss
Copy link
Contributor

ehuss commented Nov 8, 2018

I was testing latest cargo, and had some downloads going on with another machine slowing my connection, and I started getting large numbers of timeouts. So I set up some artificial tests and confirmed that #6130 is causing problems.

To repro:

  1. Build master cargo (anything with Fix timeouts firing while tarballs are extracted  #6130).
  2. Set up a rate limiter on your computer. I set mine to 200kbps. The lower the limit, the more errors you get.
  3. cargo fetch something with a lot of crates (like cargo itself) with an empty cache.
  4. Wait about 30 seconds. It should have downloaded a few crates, and then you should get a sudden flood of timeouts for a large number of crates. If you wait long enough, it should run out of retries and eventually fail.

Turning on http.multiplexing makes it worse, but it is not necessary. You can also make it fail faster with http.timeout of 10 or even 5.
@alexcrichton alexcrichton mentioned this issue Nov 8, 2018
Merged
alexcrichton added a commit to alexcrichton/cargo that referenced this issue Nov 8, 2018
@alexcrichton
Timeout batch downloads, not each download
4e1e3f7 
This commit switches the timeout logic implemented in rust-lang#6130 to timeout
an entire batch of downloads instead of each download individually.
Previously if *any* pending download didn't receive data in 30s we would
time out, or if *any* pending download didn't receive 10 bytes in 30s we
would time out. On very slow network connections this is highly likely
to happen as a trickle of incoming bytes may not be spread equally
amongst all connections, and not all connections may actually be active
at any one point in time.

The fix is to instead apply timeout logic for an entire batch of
downloads. Only if zero total data isn't received in the timeout window
do we time out. Or in other words, if any data for any download is
receive we consider it as not being timed out. Similarly any progress on
any download counts as progress towards our speed limit.

Closes rust-lang#6284
bors added a commit that referenced this issue Nov 9, 2018
@bors
Auto merge of #6285 - alexcrichton:more-timeouts, r=ehuss
5f448d5 
Timeout batch downloads, not each download

This commit switches the timeout logic implemented in #6130 to timeout
an entire batch of downloads instead of each download individually.
Previously if *any* pending download didn't receive data in 30s we would
time out, or if *any* pending download didn't receive 10 bytes in 30s we
would time out. On very slow network connections this is highly likely
to happen as a trickle of incoming bytes may not be spread equally
amongst all connections, and not all connections may actually be active
at any one point in time.

The fix is to instead apply timeout logic for an entire batch of
downloads. Only if zero total data isn't received in the timeout window
do we time out. Or in other words, if any data for any download is
receive we consider it as not being timed out. Similarly any progress on
any download counts as progress towards our speed limit.

Closes #6284
@bors bors closed this as completed in #6285 Nov 9, 2018
alexcrichton added a commit to alexcrichton/cargo that referenced this issue Nov 9, 2018
@alexcrichton
Timeout batch downloads, not each download
9d41017 
This commit switches the timeout logic implemented in rust-lang#6130 to timeout
an entire batch of downloads instead of each download individually.
Previously if *any* pending download didn't receive data in 30s we would
time out, or if *any* pending download didn't receive 10 bytes in 30s we
would time out. On very slow network connections this is highly likely
to happen as a trickle of incoming bytes may not be spread equally
amongst all connections, and not all connections may actually be active
at any one point in time.

The fix is to instead apply timeout logic for an entire batch of
downloads. Only if zero total data isn't received in the timeout window
do we time out. Or in other words, if any data for any download is
receive we consider it as not being timed out. Similarly any progress on
any download counts as progress towards our speed limit.

Closes rust-lang#6284
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Timeout batch downloads, not each download alexcrichton/cargo
1 participant
@ehuss