Warnings that occur when HTTP2 downloads are throttled are a little bit aggressive

[thoughtpolice]

https://github.com/thoughtpolice/buck2/actions/runs/5418361387/jobs/9850380998

If you check the Build Buck2 with Buck2 step, you'll a duplicate number of rather aggressive annoying — I believe these originate from being too aggressive when downloading packages from https://crates.io, and getting rate limited:

WARN buck2_common::http::retries: Retrying a HTTP error after 2 seconds: HTTP Client error: HTTP: Error sending request: http2 error: stream error received: refused stream before processing any application logic

However, it's actually a bit worse when superconsole is in use in your terminal — here it just uses a more typical log stream that streams once — because it gets spammed a bunch during refresh of the active targets being built. Looks like it happens dozens of times in Windows Terminal for me, at least. I can attach a small asciinema of this but it should be easy to reproduce, I think.

Honestly, I don't think this warning as it stands now is really that important, a little rate limiting is expected if you're doing a fresh build with no cache, I think. You're downloading like 100+ dependent crates pretty quickly in this scenario. The download phase of the crates being extended by a second or two is probably OK if you end up getting a big set of them in batches, it may only add a few seconds to the initial download actions.

I think better logic would be if Buck2:

• Downloads with typical exponential backoff during throttling (I assume this is the case already), and
• Only throws an error after a number of retries results in, say, a backoff window of 60 seconds or more occurs.

60 seconds is arbitrary but seems like it's egregious enough to make that a big orange warning. A hair you'd have to split is whether or not you would only do one warning or just warn on every single action. Currently it looks like the warning occurs for every action that gets throttled, but maybe you only want one warning per HTTP domain; e.g. if two download_file actions to download two separate files from http://foo.example.com get rate limited, only report once for the foo.example.com domain — and if another download_file call to http://bar.example.com also gets rate limited, report that, so you'd only have two warnings total.