Fix 429 errors on OVSX requests

[msujew]

What it does

After #13825 we started having a few 429 errors (and adopters as well) when downloading plugins from extension packs.

This change correctly propagates the OVSX rate limit through all layers of the application including the runtime. This can be seen in the ovsx-http-client.ts file changes. This should ensure that users never hit 429 errors from OVSX. If they do, they can use a lower rate limit via a CLI option.

How to test

I've upped the rate limit in the CI - the CI should run successfully.

Also, simply running the theia download:plugins CLI command should run without errors.

Review checklist

• As an author, I have thoroughly tested my changes and carefully followed the review guidelines

Reminder for reviewers

• As a reviewer, I agree to behave in accordance with the review guidelines

[tsmaeder]

Could you explain a bit how #13825 lead to the 429 errors? After all, we did not rate limit before and we didn't after. Did we just increase the number of requests?

[msujew]

Did we just increase the number of requests?

@tsmaeder Yes, exactly :) After #13825, we perform 2 requests for every query request - one with and one without the targetPlatform parameter. So even if the outer loop performs rate limiting, we still hit the ovsx servers twice as hard and therefore get 429 responses in return.

[tsmaeder]

I just ran yarn download:plugins with the PR branch and I'm getting three 429 errors. Either we're still not rate-limiting properly or rate-limiting is broken on the open-vsx side. 🤷‍♂️

[msujew]

@tsmaeder I've added exponential backoff to our requests. I think this should work around the issues of open-vsx reasonably well.

[tsmaeder]

As a last nitpick, can we get a warning if we ran into 429's even if we eventually succeed? IMO it's important info also for open-vsx.