ClientResponseError

[jkingslake]

Looks like the full recipe failed with ClientResponseError. Specifically, a 429 error with the message 'Too Many Requests'.

https://pangeo-forge.org/dashboard/recipe-run/1047?feedstock_id=76

A first attempt to avoid this could be to simply add a short wait to the process_input function.

[cisaacstern]

@jkingslake, thanks for noticing this and opening an issue. This is exactly the way we're hoping to troubleshoot these recipes.

I've taken a closer look and unfortunately #3 will not be able to fix this. What we will need is a solution for pangeo-forge/pangeo-forge-recipes#389, which will limit concurrency at the caching stage. By the time we get to process_input, data has already been moved off of the provider's server and into our cloud cache, so waiting at that stage will not resolve concurrency issues that arise when fetching data from the provider.

I will also make a note on that issue.

[jkingslake]

Thanks @cisaacstern. I suspected this might be the case.

[jkingslake]

@cisaacstern, have you had a chance to take a look at this at all?