Skip to content
This repository has been archived by the owner on Jul 12, 2023. It is now read-only.

Retry and parallelize stats puller #1905

Merged
merged 1 commit into from
Mar 4, 2021
Merged

Conversation

sethvargo
Copy link
Member

@sethvargo sethvargo commented Mar 4, 2021

This updates the stats-puller to retry the HTTP request 3 times. This will make us more resilient to temporary network blips. It also parallelizes pulling (default 5).

Release Note

Add client-side retry logic and parallelize stats puller. The default parallelize is 5, but it can be customized with `STATS_PULLER_MAX_WORKERS`. There is also a **behavior change**. The stats-puller previously always returned success (but logged errors on failure). This changes the puller to return a non-200 response code if there are still failures after all retries have executed.

This updates the stats-puller to retry the HTTP request 3 times. This will make us more resilient to temporary network blips. It also parallelizes pulling (default 5).
@sethvargo sethvargo requested a review from a team as a code owner March 4, 2021 19:07
@googlebot googlebot added the cla: yes Auto: added by CLA bot when all committers have signed a CLA. label Mar 4, 2021
@sethvargo
Copy link
Member Author

@sethvargo sethvargo merged commit 61ed4bb into main Mar 4, 2021
@sethvargo sethvargo deleted the sethvargo/pull_retry_parallel branch March 4, 2021 20:07
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 6, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
cla: yes Auto: added by CLA bot when all committers have signed a CLA.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants