rate-limit brupop agent calls to the Bottlerocket update API #496
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gthao313
approved these changes
Aug 7, 2023
| }, | ||
| ); | ||
| } | ||
| match bottlerocket_shadows |
ecpullen
approved these changes
Aug 8, 2023
agent/src/apiclient.rs
Outdated
| pub(super) async fn invoke_apiclient( | ||
| args: Vec<String>, | ||
| rate_limiter: Option<&SimpleRateLimiter>, | ||
| ) -> Result<Output> { | ||
| let mut attempts: i8 = 0; | ||
| // Retry up to 5 times in case the Update API is busy; Waiting 10 seconds between each attempt. | ||
| while attempts < MAX_ATTEMPTS { |
There was a problem hiding this comment.
Is there a reason not to use Retry::spawn?
There was a problem hiding this comment.
I suppose not. Looking at this, I also wonder if we should be retrying any error with the API (e.g. what if the apiserver is being restarted).
This hides away the details of the Bottlerocket API in a private module, then renames the public functions to be more descriptive. This commit also `cargo fmt`s the agent crate.
|
ecpullen
approved these changes
Aug 8, 2023
This was referenced Aug 8, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue number:
#483
Description of changes:
This PR is probably best reviewed commit-by-commit. It appears large, but most of the bigger changes are just simple refactors.
There were reports in #483 of Brupop being unable to update hosts, entering the agent's crash loop. Sometimes the updates would end up succeeding, but sometimes they would become stuck.
Upon inspection of the brupop logs and systemd journal, it seemed like even under conditions where brupop succeeded its updates, it would often be stalled by the apiserver due to the updates lock being held. Brupop would apparently starve itself of this lock by refreshing updates too often and calling the API too quickly.
This change adds ratelimiting to the update-related API calls. It also refactors a few portions of the agent.
Testing done:
Terms of contribution:
By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.