Add static stability support to IMDS credentials provider

[ysaito1001]

Motivation and Context

Implements #2117

Description

This PR adds static stability support to the IMDS credentials provider. The goal is to make ImdsCredentialsProvider more robust in providing credentials even when IMDS is unreachable. We do this by letting it hold the last retrieved credentials, and these fallback credentials can be served as a backup.

In addition to the failure scenario above, we also need to consider the case where IMDS may return expired credentials. Static stability support specifies that ImdsCredentialsProvider should extend the credentials expiry by the calculated amount. The implementation of that is defined in ImdsCredentialsProvider::maybe_extend_expiration and references that in botocore.

By implementing the above, the Rust SDK will not fail fast in sending a request with expired credentials and allows the target service to make the ultimate decision as to whether the request sent is valid.

Testing

For the ease of maintaining tests in the future, we'd like to keep them in aws-config. Some of the tests require "SDK can send a request...", but we can reinterpret the test specification and bring it closer to the shore of ImdsCredentialsProvider. From an ImdsCredentialsProvider's perspective, credentials being returned from provide_credentials is equivalent to "SDK can send a request". Furthermore, the Rust SDK, by design, separates credentials providers from credentials caching at the time of writing (01/27/2023); ImdsCredentialsProvider itself does not cache credentials the way LazyCredentialsCache does. Thus, the test specification like "IMDS must only be called once" can be translated to ImdsCredentialsProvider correctly extending expired credentials.

• Expired Credential Test Cases
  • SDK can use IMDS credential provider if first IMDS call returns expired credentials (implemented by expired_credentials_should_be_extended).
  • SDK can send request if expired credentials are available (here's no explicit test added for this as it is implicitly handled by two tests for Refresh Failures).
  • SDK can perform 3 successive requests with expired credentials. IMDS must only be called once (implicitly realized by expired_credentials_should_be_extended).
• Refresh Failures
  • SDK can send a request after a read timeout during a refresh (implemented by read_timeout_during_credentials_refresh_should_yield_last_retrieved_credentials).
  • SDK can send a request after receiving a 500 HTTP response from IMDS during a refresh (implemented by fallback_credentials_should_be_used_when_imds_returns_500_during_credentials_refresh).
• Logging
  • SDK must log a message when expired credentials are used (implemented by expired_credentials_should_be_extended).
• Passed tests in CI

Checklist

• I have updated CHANGELOG.next.toml if I made changes to the AWS SDK, generated SDK code, or SDK runtime crates

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[github-actions]

A new generated diff is ready to view.

• AWS SDK (ignoring whitespace)
• No codegen difference in the Server Test
• No codegen difference in the Server Test Python

A new doc preview is ready to view.

[jdisanti]

LGTM!

[github-actions]

A new generated diff is ready to view.

• AWS SDK (ignoring whitespace)
• No codegen difference in the Server Test
• No codegen difference in the Server Test Python

A new doc preview is ready to view.

[github-actions]

A new generated diff is ready to view.

• AWS SDK (ignoring whitespace)
• No codegen difference in the Server Test
• No codegen difference in the Server Test Python

A new doc preview is ready to view.

[mustakimali]

Hi, we are seeing some errors in production following the addition of the logic here

    fn maybe_extend_expiration(&self, expiration: SystemTime) -> SystemTime {
        // [ ... ]
        // calculate credentials' refresh offset with jitter
        let refresh_offset =
            CREDENTIAL_EXPIRATION_INTERVAL + Duration::from_secs(rng.u64(120..=600));
        let new_expiry = self.time_source.now() + refresh_offset;

        if new_expiry < expiration {
            return expiration;
        }

        tracing::warn!(
            "Attempting credential expiration extension due to a credential service availability issue. \
            A refresh of these credentials will be attempted again within the next {:.2} minutes.",
            refresh_offset.as_secs_f64() / 60.0,
        );

        new_expiry
    }

The error we are seeing happening infrequently and when it happens, it lasts for 10/15mins.

Error { code: \"ExpiredTokenException\", message: \"The security token included in the request is expired\", aws_request_id: \"U9LKTDURCE8893LKBN5R1Q4ASFVV4KQNSO5AEMVJF66Q9ASUAAJG\" }

In our setup the credentials are coming from kube2iam and the expiry is set to maximum 30 mins. We are wondering if this is conflicting with any of the assumptions made here? Of course this could entrely be a bug in kube2iam but I wanted to ask here first since this is an ongoing effort to change how the SDK threats expired credentials.

Looking at the notes in #2117

Per internal AWS guidelines, SDKs must not "fail fast" and attempt to refresh expired IMDS credentials without first receiving a service exception. This is because a service may begin accepting expired credentials due to a credential service availability issue.

Is it possible the sdk has been updated to allow the usage of expired tokens without updating the upstream services to relax the validation?

[rcoh]

It seems like our logic might be incorrect—I don't think the intention was to always increase the expiry on credentials coming from IMDS (as seems to be the case in the diff). We'll dig into it and get a fix out.

Do you mind verifying if you see

        tracing::warn!(
            "Attempting credential expiration extension due to a credential service availability issue. \
            A refresh of these credentials will be attempted again within the next {:.2} minutes.",
            refresh_offset.as_secs_f64() / 60.0,
        );

in your logs? just to confirm that this is the culprit?

[mustakimali]

Yes I see this log line in the affected pods. In fact this is what lead us to this commit.

[ysaito1001]

Thank you for reporting the issue. Opened a PR to address it.

[rcoh]

this should be fixed in the latest release to aws-sdk-rust