Issues with static stability / expiration in IMDS client

[rcoh]

(created from customer comment)

We are seeing some errors in production following the addition of the logic here

    fn maybe_extend_expiration(&self, expiration: SystemTime) -> SystemTime {
        // [ ... ]
        // calculate credentials' refresh offset with jitter
        let refresh_offset =
            CREDENTIAL_EXPIRATION_INTERVAL + Duration::from_secs(rng.u64(120..=600));
        let new_expiry = self.time_source.now() + refresh_offset;

        if new_expiry < expiration {
            return expiration;
        }

        tracing::warn!(
            "Attempting credential expiration extension due to a credential service availability issue. \
            A refresh of these credentials will be attempted again within the next {:.2} minutes.",
            refresh_offset.as_secs_f64() / 60.0,
        );

        new_expiry
    }

The error we are seeing happening infrequently and when it happens, it lasts for 10/15mins.

Error { code: \"ExpiredTokenException\", message: \"The security token included in the request is expired\", aws_request_id: \"U9LKTDURCE8893LKBN5R1Q4ASFVV4KQNSO5AEMVJF66Q9ASUAAJG\" }

In our setup the credentials are coming from kube2iam and the expiry is set to maximum 30 mins. We are wondering if this is conflicting with any of the assumptions made here? Of course this could entrely be a bug in kube2iam but I wanted to ask here first since this is an ongoing effort to change how the SDK threats expired credentials.

Looking at the notes in #2117

Per internal AWS guidelines, SDKs must not "fail fast" and attempt to refresh expired IMDS credentials without first receiving a service exception. This is because a service may begin accepting expired credentials due to a credential service availability issue.

Is it possible the sdk has been updated to allow the usage of expired tokens without updating the upstream services to relax the validation?

Originally posted by @mustakimali in #2258 (comment)