backend/s3: SSO Failures

[SuperP4rks]

Terraform Version

1.6.4

Terraform Configuration Files

N/A

Debug Output

N/A

Expected Behavior

Backend to Authenticate

Actual Behavior

cannot obtain credentials

Steps to Reproduce

Terraform Init

Additional Context

Hi Terraform Team,

Upgraded to 1.6.4 version of terraform, when running an init, presented with No valid credentials. When downgrading back to 1.6.3 it works fine.

We do use aws sso for authentication. I have a feeling the latest bug was introduced in the below.

There is a chance this is not a bug and that there is just no documentation that can be found to identify the endpoints.sso, but i do question what the value is of this parameter when this is normally setup in the aws cli v2.

backend/s3: Add the parameter endpoints.sso to allow overriding the AWS SSO API endpoint. (#34195)

│ Error: No valid credential sources found
│ 
│ Please see https://www.terraform.io/docs/language/settings/backends/s3.html
│ for more information about providing credentials.
│ 
│ Error: failed to refresh cached credentials, the SSO session has expired or is invalid: failed to read cached SSO token file

References

No response

[crw]

Thanks for the report, I've notified the team.

[jar-b]

@jordancparker - Could you share the s3 backend configuration block?

[SuperP4rks]

Hi @jar-b,

I have these in a tfbackend file, which i inject in with a simple. As mentioned all the authentication pieces are managed in awscli. I export the profile as an environment variable. So when the init is ran it identified the profile and the terraform version.

But it seems to be looking for a cached file that does not exist.

versions.tf

backend "s3" {}

tfbackend

region="insert"
bucket="insert"
key="insert"
dynamodb_table="insert"
assume_role={
  role_arn="insert"
}

[gdavison]

Hi @jordancparker, could you run terraform init with debug logging enabled, please? Set the environment TF_LOG=DEBUG (See https://developer.hashicorp.com/terraform/internals/debugging for more information)

The parameter endpoints.sso allows overriding the AWS SSO API endpoint, and unless it's been set in your configuration, it shouldn't affect you. Most users don't need to use it, but if use_fips_endpoint is set, the AWS SDK returns an invalid endpoint, so it needs to be overridden to the correct endpoint (See hashicorp/terraform-provider-aws#33952 and aws/aws-sdk-go-v2#2336)

[SuperP4rks]

Hi @gdavison,

So comparing 1.16.3 and 1.16.4, the below two logs are not in 1.16.4. It would seem that its not aware it needs to request its cached credentials.

[DEBUG] backend-s3.aws-base: HTTP Request Sent: aws.operation=GetRoleCredentials aws.service=SSO tf_backend.operation=Configure
[DEBUG] backend-s3.aws-base: HTTP Response Received: aws.operation=GetRoleCredentials aws.service=SSO tf_backend.operation=Configure

Are you not able to recreate this you're side?

[gdavison]

The cached credentials are local to the machine, so it doesn't make the API calls if it can't locate the cached credentials.

Can you please share the logs here?

[SuperP4rks]

Hi @gdavison ,

I cant share further than what i have above due to sensitivity, specially with the logs exporting authorization keys and certificate bundles.

I unfortunately do not agree with the above statement, cached may be the wrong word. But doing a diff on the Debug logs.

1.6.3 - Is sending a HTTP Request to collect the GetRoleCredentials and Receiving them.
1.6.4 - is not even logging this event and jumping directly to installing modules.

[gdavison]

Sorry, when you say

I unfortunately do not agree with the above statement, cached may be the wrong word

which statement do you mean?

In 1.6.4, the error message includes, in part

failed to read cached SSO token file

This error is returned when the AWS SDK is not able to locate the cached SSO credentials on the local filesystem (https://github.com/aws/aws-sdk-go-v2/blob/435199fc01ab47020ab36dab07d8115e20687f73/credentials/ssocreds/sso_cached_token.go#L150). Since it can't find them, it's not making the HTTP request to the SSO API.

In 1.6.3, it is able to find the cached credentials on the local filesystem, so it's making the HTTP request.

So something has changed in how the AWS SDK is resolving the cached credentials file.

[SuperP4rks]

Apologies, I may have misread the response on the caching of credentials.

But I do agree, looking at the commits for this feature some of the versions have been bumped with the dependant packages.

Do we think they have released a patch already, or changed the way in which the awscli config file is setup to interact with the go library?

[gdavison]

In your configuration file, does the sso_start_url parameter have a trailing #? We've found a bug in the parsing code for those files that strips the # character: aws/aws-sdk-go-v2#2369

As a workaround, you may be able to re-run aws sso configure without the trailing #

Alternatively, re-run aws sso configure with a session name. This has the additional benefit that the credentials can be refreshed without manually re-authenticating. https://docs.aws.amazon.com/cli/latest/userguide/sso-configure-profile-token.html

[SuperP4rks]

hi @gdavison,

Just tested removing the #, then re-logging in and thats fixed the issue and works like a dream.

Thanks for supporting and getting us onto the latest version with 1.6.4 !!!

[AlliotTech]

I have encountered the same problem in 1.6.4.(1.6.3 is ok)

vim ~/.aws/config 

Remove the # from sso_start_url can solve this issue.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.