S3 backend config role_arn can't be assumed when running dockerized

[antonosmond]

Terraform Version

Terraform v0.9.3

Affected Resource(s)

• backend

Terraform Configuration Files

terraform = {
  required_version = "~> 0.9.3"

  backend "s3" {
    region     = "eu-west-1"
    bucket     = "anto-terraform-state"
    key        = "platform/network.tfstate"
    role_arn   = "arn:aws:iam::000000000000:role/terraform"
  }
}

Debug Output

Initializing the backend...

Error configuring the backend "s3": The role "arn:aws:iam::000000000000:role/terraform" cannot be assumed.

There are a number of possible causes of this - the most common are:
* The credentials used in order to assume the role are invalid
* The credentials do not have appropriate permission to assume the role
* The role ARN is not valid

Please update the configuration in your Terraform files to fix this error
then run this command again.

Expected Behavior

The backend role_arn is assumed and the backend is successfully initialized

Actual Behavior

The backend role_arn can't be assumed

Steps to Reproduce

1. Set AWS credentials ensuring they have access to assume the role e.g.
   export AWS_ACCESS_KEY_ID=**************
export AWS_SECRET_ACCESS_KEY=**************************

2. cd to directory with backend config

3. run terraform init (NOT dockerized) to ensure the credentials are set correctly and can assume the specified role - backend should be initialized successfully.

4. run the same initialization but dockerized, ensuring the same credentials are passed to the container and the backend config in the current directory is available e.g.

docker run -ti --rm \
  --volume "$(pwd):/terraform" \
  --workdir "/terraform" \
  -e "AWS_ACCESS_KEY_ID" \
  -e "AWS_SECRET_ACCESS_KEY" \
  "hashicorp/terraform:0.9.3" \
  init

[antonosmond]

I've also tried just launching the terraform container interactively e.g.

docker run -ti --rm \
  --volume "$(pwd):/terraform" \
  --workdir "/terraform" \
  -e "AWS_ACCESS_KEY_ID" \
  -e "AWS_SECRET_ACCESS_KEY" \
  --entrypoint sh \
  "hashicorp/terraform:0.9.3"

ensuring the backend config is available, setting the credentials and running terraform init and get the same error.

[jabley]

I'm wondering whether there is a time-based aspect to this issue?

I restarted Docker for Mac, rebooted Mac, and the problem was still apparent.

It disappeared the next day.

It intermittently reappears.

[xakraz]

Same issue with Terraform v0.10.8

[ggermis]

getting the same issue even when running non-dockerized

$ terraform version
Terraform v0.11.0

I have my ~/.aws/credentials file setup with several profiles. I use the default profile to assume roles on other accounts

[default]
aws_access_key_id = MY_KEY
aws_secret_access_key = MY_SECRET
region = eu-west-1

From the aws cli it works and I get my temporary credentials:

$ aws --profile default sts assume-role --role-arn arn:aws:iam::111111111111:role/terraform --role-session-name terraform
{
    "Credentials": {
        "AccessKeyId": "TEMP_KEY",
        "SecretAccessKey": "TEMP_SECRET",
        "SessionToken": "SESSION_TOKEN",
        "Expiration": "2017-11-30T10:13:15Z"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "ID:terraform",
        "Arn": "arn:aws:sts::111111111111:assumed-role/terraform/terraform"
    }
}

The only AWS_ variables set in my environment is:

$ env | grep AWS
AWS_PROFILE=default

My terraform backend configuration looks like:

terraform {
  required_version = ">= 0.11.0"
  backend "s3" {
    region          = "eu-west-1"
    bucket          = "terraform"
    key             = "dev/vpc.tfstate"
    encrypt         = "true"
    profile         = "default"
    role_arn        = "arn:aws:iam::111111111111:role/terraform"
    session_name    = "terraform"
  }
}

When trying to run a $ terraform init I get the "role cannot be assumed" error. Trace output for terraform init:

...
2017/11/30 10:15:04 [INFO] Building AWS region structure
2017/11/30 10:15:04 [INFO] Building AWS auth structure
2017/11/30 10:15:04 [INFO] Setting AWS metadata API timeout to 100ms
2017/11/30 10:15:04 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
2017/11/30 10:15:04 [INFO] Attempting to AssumeRole arn:aws:iam::111111111111:role/terraform (SessionName: "terraform", ExternalId: "", Policy: "")
2017/11/30 10:15:04 [INFO] AWS Auth provider used: "SharedCredentialsProvider"
Error initializing new backend:
2017/11/30 10:15:24 [DEBUG] plugin: waiting for all plugin processes to complete...
Error configuring the backend "s3": The role "arn:aws:iam::111111111111:role/terraform" cannot be assumed.

  There are a number of possible causes of this - the most common are:
    * The credentials used in order to assume the role are invalid
    * The credentials do not have appropriate permission to assume the role
    * The role ARN is not valid

Please update the configuration in your Terraform files to fix this error
then run this command again.

My policy does not require MFA

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": [
            "arn:aws:iam::111111111111:role/terraform"
        ]
    }
}

I tried removing the .terraform subdirectory, and re-running $ terraform init but no change..

The weird thing is that it did work yesterday. I have setup and tore down my environment repeatedly the last few days... The code is in version control, and I haven't changed my ~/.aws/credentials and today it just stopped working

I did play around with installing boto3 and boto through pip, but am assuming this won't impact behaviour of terraform since it's a go application

Any insights into this issue would be much appreciated

[ggermis]

it turned out to be a network issue.. the strange thing is that the aws cli did work, while terraform didn't.. guess it's using a different network stack to do DNS resolving or something

[jsthibault]

Hi,

I have the same problem, did you find any solutions ?

it seems like, sometimes it worked and sometimes it's doesn't. It randomly works.

Regards,

[jvanwygerden]

Having the same issue - appears to be intermittent. Is there a known resolution for this issue

[ken5scal]

Hi, I'm experiencing the same issue

[yogeshdass]

It works great with Terraform version v0.13.3. Here is how I am doing it

terraform {
backend "s3" {
bucket = "terraform-states"
region = "ap-south-1"
profile = "terraform"
key = "prod/terraform.tfstate"
role_arn = "arn:aws:iam::037912942455:role/terraform"
session_name = "terraform"
}
}

profile "terraform" user has permission to just assume the terraform role

[ashkv92]

@yogeshdass Setting the profile corresponding to the role defined under .aws/config helped in my case. thanks for the pointer!!

[vurachaitanya]

Please check your docker local time file if AWS Cli might have some time get will also not work...It fixed for me after changing my local time on my VM

[dr-edmond-merkle]

Greetings,

I actually experienced this issue locally outside of docker as well. I had initialized the backend state as an IAM role (Role A) provisioned in the same AWS account as where the resources were deployed (Account A). No role_arn was specified in the terraform s3 backend. There was a role_arn set on the aws provider, Role A, but not on the terraform backend.

Next step was to set a role on the terraform backend, so that an "admin" account (Account B) IAM role (Role B) that had permissions to assume Role A, could execute the terraform module.

I started to experience the error after adding the role_arn to the backend. When enbabling TRACE logging with TF_LOG, I noticed that there were multiple attempts to assume Role A as a part of the terraform execution.

The fix for me was removing the role_arn from the aws provider, and only having the role_arn on the backend. After that, the terraform execution succeeded. Once the terraform execution succeeded, I was able to add the role_arn for Role A back to the aws provider without further issue.

Maybe this might help someone.

[sgpinkus]

I also experienced this issue outside of docker. I can aws sts --role-arn assume-role <s3-role-arn> --role-session-name foo, but terraform plan gives dredded "The role "<s3-role-arn>" cannot be assumed ..." error. I can't rule out an intermittent network glitch as others have mentioned but for me I think the fix was to set session_name = "terraform" in the s3 backend config.This makes some sense if the s3 backend is not setting a default for this value because aws sts assume-role requires you set one.

[joeguaneri]

Going through the logs led me to look at this: hashicorp/aws-sdk-go-base#4. When I compared the issues, they seem to behave similar to me. It looks like it's the credential process WITH assume role that is the issue. If the backend credentials use the default provider, the issue goes away. It looks like there is a built in assumption some place with the default credentials.

[csbain]

Stumbling across this a bit late... but I've encountered the same issue and found this helpful: https://support.hashicorp.com/hc/en-us/articles/1500005343862-Required-Additional-Configuration-When-Using-IMDSv2
Has to do with httpPutResponseHopLimit part of the ec2 instance metadata: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_InstanceMetadataOptionsResponse.html

[gdavison]

Thanks for the additional info, @csbain. That setting is vital when running in a container in AWS. The original error reported looks like they were running locally.

In @antonosmond's original examples, it looks like the value of the environment variables passed to the docker run command is not set. According to Docker documentation, environment variables must be passed as -e VAR=value.

We have made many changes to authentication since this issue was reported, and many more are coming in v1.6. I'm going to close this issue. If you are still encountering this problem, please create a new issue

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.