AssumeRoleTokenProviderNotSetError when using assume_role with mfa enabled

[chrishowell]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform version: 0.12.10
Go runtime version: go1.13.1
provider.aws ~> 2.32.0

Affected Resource(s)

• provider "aws"

Terraform Configuration Files

env

AWS_SDK_LOAD_CONFIG=1
AWS_PROFILE=bar-admin

~/.aws/config

[default]
region = eu-west-1
output = json

[profile bar-default]
aws_access_key_id=<key>
aws_secret_access_key=<secret>

[profile bar-admin]
role_arn=arn:aws:iam::<account_id>:role/admin
source_profile=bar-default
mfa_serial=arn:aws:iam::<account_id>:mfa/ch

resource "aws_iam_group" "admins" {
  name = "admins"
}

resource "aws_iam_role" "admin_mfa_role" {
  name = "admin"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "Bool": {
          "aws:MultiFactorAuthPresent": "true"
        }
      }
    }
  ]
}
EOF
}

resource "aws_iam_group_policy_attachment" "admins_assume_role" {
  group      = "${aws_iam_group.admins.name}"
  policy_arn = "${aws_iam_policy.assume_role.arn}"
}

resource "aws_iam_policy" "assume_role" {
  name = "assume_role"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Action": "sts:AssumeRole",
    "Resource": "arn:aws:iam::*:role/*"
  }
}
EOF
}

resource "aws_iam_user" "ch" {
  name = "ch"
}

resource "aws_iam_user_login_profile" "ch_login" {
  user    = "${aws_iam_user.ch.name}"
  pgp_key = "keybase:chrishowell"
}

resource "aws_iam_user_group_membership" "ch_groups" {
  user = "${aws_iam_user.ch.name}"

  groups = [
    "${aws_iam_group.admins.name}"
  ]
}

Debug Output

https://gist.github.com/chrishowell/ddd169c24ba4f0fcaba70a3e2f624a5a

Panic Output

N/A

Expected Behavior

As of terraform-provider-aws_v2.32.0 I believe assume_role with mfa enabled should work.

Actual Behavior

Error: error creating EC2 Metadata session: AssumeRoleTokenProviderNotSetError: assume role with MFA enabled, but AssumeRoleTokenProvider session option not set.

Steps to Reproduce

1. terraform plan

Important Factoids

Running locally on Mac OSX Catalina
Brew install of Terraform
No ~/.aws/configuration file present

References

• Update module hashicorp/aws-sdk-go-base to v0.4.0 #10379
• AWS config role_arn not working #5592

[lachlancooper]

I can reproduce this issue exactly as described.

I noticed there's a difference in behaviour between specifying the profile in an env var (AWS_PROFILE=bar-admin) vs. the provider config directly (profile = "bar-admin"). Using the env var results in the above error creating EC2 Metadata session message, which (since we're running locally, not on EC2) is perhaps a clue to why this is failing. I think this is closely related to hashicorp/aws-sdk-go-base#7

Specifying the profile in provider config still doesn't work though, and I can't see any indication in the trace logs that terraform is even trying to assume a role. The only error I get is:

Error refreshing state: AccessDenied: Access Denied
	status code: 403, request id: 795ACFB7B863F762, host id: XRoHa+G5vU0TKxQkWwyld0p6g/u5Er8T/rd3NjU1rd7odmsFi1JGiZXb3/q42O2DYOCaByEmoFA=

[sean-kang]

Any update on this issue? I am having the exactly same problem. I think that using assume_role with MFA is probably the most common way of access control in big organizations. In my opinion, specifying profile should be enough for Terraform to recognize the use of assume_role with MFA since the configuration is already in ~/.aws/config and ~/.aws/credentials.

[ryan-gerstenkorn-sp]

As a workaround for anyone running into this when using aws-vault (maybe others?) it seems you can remove mfa_serial from the role profile while having it still in the base profile and it will work.

So for example:

[profile base]
credential_process = sh -c 'aws-vault exec base --json 2> $(tty)'
mfa_serial = arn:aws:iam::<redacted>:mfa/<user_name>
region = us-east-1

[profile assumed_role]
role_arn = arn:aws:iam::<redacted>:role/<role_to_assume>
source_profile = base
region = us-east-1

Note that the assumed_role does not have a mfa_serial option set.

And then for your provider something like this should work:

provider "aws" {
  region  = "us-east-1
  profile = "assumed_role"
}

One problem is this role doesn't seem to work with aws-vault now. To fix that you will likely want to make an assumed_role profile and an assumed_role_tf profile to support both.

It almost seems like in this specific case if an error was not raised, this would simply work.

[mikkopiu]

One problem is this role doesn't seem to work with aws-vault now. To fix that you will likely want to make an assumed_role profile and an assumed_role_tf profile to support both.

To add (for aws-vault users) there's also include_profile so you can do this:

[profile base]
credential_process = sh -c 'aws-vault exec base --json 2> $(tty)'
mfa_serial = arn:aws:iam::<account a>:mfa/<user_name>
region = us-east-1

[profile assumed_role]
role_arn = arn:aws:iam::<account b>:role/<role_to_assume>
source_profile = base
region = us-east-1
# For aws-vault only:
include_profile = base

This way aws-vault exec will know about the mfa_serial and prompt for MFA accordingly, and Terraform won't see the mfa_serial in assumed_role and lets aws-vault do the work.

[alexander-wiechert]

this does not work for me. Below is my ~/.aws/credentials file.

[default]
region = eu-central-1
aws_access_key_id = xxx
aws_secret_access_key = xxx
mfa_serial = arn:aws:iam::acount_id_1:mfa/username
credential_process = sh -c 'aws-vault exec default --json 2> $(tty)'

[profile_2]
role_arn = arn:aws:iam::account_id_2:role/SuperAdmin
source_profile = default
include_profile = default`


In terraform I setup the provider like this
`provider "aws" {
  region                  = eu-central-1
  profile                 = "profile_2"
}

If I issue the terraform command I got this:

terraform init --backend-config=backends/mb.tfbackend
Initializing modules...

Initializing the backend...

Error: No valid credential sources found for AWS Provider.
Please see https://terraform.io/docs/providers/aws/index.html for more information on
providing credentials for the AWS Provider

Additionaly I had to issue

aws-vault add default

for another test I tried this
aws --profile profile_2 s3 ls

An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:iam::account_1:user/ai75169 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::account_2:role/SuperAdmin

[IceBear2k]

One problem is this role doesn't seem to work with aws-vault now. To fix that you will likely want to make an assumed_role profile and an assumed_role_tf profile to support both.

To add (for aws-vault users) there's also include_profile so you can do this:

[profile base]
credential_process = sh -c 'aws-vault exec base --json 2> $(tty)'
mfa_serial = arn:aws:iam::<account a>:mfa/<user_name>
region = us-east-1

[profile assumed_role]
role_arn = arn:aws:iam::<account b>:role/<role_to_assume>
source_profile = base
region = us-east-1
# For aws-vault only:
include_profile = base

This way aws-vault exec will know about the mfa_serial and prompt for MFA accordingly, and Terraform won't see the mfa_serial in assumed_role and lets aws-vault do the work.

This workaround works well but isn't flawless. When using VS Code with the Terraform plugin containing Terraform Language Server for example, the language server will perform Terraform commands in the background. While it doesn't output any error message, it tries to call aws-vault which in turn ends up writing a file not a tty in the project directory, since the language server doesn't provide a tty. I have yet to find a solution to that other than adding not a tty to .gitignore which seems like a pretty crappy solution.

[zmingxie]

I ran into a similar issue with the terraform language server, and I change the credential helper process line to something like this which will trigger a dialog box for the MFA prompt:

credential_process=aws-vault exec base --json --prompt=osascript

I do wish this configuration is more flexible because I do prefer TTY input in most other cases.

This workaround works well but isn't flawless. When using VS Code with the Terraform plugin containing Terraform Language Server for example, the language server will perform Terraform commands in the background. While it doesn't output any error message, it tries to call aws-vault which in turn ends up writing a file not a tty in the project directory, since the language server doesn't provide a tty. I have yet to find a solution to that other than adding not a tty to .gitignore which seems like a pretty crappy solution.

[IceBear2k]

I ran into a similar issue with the terraform language server, and I change the credential helper process line to something like this which will trigger a dialog box for the MFA prompt:

credential_process=aws-vault exec base --json --prompt=osascript

I do wish this configuration is more flexible because I do prefer TTY input in most other cases.

This workaround works well but isn't flawless. When using VS Code with the Terraform plugin containing Terraform Language Server for example, the language server will perform Terraform commands in the background. While it doesn't output any error message, it tries to call aws-vault which in turn ends up writing a file not a tty in the project directory, since the language server doesn't provide a tty. I have yet to find a solution to that other than adding not a tty to .gitignore which seems like a pretty crappy solution.

I'm on Ubuntu and I believe osascript is macOS specific. So, it doesn't sound like a solution I could use.

Anyway, this works for me but I'm not proud of it:

credential_process=sh -c "if [ $(echo $(tty) | cut -c 1-5) = '/dev/' ]; then aws-vault exec base --json 2> $(tty); else return 0; fi"

Works as before, minus not a tty files created by Terraform Language Server.

[bertrandmartel]

without using aws-vault, I was able to make it work with credential_process to generate the credentials with a local script and cache the tokens in a new profile tf_temp

The script would:

• check if the token is still valid for the profile tf_temp
• if token is valid, extract the token from existing config using aws configure get xxx --profile tf_temp
• if token is not valid, prompt use to enter mfa token
• generate the session token with aws assume-role --token-code xxxx ... --profile your_profile
• set the temporary profile token tf_temp using aws configure set xxx --profile tf_temp

checkout this gist
stackoverflow post

[shearn89]

This still appears to be an issue - I have a fairly simple template but recently started using a role in the account and can now no longer deploy...

[crigertg]

I've encountered this issue too. Using workarounds always increases the complexity and should not be required. I think it's a common way to use 2FA when assuming a role. 😢

[matthewmrichter]

In our case, quick and dirty fix is:

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "< 4.0.0"
    }

Using > 4.0.0 we get Error: error configuring Terraform AWS Provider: loading configuration: assume role with MFA enabled, but AssumeRoleTokenProvider session option not set.

It's frustrating that this doesn't work. aws-vault has a lot of market share for role assumption.

[matthewmrichter]

Note, I finally read https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-4-upgrade#changes-to-authentication and it gave me a pretty big clue. Removing the profile attribute from the provider "aws" {} block allows me to run terraform in an assumed AWS role using aws-vault.

[saholman]

I use the following bash function to get around this issue:

function set_aws_creds () {
    unset AWS_ACCESS_KEY_ID
    unset AWS_SECRET_ACCESS_KEY
    unset AWS_SESSION_TOKEN

    ROLE="$( aws.exe configure get role_arn --profile $1 )"
    read AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN <<< $(aws.exe sts assume-role --role-arn $ROLE --role-session-name set-aws-creds-session --output text | awk '/^CREDENTIALS/ {print $2, $4, $5 }')
    export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
    export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
    export AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN
}

This is based on this StackOverflow answer: https://stackoverflow.com/a/53199639.

[mreeves1]

I use https://github.com/broamski/aws-mfa to get around this issue.

I run this script below and then two "dynamic" profiles are created that will last 12 hours.

This uses the main profile "mreeves" to fetch temporary creds for two profiles "mreeves-admin" and "mreeves-power" it saves to your ~/.aws/credentials file. See this PR to sceptre for how someone suggested they fix it so the boto session cache is respected: Sceptre/sceptre#674.

There should be a way to "pre-auth" your creds, enter your mfa and then terraform would simply use boto session cache. That's how the aws cli works.

#! /usr/bin/env bash

# Authenticate assume roles so mfa is cached
# Workaround because terraform also does not handle an already authed MFA profile correctly:
# https://github.com/hashicorp/terraform-provider-aws/issues/10491
# https://stackoverflow.com/questions/52432717/terraform-unable-to-assume-roles-with-mfa-enabled

DURATION="43200" # 12 hours

aws-mfa --device arn:aws:iam::1234567890:mfa/mreeves \
	--duration "$DURATION" \
	--profile mreeves \
        --assume-role arn:aws:iam::1234567890:role/human-admin \
        --long-term-suffix none \
	--short-term-suffix admin

aws-mfa --device arn:aws:iam::1234567890:mfa/mreeves \
	--duration "$DURATION" \
	--profile mreeves \
        --assume-role arn:aws:iam::1234567890:role/human-power \
        --long-term-suffix none \
	--short-term-suffix power

[gdavison]

Hello, everyone. The issue #2420 also addresses the lack of support for MFA tokens in the AWS Provider. So that any future discussion will be in one place, I'm going to close this issue.

See #2420 (comment) for the current status of the issue

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.