Terraform access to AWS credentials when using profiles

[spanktar]

Terraform Version

0.6.15

Affected Resource(s)

• provider: AWS

Terraform Configuration Files

provider "aws" {
    profile =                 "${var.profile}"
    shared_credentials_file = "~/.aws/credentials"
    region =                  "${var.aws_region}"
}

Expected Behavior

When attempting to switch from using variables to using AWS profiles (credentials file), I've run into an issue. I would expect to be able to access the AWS credentials somehow in Terraform. Formerly, we had a file called keys.tf (.gitignore'd) where we put variables for the AWS credentials, for example:

variable "aws_access_key" { default = "foo" }

Then later we could use it if we need to (and we do need to occasionally), for example in a template file to pass to an instance:

access_key = "${var.aws_access_key}"

When using profiles (as shown at the top), we no longer have access to this. It would be great if we could use profiles, and glean the runtime values out of the provider. Something like this:

${provider.aws.access_key}
or
${aws.access_key.value}
or something

For now we have to abandon using profiles (which sucks because we have multiple AWS accounts we switch to and from very often) because of this.

Thoughts?

[mtougeron]

I've had good success using https://github.com/redredgroovy/terraform-provider-vault in the past. But be sure to read the caveat about security of the credentials for the tfstate file(s).

[crania]

It seem interpolation for a profile does not work. We have a simple config and simply replacing the profile string with a variable "${var.profile}" will throw the error. Is there a reason profile cannot take a variable while the AWS keys can?

Allowing interpolation would be a simple fix for our situation.

[mtougeron]

@crania can you post an example of your code? This is what I use and it hasn't been a problem. I'm running v0.7.7

provider "aws" {
  region = "${var.aws-region}"
  profile = "${var.aws-profile}"
  allowed_account_ids = ["${module.aws.account-id}"]
}

[tobinquadros]

I'm struggling with this as well.

main.tf has:

provider "aws" {
  profile = "${var.aws_profile}"
  region = "${var.aws_region}"
  allowed_account_ids = ["${var.aws_account_id}"]
}

variables.tf has:

variable "aws_profile" {}
variable "aws_region" {}
variable "aws_account_id" {}

terraform.tfvars has:

aws_profile = "myprofile"
aws_region = "us-east-1"
aws_account_id = "xxxxxxxxxxxx"

And it constantly fails to get my state file from s3 because terraform seems to always be using my [default] aws profile. If I set [myprofile] to [default] in the aws credentials/config files it works fine. Unfortunately, I'm currently using terraform on a project that is not in my default profile's account.

Seeing this on both Terraform version 0.7.10 & 0.7.13

[mtougeron]

@tobinquadros The problem is that when you use terraform remote config it uses its own profile variable that is entirely separate from the provider "aws" There is #1964 that discusses allowing setting of the remote storage via a config instead of being a separate command. It is currently slated to be in v0.9.

[tobinquadros]

Thanks for the heads up. I found that issue the other day and added -backend-config="profile=VALUE" to my terraform remote config setup and it's working. Although, it seems I had to run the command twice before it succeeded.

[spanktar]

I just thought of another workaround. Make several different credential files. For example:

~/.aws/foo-credentials
~/.aws/bar-credentials

each one having a [default] stanza.

Then use shared_credentials_file variable to swap. Our remote state setup shell script is static anyway, and we have one for each env, so setting this up once would work.

[richarddowner]

This still seems to be a problem.

I am setting terrafrom remote config like this:
terraform remote config -backend=s3 -backend-config='profile=Developer-test' ...

Which successfully configures and pulls the remote state. However, terraform apply does not work.

I am setting provider like this:

variable "profile" {
	type = "string"
}

provider "aws" {   
	region = "ap-southeast-2"
        profile = "${var.profile}" 
}

Errors with:
Error reloading remote state: AccessDenied: Access Denied status code: 403, request id: 398ECE73BA2846C1

Anyone able to advise? Cheers

[spanktar]

I wrote a wrapper in bash to solve it. Nothing native that I know of yet.

…

On Thu, Feb 23, 2017 at 20:18 Richard Downer ***@***.***> wrote: This still seems to be a problem. I am setting terrafrom remote config like this: terraform remote config -backend=s3 -backend-config='profile=Developer-test' ... Which successfully configures and pulls the remote state. However, terraform apply does not work. I am setting provider like this: `variable "profile" { type = "string" } provider "aws" { region = "ap-southeast-2" profile = "${var.profile}" }` Errors with: Error reloading remote state: AccessDenied: Access Denied status code: 403, request id: 398ECE73BA2846C1 Anyone able to advise? Cheers — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7761 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAks1Dt340z1iGh4V6fX0nZlO4LTTfboks5rfloMgaJpZM4JSP7f> .

[richarddowner]

@spanktar may I ask what your wrapper does?

[spanktar]

Sure. I wrote it at my last job so I'd have to rewrite it as open source, which I'll do. There were two approaches: 1. Put a .tf file in the working directory with the credentials for that env in it and parse 2. Parse the .aws/credentials file Either way you get the keys you need and pass them as tfvars to terraform Pretty basic really. I also did a few other things we liked, for instance, we added the current git hash of the build as an identifier tag to the bastion instance so we could reference which point in the codebase had been used to build. I called it "atmosphere" (a wrapper around a planet like terraforming) and basically provided a bash function named "tf"

…

On Thu, Feb 23, 2017 at 20:31 Richard Downer ***@***.***> wrote: @spanktar <https://github.com/spanktar> may I ask what your wrapper does? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7761 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAks1PM3DkmmZLrAfLYx96uqqlf9nd2Bks5rfl0wgaJpZM4JSP7f> .

[evanstachowiak]

Even better would be able to just set AWS_PROFILE and not have to specify a profile name in the provider. This is how most everything else works when interfacing with aws.

[cornfeedhobo]

Yeah, please don't simplify this to relying on environment variables. That would make working with multiple accounts even more complex.

Please follow the official aws convention and rely on a tiered approach, similar to what is already being done with packer.

Thank you @spanktar for this request :-)

[Gary-Armstrong]

Is this still not working? I tried to have TF use a profile in my ~/.aws/credentials and it would only read in the default.

Also when I removed all profiles except the one I wanted to use (obviously I can't continue working this way), the failure moved to the s3 backend. I am curious if the TF credentials code is unified with the backend credentials code? Or is it a separate mechanism?

[cornfeedhobo]

For those following this: I currently see this as working with >= v0.9.0

provider "aws" {
  region  = "us-east-1"
  profile = "${var.aws_profile}"
}

[Gary-Armstrong]

I will add that I did get it working also, but @cornfeedhobo 's solution did not work for me. I finally got it working via export AWS_PROFILE=whatever and implemented a solution in my .bash_profile. I would have MUCH rather been able to code the AWS profile name into TF.

Might try again later if I get a spare minute.

[cornfeedhobo]

@Gary-Armstrong what terraform version? I am not setting any AWS environment variables

[Gary-Armstrong]

v0.9.2
I'll try it again. Been a long week, and it's only Wednesday. :)

[jonbrouse]

@Gary-Armstrong If possible and present, try removing the .terraform directory from your cwd. I had issues using the profile argument as well and removing the directory allowed me to use different AWS profiles.

[spanktar]

See: hashicorp/terraform-provider-aws#233 (comment)

[MikeRippon]

+1 for deleting the /.terraform directory (for people coming here from google)

I just switched from AWS_ACCESS_KEY & AWS_SECRET_ACCESS_KEY to using ~/.aws/credentials, along with specifying the profile in a --backend-config=abc.tfvars file. terraform init failed to find the credentials until I deleted .terraform

[ghost]

@LittleMikeDev I tried removing .terraform directory but still no luck. I'm on the latest version of terraform 0.11.13

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.