Feature/fix cred chain

[nkupton]

Proposal to fix issues like:
#7
hashicorp/terraform-provider-aws#5018

The recent introduction of session-derived credentials from the AWS SDK was a huge improvement to searching the supported methods of loading credentials. However that code path was only being exercised on error, after validation of the original credential chain. In ECS and EC2 environments, this would never produce an error, so long as the metadata service returned its default credentials. The end result was to ignore the contents of any ~/.aws/config file, which is an important method of introducing other options (assume_role, etc). The root cause is also well documented in #7

To fix, I have moved the ECS and EC2 metadata credentials fetching to the end of the validation chain. This allows the proper use of credentials_source = EcsContainer or EC2InstanceMetadata in earlier paths for loading credentials, and eliminates need to use the skip_metadata_api_check workaround when in EC2.

I've also included a fix for #19

Fix order for credentials chain, building upon Dirk Avery's work.

* Credential chain validation will now try, in order:
    - static variables
    - environment variables
    - shared credentials file
    - AWS SDK session-derived credentials (including shared config file)
    - ECS and EC2 metadata agents
* Previous behavior ignored the config file when running in ECS/EC2,
  because those credentials would prematurely satisfy the contract.
  This prevented the use of config file options, in particular
  credentials_source = EcsContainer or EC2InstanceMetadata

Add missing profile #19

[hashicorp-cla]

All committers have signed the CLA.

[ian-axelrod]

How's the progress on this getting merged in? I'm trying to launch a terraform job via cci using fargate, and I am running into this issue.

[nkupton]

@ian-axelrod I've been using this workaround in the interim: first aws sts assume-role to get a set of temporary credentials provided by the metadata service, use those as the source for any profiles in ~/.aws/config, and then break the metadata fetching via one of these environment variables before calling Terraform:
For EC2: AWS_EC2_METADATA_DISABLED=true
For ECS: AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=

[ian-axelrod]

Yup, that is what I ended up doing after I posted the question. It works fine, but is of course not ideal :). Cheers!

[TimJones]

My setup is really hurting due to #19 , is there any work-around for that? Can that fix be fast-tracked at all? Anything we can do to at all? This fix has been sitting for 2 months already...

[dabbeg]

This is currently affecting us too, would be awesome to get this in ASAP. Can we have some eyes on this? @radeksimko @aeschright

[bflad]

Thank you for submitting this, @nkupton! Please note that I will be looking to pull this in after #32, hopefully today or tomorrow, since that introduces the necessary unit testing that verifies the bug. 👍

This in a potentially breaking change for a lot of practitioners who may be dependent on the (unfortunate) current preference of EC2 Metadata over ECS Credentials, so it will only be included in Terraform AWS Provider v3.0.0 and Terraform S3 Backend (bundled with Terraform CLI) v0.13.0, both of which are releasing in the next few weeks.

[bflad]

Looks good and fixes so many issues. Thanks so much, @nkupton 🚀