Does not load root certificates from System Keychain on Mac OS X El Capitan

[mattgartman]

Terraform Version

Terraform v0.8.5 (latest available at terraform.io)

Affected Resource(s)

aws provider. Possibly all providers that would need to access the internet through a proxy that breaks SSL (MiTM/DLP)

Terraform Configuration Files

provider "aws {
  region = "us-east-1"
}

Debug Output

• RequestError: send request failed
  caused by: Post https://sts.amazonaws.com/: x509: certificate signed by unknown authority

Panic Output

n/a

Expected Behavior

Terraform plan/apply should execute successfully behind a proxy that breaks SSL (MiTM/DLP)

Actual Behavior

Terraform error's out not able to validate x509 certificate that is loaded in the "System" Keychain. The latest versions of MacOS no longer allow user specified Root CA's to be added to the "System Roots" keychain. They are now loaded into the "System" keychain.

Steps to Reproduce

Running MacOS "El Capatian", configure a system to use a proxy that breaks SSL and provides a custom cert signed by a custom Root Certificate Authority. The custom Root Certificate Authority's certificated would be loaded in the "System" keychain.

1. terraform plan
2. terraform apply

Important Factoids

• Mac OS X El Capitan (Version 10.11.5)
• System configured Proxy (via Network settings and via http_proxy/https_proxy ENV variables)
• Proxy breaks SSL and presents a custom certificate for websites to allow a Data Loss Prevention (DLP) system to inspect traffic.
• The certificate provided back to the client is signed by an internal PKI that is configured as a trusted root CA configured in the "System" keychain. Recent releases of OS X do not allow custom root certificate authorities to be configured in the "System Roots" keychain, which is what the standard GO libraries seem to load.

References

https://github.com/hashicorp/go-rootcerts
golang/go#14514

Workaround

Setting the "insecure" flag in the 'aws' provider will allow Terraform to work in this configuration. It is not desirable to use the insecure flag however.

provider "aws {
  region = "us-east-1"
  insecure = true
}

[pmcatominey]

Hi @mattgartman, this is an issue I have also experienced working on the azurerm provider!

You are correct in noting that the Go standard library only loads the System Roots keychain, the linked Go issue was resolved for cgo builds only unfortunately. There is however golang/go#16532 which links to a couple of CLs:

https://golang.org/cl/36941
https://golang.org/cl/36942

Hopefully this will lead to a fix in an upcoming Go release, in the meantime I'm not sure if it would be practical/ideal to fix in each providers code.

[mitchellh]

We consider this a Go issue and as @pmcatominey said we're waiting on some upstream Go issues to help resolve this. There are definitely workarounds we could implement but we've never found one we've been happy with. If you have any suggestions I'm open to it. Sorry!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.