Bug 1759617: pkg/terraform/exec/plugins: Bump Terraform AWS to 2.36.0

[wking]

Also bump the AWS SDK to the latest 1.25.35. Changelogs here and here. Among other things, this brings in a fix for the eventual-consistency race:

level=error msg="Error: \"terraform-20191008145747927600000001\": NoSuchBucket: The specified bucket does not exist"
level=error msg="\tstatus code: 404, request id: BCC3822AF0933B5C, host id: QVhV5kSr/ztRk3204PNQoVKN6jT7LY0Pk2vJHEKBdG5W4Eok9T+JmZXj8M/1XtBDP5fYEC4HcOI="
level=error
level=error msg="  on ../tmp/openshift-install-302978166/bootstrap/main.tf line 1, in resource \"aws_s3_bucket\" \"ignition\":"
level=error msg="   1: resource \"aws_s3_bucket\" \"ignition\" {"

Generated with:

$ cd pkg/terraform/exec/plugins
$ emacs Gopkg.toml  # terraform-provider-aws -> =2.36.0
$ dep ensure
Solving failure: No versions of github.com/aws/aws-sdk-go met constraints:
  v1.19.26: Could not introduce github.com/aws/aws-sdk-go@v1.19.26 due to multiple problematic subpackages:
  Subpackage github.com/aws/aws-sdk-go/service/applicationinsights is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/forecastservice is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/iotevents is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/lakeformation is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/personalize is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/qldb is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/servicequotas is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)
  ...
$ emacs Gopkg.toml  # aws-sdk-go -> =1.25.35, the latest per https://github.com/aws/aws-sdk-go/tags https://github.com/aws/aws-sdk-go/blob/v1.25.35/CHANGELOG.md
$ dep ensure

[openshift-ci-robot]

@wking: This pull request references Bugzilla bug 1759617, which is invalid:

• expected the bug to target the "4.3.0" release, but it targets "4.4.0" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1759617: pkg/terraform/exec/plugins: Bump Terraform AWS to 2.36.0

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[wking]

/bugzilla refresh

[openshift-ci-robot]

@wking: This pull request references Bugzilla bug 1759617, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[wking]

Ranking in AWS-flake land over the past 48h:

$ curl -s 'https://search.svc.ci.openshift.org/search?name=aws&search=level.error+msg.*in+resource&context=0&maxAge=48h' | jq -r '. | to_entries[].value | to_entries[].value[].context[]' | sed -n 's/.*in resource //p' | sort | uniq -c | sort -n
      2 \"aws_ami_copy\" \"main\":"
      2 \"aws_route53_record\" \"api_internal\":"
      5 \"aws_s3_bucket\" \"ignition\":"
      6 \"aws_route53_record\" \"etcd_a_nodes\":"
      9 \"aws_instance\" \"bootstrap\":"
     13 \"aws_instance\" \"master\":"
$ curl -s 'https://search.svc.ci.openshift.org/search?name=aws&search=level.error+msg.*in+resource.*aws_s3_bucket&context=3&maxAge=48h' | jq -r '. | to_entries[].value | to_entries[].value[].context[]' | sed -n 's/.*Error: //p' | sort | uniq -c | sort -n
      5 NoSuchBucket: The specified bucket does not exist"

[wking]

Hmm, from images:

# github.com/openshift/installer/pkg/terraform/exec/plugins/vendor/github.com/terraform-providers/terraform-provider-aws/aws
pkg/terraform/exec/plugins/vendor/github.com/terraform-providers/terraform-provider-aws/aws/awserr.go:18:5: undefined: errors.As

Looks like it needs Go 1.13.

[wking]

Looks like everything since Terraform's AWS-provider 2.30.0 needs Go 1.13. Not sure if we can get that bumped at this point, or if it needs to wait for 4.4.

[wking]

Go 1.13 bump: openshift/release#5928

[wking]

Looking through the provider changelog, there are also a number of "Final retries after timeouts..." fixes, e.g. for instances in 2.28.0 (hashicorp/terraform-provider-aws#9879) that may help with some of our other timeouts. Although over the past 48 hours, all of the instance timeouts have been:

$ curl -s 'https://search.svc.ci.openshift.org/search?name=aws&search=level.error+msg.*in+resource.*aws_instance&context=3&maxAge=48h' | jq -r '. | to_entries[].value | to_entries[].value[].context[]' | sed -n 's/.*Error: //p' | sort | uniq -c | sort -n
     22 Internal error on launch"

[sdodson]

/hold
The scope of this change and the requisite bump to Go 1.13 leads me to believe we should wait until master branches for 4.4 then consider a backport after it's soaked for a bit.

[LorbusChris]

This looks like a good candidate for folding into the Go Modules migration that I'm working on over at #2745, objections?

[openshift-ci-robot]

@wking: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sdodson]

/hold cancel
Master is of course open for 4.4, so have at it.

[openshift-ci-robot]

@wking: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws	2390bed	link	/test e2e-aws
ci/prow/images	2390bed	link	/test images
ci/prow/e2e-libvirt	2390bed	link	/test e2e-libvirt
ci/prow/e2e-openstack	2390bed	link	/test e2e-openstack
ci/prow/unit	2390bed	link	/test unit
ci/prow/e2e-aws-scaleup-rhel7	2390bed	link	/test e2e-aws-scaleup-rhel7
ci/prow/e2e-aws-upgrade	2390bed	link	/test e2e-aws-upgrade
ci/prow/e2e-aws-fips	2390bed	link	/test e2e-aws-fips
ci/prow/govet	2390bed	link	/test govet
ci/prow/tf-lint	2390bed	link	/test tf-lint
ci/prow/shellcheck	2390bed	link	/test shellcheck
ci/prow/yaml-lint	2390bed	link	/test yaml-lint

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[sdodson]

/close
This will be part of #2745

[openshift-ci-robot]

@sdodson: Closed this PR.

In response to this:

/close
This will be part of #2745

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.