Endpoint URL seems to be ignored in 0.22.0 release

[diranged]

Slack Thread: https://crossplane.slack.com/archives/CEG3T90A1/p1639700810306500

What happened?

I've been testing the provider-aws package locally with localstack as part of our dev environment using the 0.21.2 release and it's been working great. Upgrading to the 0.22.0 release breaks development though - it seems that the endpoint.url.static path is not being respected in at least some API calls:

Error if you pass in a bogus region:

2021-12-17T00:23:03.792Z	DEBUG	provider-aws	Cannot observe external resource	{"controller": "managed/bucket.s3.aws.crossplane.io", "request": "/test-bucket1-sh7t5-zmtgg", "uid": "dd3904ff-0312-4cc2-9f5f-3c6e76018ec4", "version": "2384231", "external-name": "local-crossplane-test-bucket1", "error": "failed to query Bucket: operation error S3: HeadBucket, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head \"https://local-crossplane-test-bucket1.s3.unknown.amazonaws.com/\": dial tcp: lookup local-crossplane-test-bucket1.s3.unknown.amazonaws.com on 10.96.0.10:53: no such host", "errorVerbose": "operation error S3: HeadBucket, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head \"https://local-crossplane-test-bucket1.s3.unknown.amazonaws.com/\": dial tcp: lookup local-crossplane-test-bucket1.s3.unknown.amazonaws.com on 10.96.0.10:53: no such host\nfailed to query Bucket\ngithub.com/crossplane/provider-aws/pkg/clients.Wrap\n\t/home/runner/work/provider-aws/provider-aws/pkg/clients/aws.go:977\ngithub.com/crossplane/provider-aws/pkg/controller/s3.(*external).Observe\n\t/home/runner/work/provider-aws/provider-aws/pkg/controller/s3/bucket.go:108\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\t/home/runner/work/provider-aws/provider-aws/vendor/github.com/crossplane/crossplane-runtime/pkg/reconciler/managed/reconciler.go:681\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/work/provider-aws/provider-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/work/provider-aws/provider-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/work/provider-aws/provider-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.16.12/x64/src/runtime/asm_arm64.s:1130"}

Error when the region is valid:

2021-12-17T00:36:38.748Z	DEBUG	provider-aws	Cannot observe external resource	{"controller": "managed/bucket.s3.aws.crossplane.io", "request": "/crossplane-test-bucket", "uid": "a0f31fb1-eb3a-40c9-b632-faaf7feeb71c", "version": "2386916", "external-name": "crossplane-test-bucket", "error": "failed to query Bucket: api error Forbidden: Forbidden", "errorVerbose": "api error Forbidden: Forbidden\nfailed to query Bucket\ngithub.com/crossplane/provider-aws/pkg/clients.Wrap\n\t/home/runner/work/provider-aws/provider-aws/pkg/clients/aws.go:965\ngithub.com/crossplane/provider-aws/pkg/controller/s3.(*external).Observe\n\t/home/runner/work/provider-aws/provider-aws/pkg/controller/s3/bucket.go:108\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\t/home/runner/work/provider-aws/provider-aws/vendor/github.com/crossplane/crossplane-runtime/pkg/reconciler/managed/reconciler.go:681\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/work/provider-aws/provider-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/work/provider-aws/provider-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/work/provider-aws/provider-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.16.12/x64/src/runtime/asm_arm64.s:1130"}

How can we reproduce it?

Following https://github.com/crossplane/provider-aws/blob/master/examples/providerconfig/localstack.yaml, create the following resources:

apiVersion: aws.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    secretRef:
      key: credentials
      name: crossplane-aws-localstack
      namespace: crossplane
    source: Secret
  endpoint:
    hostnameImmutable: true
    url:
      static: http://localstack:4566
      type: Static
---
apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
  name: crossplane-aws
spec:
  args:
  - --debug
  env:
  podSecurityContext:
    fsGroup: 2000
  replicas: 1
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: crossplane-aws
spec:
  controllerConfigRef:
    name: crossplane-aws
  ignoreCrossplaneConstraints: false
  package: crossplane/provider-aws:v0.22.0
  packagePullPolicy: IfNotPresent
  revisionActivationPolicy: Automatic
  revisionHistoryLimit: 1
  skipDependencyResolution: false

then create a bucket

apiVersion: s3.aws.crossplane.io/v1beta1
kind: Bucket
metadata:
  name: crossplane-test-bucket
spec:
  writeConnectionSecretToRef:
    name: crossplane-test-bucket
    namespace: crossplane
  forProvider:
    acl: private
    locationConstraint: us-east-1
    publicAccessBlockConfiguration:
      blockPublicPolicy: true
    accelerateConfiguration:
      status: Enabled
    versioningConfiguration:
      status: Enabled
    tagging:
      tagSet:
        - key: test
          value: val1
        - key: secondKey
          value: val2
        - key: key3
          value: val3
    objectLockEnabledForBucket: false
    serverSideEncryptionConfiguration:
      rules:
        - applyServerSideEncryptionByDefault:
            sseAlgorithm: AES256
    corsConfiguration:
      corsRules:
        - allowedMethods:
            - GET
          allowedOrigins:
            - '*'
          allowedHeaders:
            - '*'
          exposeHeaders:
            - x-amz-server-side-encryption
    lifecycleConfiguration:
      rules:
        - status: Enabled
          filter:
            prefix: ola/
          expiration:
            days: 15

then see the failures in the providers logs:

2021-12-17T14:58:26.907Z	DEBUG	provider-aws	Cannot observe external resource	{"controller": "managed/bucket.s3.aws.crossplane.io", "request": "/crossplane-test-bucket", "uid": "c6a42ee6-16df-488e-80c7-24c4f3bafe76", "version": "3483", "external-name": "crossplane-test-bucket", "error": "failed to query Bucket: api error Forbidden: Forbidden", "errorVerbose": "api error Forbidden: Forbidden\nfailed to query Bucket\ngithub.com/crossplane/provider-aws/pkg/clients.Wrap\n\t/home/runner/work/provider-aws/provider-aws/pkg/clients/aws.go:965\ngithub.com/crossplane/provider-aws/pkg/controller/s3.(*external).Observe\n\t/home/runner/work/provider-aws/provider-aws/pkg/controller/s3/bucket.go:108\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\t/home/runner/work/provider-aws/provider-aws/vendor/github.com/crossplane/crossplane-runtime/pkg/reconciler/managed/reconciler.go:681\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/work/provider-aws/provider-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/work/provider-aws/provider-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/work/provider-aws/provider-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.16.12/x64/src/runtime/asm_arm64.s:1130"}

What environment did it happen in?

Crossplane version: 1.5.1
Kubernetes Provider: KIND 1.21.1

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:31:32Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-21T23:06:30Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/arm64"}

[haarchri]

in
provider-aws 0.21.x we used the following implementation: cfg.EndpointResolver https://github.com/crossplane/provider-aws/blob/v0.21.2/pkg/clients/aws.go#L132

provider-aws 0.22.x we used the following implementation: cfg.EndpointResolverWithOptions because cfg.EndpointResolver is deprecated https://github.com/crossplane/provider-aws/blob/v0.22.0/pkg/clients/aws.go#L154

at the moment we not entering the following block https://github.com/crossplane/provider-aws/blob/v0.22.0/pkg/clients/aws.go#L154-L203

think we hit: aws/aws-sdk-go-v2#1513