Support s3 data repository in dynamic provision

[Jeffwan]

Is this a bug fix or adding new feature?
/feature

Fixes: #16

What is this PR about? / Why do we need it?
Enable S3 data repository integration when creating FSx for Lustre

What testing is done?
UT and manual testing.

Chunk size is not in this PR yet. I explained reason in #16. If there's use case, I will file another PR to support it.

[Jeffwan]

Here's the controller logs, I notice there's a error message but seems not a fatal error. Do you know the reason it cancel request? Looks like FSx has been created at that time.

I0308 00:35:44.554646       1 controller.go:125] ControllerGetCapabilities: called with args &csi.ControllerGetCapabilitiesRequest{XXX_NoUnkeyedLiteral:struct {}{}, XXX_unrecognized:[]uint8(nil), XXX_sizecache:0}
I0308 00:35:44.558819       1 controller.go:39] CreateVolume: called with args &csi.CreateVolumeRequest{Name:"pvc-1c013d0a-413a-11e9-9659-0206ced0c378", CapacityRange:(*csi.CapacityRange)(0xc00018d6b0), VolumeCapabilities:[]*csi.VolumeCapability{(*csi.VolumeCapability)(0xc00023c340)}, Parameters:map[string]string{"s3OutputPath":"s3://dl-benchmark-result/export", "securityGroupIds":"sg-0afdb18c8af898a78", "subnetId":"subnet-0cd89fc1c1bc6ef51", "s3ImportPath":"s3://dl-benchmark-result"}, ControllerCreateSecrets:map[string]string(nil), VolumeContentSource:(*csi.VolumeContentSource)(nil), AccessibilityRequirements:(*csi.TopologyRequirement)(nil), XXX_NoUnkeyedLiteral:struct {}{}, XXX_unrecognized:[]uint8(nil), XXX_sizecache:0}
I0308 00:36:01.233133       1 cloud.go:213] WaitForFileSystemAvailable filesystem status is: CREATING
I0308 00:36:16.229863       1 cloud.go:213] WaitForFileSystemAvailable filesystem status is: CREATING
I0308 00:36:31.249703       1 cloud.go:213] WaitForFileSystemAvailable filesystem status is: CREATING
I0308 00:36:46.246510       1 cloud.go:213] WaitForFileSystemAvailable filesystem status is: CREATING
I0308 00:37:01.240796       1 cloud.go:213] WaitForFileSystemAvailable filesystem status is: CREATING
I0308 00:37:16.236066       1 cloud.go:213] WaitForFileSystemAvailable filesystem status is: CREATING
I0308 00:37:31.231999       1 cloud.go:213] WaitForFileSystemAvailable filesystem status is: CREATING
I0308 00:37:46.292670       1 cloud.go:213] WaitForFileSystemAvailable filesystem status is: CREATING
I0308 00:38:01.237359       1 cloud.go:213] WaitForFileSystemAvailable filesystem status is: CREATING
I0308 00:38:16.229015       1 cloud.go:213] WaitForFileSystemAvailable filesystem status is: CREATING
I0308 00:38:31.234086       1 cloud.go:213] WaitForFileSystemAvailable filesystem status is: CREATING
I0308 00:38:46.246309       1 cloud.go:213] WaitForFileSystemAvailable filesystem status is: CREATING
I0308 00:39:01.228134       1 cloud.go:213] WaitForFileSystemAvailable filesystem status is: CREATING
I0308 00:39:16.236781       1 cloud.go:213] WaitForFileSystemAvailable filesystem status is: CREATING
I0308 00:39:31.227588       1 cloud.go:213] WaitForFileSystemAvailable filesystem status is: CREATING
I0308 00:39:46.234328       1 cloud.go:213] WaitForFileSystemAvailable filesystem status is: CREATING
I0308 00:40:01.229483       1 cloud.go:213] WaitForFileSystemAvailable filesystem status is: CREATING
I0308 00:40:16.254913       1 cloud.go:213] WaitForFileSystemAvailable filesystem status is: CREATING
I0308 00:40:31.227331       1 cloud.go:213] WaitForFileSystemAvailable filesystem status is: CREATING
E0308 00:40:46.200966       1 driver.go:88] GRPC error: rpc error: code = Internal desc = Filesystem is not ready: RequestCanceled: request context canceled
caused by: context canceled
I0308 00:40:49.559957       1 controller.go:39] CreateVolume: called with args &csi.CreateVolumeRequest{Name:"pvc-1c013d0a-413a-11e9-9659-0206ced0c378", CapacityRange:(*csi.CapacityRange)(0xc0001d4420), VolumeCapabilities:[]*csi.VolumeCapability{(*csi.VolumeCapability)(0xc00023c340)}, Parameters:map[string]string{"s3ImportPath":"s3://dl-benchmark-result", "s3OutputPath":"s3://dl-benchmark-result/export", "securityGroupIds":"sg-0afdb18c8af898a78", "subnetId":"subnet-0cd89fc1c1bc6ef51"}, ControllerCreateSecrets:map[string]string(nil), VolumeContentSource:(*csi.VolumeContentSource)(nil), AccessibilityRequirements:(*csi.TopologyRequirement)(nil), XXX_NoUnkeyedLiteral:struct {}{}, XXX_unrecognized:[]uint8(nil), XXX_sizecache:0}
I0308 00:41:04.797710       1 cloud.go:213] WaitForFileSystemAvailable filesystem status is: CREATING
I0308 00:41:19.793207       1 cloud.go:213] WaitForFileSystemAvailable filesystem status is: CREATING
I0308 00:41:34.787117       1 cloud.go:213] WaitForFileSystemAvailable filesystem status is: CREATING
I0308 00:41:49.785114       1 cloud.go:213] WaitForFileSystemAvailable filesystem status is: AVAILABLE

[Jeffwan]

/hold

[Jeffwan]

/cc @leakingtapan

[Jeffwan]

I just use s3 as prefix and seems FSx will support different backend later on.

[leakingtapan]

Here's the controller logs, I notice there's a error message but seems not a fatal error.

It is because external provisioner timed out on CreateVolume . This error is okay because of the later retry and the implementation is idempotent.

[Jeffwan]

New changes address feedbacks and also add validation on importPath and exportPath. There's few edge cases like

• must be same bucket
• s3ExportPath can not be used stand alone
• add notes in documentation

[Jeffwan]

@leakingtapan Could you help check build error message? Seems it's error on dependencies?

[leakingtapan]

go: golang.org/x/net@v0.0.0-20180906233101-161cd47e91fd: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /home/travis/gopath/pkg/mod/cache/vcs/4a22365141bc4eea5d5ac4a1395e653f2669485db75ef119e7bbec8e19b12a21: exit status 128:
	fatal: remote error: Internal Server Error
	fatal: The remote end hung up unexpectedly

I've seem similar errors a lot recently. Mostly recoverable by retry.

[Jeffwan]

Delete validation logic and just use original error msg in pvc events

  ----       ------              ----  ----                                                                       -------
  Warning    ProvisioningFailed  23s   fsx.csi.aws.com_fsx-csi-controller-0_9362c60f-4138-11e9-b2fd-621af0dcfad3  failed to provision volume with StorageClass "fsx-sc-single": rpc error: code = Internal desc = Could not create volume "pvc-41d3e097-41eb-11e9-ba13-0ef78fa15d86": CreateFileSystem failed: BadRequest: Export path provided without specifying any import path.
             status code: 400, request id: fcc84abc-519d-4c0b-bfa7-2d4fde7f34ce
  Warning    ProvisioningFailed  22s  fsx.csi.aws.com_fsx-csi-controller-0_9362c60f-4138-11e9-b2fd-621af0dcfad3  failed to provision volume with StorageClass "fsx-sc-single": rpc error: code = Internal desc = Could not create volume "pvc-41d3e097-41eb-11e9-ba13-0ef78fa15d86": CreateFileSystem failed: BadRequest: Export path provided without specifying any import path.
             status code: 400, request id: 9c3b5830-b635-4bc1-9bd3-5ce1cc5c56f1
  Normal     Provisioning        8s (x3 over 24s)  fsx.csi.aws.com_fsx-csi-controller-0_9362c60f-4138-11e9-b2fd-621af0dcfad3  External provisioner is provisioning volume for claim "default/fsx-claim-single"
  Warning    ProvisioningFailed  7s                fsx.csi.aws.com_fsx-csi-controller-0_9362c60f-4138-11e9-b2fd-621af0dcfad3  failed to provision volume with StorageClass "fsx-sc-single": rpc error: code = Internal desc = Could not create volume "pvc-41d3e097-41eb-11e9-ba13-0ef78fa15d86": CreateFileSystem failed: BadRequest: Export path provided without specifying any import path.
             status code: 400, request id: d4920abc-0d03-4cd1-ae34-6f0fc8bb6125
  Normal     ExternalProvisioning  2s (x5 over 24s)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "fsx.csi.aws.com" or manually created by system administrator
Mounted By:  <none>
➜  dynamic_provisioning_s

[Jeffwan]

@leakingtapan Please have another look.