Google Cloud Storage's S3 compatibility mode doesn't function properly for multipart uploads [Update: Google fixed in June 2023]

[CorruptMonkey]

Steps to reproduce

1. Configure external storage for Google Cloud Storage as you would for Amazon s3
2. Try to upload to GCS.

Expected behaviour

The file should upload

Actual behaviour

Directories can be created fine, and show up on the bucket, however files will not upload to it and result in an error in the browser.

An exception occurred while initiating a multipart upload: Error executing "CreateMultipartUpload" on "https://mystorage.storage.googleapis.com/polly.jpeg?uploads"; AWS HTTP error: Client error: `POST https://mystorage.storage.googleapis.com/polly.jpeg?uploads` resulted in a `400 Bad Request` response: <?xml version='1.0' encoding='UTF-8'?><Error><Code>InvalidArgument</Code><Message>Invalid argument.</Message><Details>PO (truncated...) InvalidArgument (client): Invalid argument. - <?xml version='1.0' encoding='UTF-8'?><Error><Code>InvalidArgument</Code><Message>Invalid argument.</Message><Details>POST object expects Content-Type multipart/form-data</Details></Error>

Server configuration

Operating system: Ubuntu 18.04.1 LTS

Web server: nginx

Database: MariaDB

PHP version: 7.2

Nextcloud version: 15.0.3

Updated from an older Nextcloud/ownCloud or fresh install:
Updated from 15.0.2

Where did you install Nextcloud from:
Download from nextcloud website

Signing status:

Signing status

No errors have been found.

List of activated apps:

Activated apps

Accessibility 1.1.0, Activity 2.8.2, Announcement center 3.4.1, AppOrder 0.6.0, Calendar 1.6.4, Collaborative tags 1.5.0, Comments 1.5.0, Contacts 3.0.3, Deck 0.5.2, Deleted files 1.5.0, DropIt 0.1.3, External storage support 1.6.0, External user support 0.5.0, Federation 1.5.0, File sharing 1.7.0, Files From Mail 0.2.0, Files Right Click 0.11.0, Gallery 18.2.0, Log Reader 2.0.0, Mail 0.11.0, Monitoring 1.5.0, Music 0.9.3, News 13.0.3, Nextcloud announcements 1.4.0, Notifications 2.3.0, Notifications for calendar event updates 0.3.2, Password policy 1.5.0, PDF viewer 1.4.0, Quota warning 1.4.0, Radio 0.6.4, Share by mail 1.5.0, Support 1.0.0, Text editor 2.7.0, Theming 1.6.0, Update notification 1.5.0, Usage survey 1.3.0, Versions 1.8.0, Video player 1.4.0

Are you using external storage, if yes which one: Amazon s3

Are you using encryption: yes on s3 bucket

Are you using an external user-backend, if yes which one: IMAP

Client configuration

Browser: Firefox

Operating system: Ubuntu 18.04 LTS

Logs

An exception occurred while initiating a multipart upload: Error executing "CreateMultipartUpload" on "https://mystorage.storage.googleapis.com/polly.jpeg?uploads"; AWS HTTP error: Client error: `POST https://mystorage.storage.googleapis.com/polly.jpeg?uploads` resulted in a `400 Bad Request` response: <?xml version='1.0' encoding='UTF-8'?><Error><Code>InvalidArgument</Code><Message>Invalid argument.</Message><Details>PO (truncated...) InvalidArgument (client): Invalid argument. - <?xml version='1.0' encoding='UTF-8'?><Error><Code>InvalidArgument</Code><Message>Invalid argument.</Message><Details>POST object expects Content-Type multipart/form-data</Details></Error>

[lubo]

Same problem here with primary storage.

[thisboyiscrazy]

It seems to be related to the fact that GCP does not handle Multipart uploads

[vdwijngaert]

Is there a workaround to this problem?

Edit: this is a "workaround" for the multipart uploader always being used, but it does NOT implement support for large files.

In lib/private/Files/ObjectStore/S3ObjectTrait.php

function writeObject($urn, $stream) {
                // Prevent multipart upload from always being used, 
                // to "fix" uploading files to Google Cloud Storage.
                $this->getConnection()->upload($this->bucket, $urn, $stream, 'private', [
                        'mup_threshold' => S3_UPLOAD_PART_SIZE,
                        'part_size' => S3_UPLOAD_PART_SIZE
                ]);
                return;
                // rest of code goes here

[Cinerar]

i believe this patch could help #288 (comment)

[solracsf]

For those reaching this post, here is the reason why GGS is not compatble, at this time, with Nextcloud.
Read https://dzone.com/articles/extending-an-amazon-s3-integration-to-google-cloud "Differences in Uploads".

[despens]

It would be great if nextcloud could support GCS storage natively as primary storage. At the moment, a minio service has to serve as an S3 proxy, which introduces quite some overhead for large files, which have to be buffered two times: in the regular nextcloud upload, and the local minio instance.

[szaimen]

As Nextcloud doesn't officially support Google Cloud Storage, I don't think that implementing this is feasible.

But cc @nextcloud/server-triage on this.

[despens]

I'm guessing that if #27034 is implemented, this issue might solve itself.

[gersur]

Use Google Cloud storage via GCS Fuse
#6256 (comment)

[solracsf]

#27034 is merged on v27 (not yet published).
Feel free to try it once out and report back.

[joshtrichards]

Unfortunately, there will still be other issues because GCS simply isn't fully S3 compliant:

#38166 (comment)

Specifically:

https://cloud.google.com/storage/docs/release-notes#October_12_2021

Might be worth voting on the Google tracker that you're a GCS user that is impacted by that issue too since it breaks S3 compatibility:

https://issuetracker.google.com/issues/229020040

[joshtrichards]

#38166 (comment)

Looks like Google improved their S3 compatibility for Multipart Uploads.

Please report back how it's functions for you: @CorruptMonkey @despens @lubo etc

[despens]

Thanks @joshtrichards for alerting to this change. Unfortunately I am unable to test since I moved to a completely different storage provider. I hope others will be able to help.

[xvybihal]

@joshtrichards I was just dealing with connecting GCS to Nextcloud and based on my short testing, it is still broken. I was able to work with small files (KBs or few MBs) just fine, but 2 GB iso file just failed by GCS erroring into 500:

Aws\S3\Exception\S3MultipartUploadException: An exception occurred while initiating a multipart upload: Error executing "CreateMultipartUpload" on "https://BUCKET.storage.googleapis.com/rhel-baseos-9.1-x86_64-boot.iso?uploads"; AWS HTTP error: Server error: `POST https://BUCKET.storage.googleapis.com/rhel-baseos-9.1-x86_64-boot.iso?uploads` resulted in a `500 Internal Server Error` response:
<?xml version='1.0' encoding='UTF-8'?><Error><Code>InternalError</Code><Message>We encountered an internal error. Please (truncated...)
 InternalError (server): We encountered an internal error. Please try again. - <?xml version='1.0' encoding='UTF-8'?><Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><Details>AN+NOG/vxzJGl8Qu70mzzsXFMLz8A9PJCg+zKY8xVI84ZtbWpsSncqESIHncVvEGdoc9oTdRCmKEKzWNcNmr0Z5+TW3uv8l1hkhUBYdQcRpCiLZsVdLBiiT8yD2dA6Ach+fOA8d3DxH4</Details></Error> at <<closure>>

 0. /srv/http/drive.acme.cz/nextcloud/3rdparty/aws/aws-sdk-php/src/Multipart/AbstractUploadManager.php line 156
    Aws\Multipart\AbstractUploadManager->transformException()

[joshtrichards]

@xvybihal That's a 500 error from GCS. Not really much we can do about that... other than retry and hope their end recovers.

It happening occasionally is somewhat expected with a cloud based storage platform, which is why most S3 implementations retry more than once.

• https://cloud.google.com/storage/docs/xml-api/reference-status#500%E2%80%94internal-server-error

As far as I can tell at the moment we run the AWS S3 SDK in standard mode so we retry at least 3 times.

• https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html
• https://cloud.google.com/storage/docs/retry-strategy#exponential-backoff

[nextcloud-command]

This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions.