cannot engage S3 accelerated endpoints due to AWS SDK API changes?

[movy]

Initial checklist

• I understand this is a bug report and questions should be posted in the Community Forum
• I searched issues and couldn’t find anything (or linked relevant results below)

Link to runnable example

No response

Steps to reproduce

In .env set COMPANION_AWS_USE_ACCELERATE_ENDPOINT=“true” and observe the error:
A custom endpoint cannot be combined with S3 Accelerate

Expected behavior

Companion should use accelerated S3 endpoint based on COMPANION_AWS_USE_ACCELERATE_ENDPOINT=“true”
setting alone.

Actual behavior

Trying to use S3 accelerated endpoints and in my .env I have defined:
COMPANION_AWS_USE_ACCELERATE_ENDPOINT=“true”

and did not defined COMPANION_AWS_ENDPOINT

Yet, Companion returns such error:
A custom endpoint cannot be combined with S3 Accelerate

Listing an accelerated endpoint via COMPANION_AWS_ENDPOINT did not help.

Actually, this error comes not from companion source code, but from AWS SDK itself. Looks like it's similar to #4135, but that workaround does not work anymore apparently.

I have tried messing with source code inside the block where endpoint is defined (https://github.com/transloadit/uppy/pull/4140/files#diff-bde6ee25e8588159566b1b6c431f00a59aac00d4226f750f1402def0eebf24bcR36), but the all my attempts produced either the error message above or a non-accelerated S3 bucket URL.

I will try to dig deeper inside SDK to see at what stage does it decide which URL to return, but if anyone more familiar with the code can have a quick look, it'd probably be faster.

Thanks heaps in advance.

[mifi]

Hi. i'm not sure how to reproduce this problem. when I set COMPANION_AWS_USE_ACCELERATE_ENDPOINT=true companion starts up without any error. i can even upload files

[mifi]

are you using an old bucket with the old behaviour? (bucket created 10+ years ago)

[movy]

The bucket was created last week, here's my env:

COMPANION_HIDE_WELCOME=true
COMPANION_DOMAINS=['domain.com','(\\w+).domain.com']
COMPANION_UPLOAD_URLS=['https://upload.domain.com/']
COMPANION_PATH=/
COMPANION_DATADIR=./output
COMPANION_DOMAIN=upload.domain.com
COMPANION_PROTOCOL=https
COMPANION_PORT=3020
COMPANION_PREAUTH_SECRET="sdfadsffasdfads"

COMPANION_ALLOW_LOCAL_URLS=false
COMPANION_AWS_KEY="ALALALALALLALALALLA"
COMPANION_AWS_SECRET_FILE="/home/ubuntu/aws_s3_secret"
COMPANION_AWS_BUCKET="bucket-name"
COMPANION_AWS_REGION="eu-west-1"
# COMPANION_AWS_PREFIX="OPTIONAL PREFIX"
COMPANION_AWS_USE_ACCELERATE_ENDPOINT=true
COMPANION_AWS_EXPIRES="13800"

The error shows only when trying to upload a file.

[mifi]

how's you frontend code look like? any codesandbox or similar to reproduce the problem? we need some steps to reproduce the problem. "trying to upload a file" can be done in many ways using uppy.

[movy]

Good point, sorry I wasn't clear about it.
I'm using Uppy dashboard + '@uppy/aws-s3-multipart' inside a Vue component, roughly:

uppyOptions: {
      id: 'uppy',
      autoProceed: true,
      allowMultipleUploadBatches: false,
      debug: true,
      restrictions: {
        maxFileSize: 2 * 1024 * 1024 * 1024, // 2Gb
        allowedFileTypes: [
          'video/mp4',
          'video/mpeg',
          'application/x-mpegURL'
        ],
        requiredMetaFields: [],
      },
      meta: {},
      infoTimeout: 10_000,
    }

 this.uppy = new Uppy({...this.uppyOptions})

this.uppy.use(this.AwsS3Multipart, {
            shouldUseMultipart: (file) => file.size > 100 * 2 ** 20,
            companionUrl: 'https://upload.domain.com/',
            getTemporarySecurityCredentials: true})

I will try to create a repro later, as it might be tricky

[movy]

Hi. i'm not sure how to reproduce this problem. when I set COMPANION_AWS_USE_ACCELERATE_ENDPOINT=true companion starts up without any error. i can even upload files

I wanted to clarify, do you receive an accelerated S3 endpoint in such case?

[mifi]

ok with s3-multipart i managed to reproduce the problem. I think what's going on is:
createPresignedPost needs endpoint: https://bucket.s3-accelerate.amazonaws.com because it doesn't support useAccelerateEndpoint: true

uppy/packages/@uppy/companion/src/server/controllers/s3.js

Line 75 in 4e86b8d

createPresignedPost(client, {

however all other S3 methods do NOT support endpoint when useAccelerateEndpoint is set to true, and because we share the s3 client between all s3 functionality, it creates an impossibility.

[movy]

Thanks so much for looking into this, I was bracing to spend my Saturday on an investigation. Please let me know if I can help with tests.

[mifi]

Sure, feel free to test the pr linked above if youre able to

[movy]

I can see now that s3ClientOptions inside

uppy/packages/@uppy/companion/src/server/s3-client.js

Line 59 in 9c8c0f0

s3ClientOptions = {

is correctly set to ...s3-accelerate.amazonaws.com/', yet in browser I still can see upload going to a regional bucket URL. Trying to find where it get's overwrited down the line.

[movy]

After some more debugging I found out that inside server-side Companion an endpoint is formed and passed around correctly as accelerated, but on client-side the actual endpoint is formed by AwsS3Multipart plugin:

uppy/packages/@uppy/aws-s3-multipart/src/createSignedURL.js

Line 84 in 1b6c260

export default async function createSignedURL ({

despite companionUrl in its init config.
Moreover, I checked network exchange between browser and companion and I cannot see endpoint passed anywhere, means AwsS3Multipart rightfully generates one on its own.

Bucket name and region are returned on /sts route, but there's nothing about accelerated attribute.

{
    "credentials": {
        "AccessKeyId": "....",
        "SecretAccessKey": "....",
        "SessionToken": "...=",
        "Expiration": "2023-12-09T17:40:03.000Z"
    },
    "bucket": "bucket-name",
    "region": "eu-west-1"
}

So, my question is: is it a bug in Companion that the actual endpoint is never passed to the client or I missed something about plugin or Companion configuration?

I've updated @uppy/aws-s3-multipart/lib/createSignedURL.js with

  const host = `${bucketName}.s3-accelerate.amazonaws.com`

And everything works as expected now. So seems like accelerated endpoint feature was never a part of @uppy/aws-s3-multipart/, unless I missed some config option to enforce it.

[mifi]

We support signing on the server using aws-s3 (which calls createPresignedPost in companion) using accelerated endpoints. I tested that and it works. Companion returns:

GET s3/params returns

{
...
    "url": "https://***.s3-accelerate.amazonaws.com/",
...
}

We also support signing on the server using aws-s3-multipart (calls getSignedUrl(client, new UploadPartCommand({...}) in companion). Companion returns:

GET s3/multipart/uploadId

{
    "url": "https://***.s3-accelerate.amazonaws.com/my-prefix/...",
    "expires": 800
}

I believe what you are asking for is the /sts endpoint (signing on the client using an AWS STS temporary security credential). Support for that was recently added in #4519 where we implemented our own client signing code (createSignedURL) for the aws-s3-multipart plugin. But we did not think about accelerated endpoints. @aduh95 do you know if it would be easy to adopt createSignedURL / getTemporarySecurityCredentials to accomodate an accelerated s3 endpoint instead of the hardcoded URL?

[mifi]

I'll create a new issue for that #4822