Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Google Drive remote source does not chunkify the request for Tus server #4857

Closed
2 tasks done
mina-sebastian opened this issue Jan 5, 2024 · 4 comments
Closed
2 tasks done

Google Drive remote source does not chunkify the request for Tus server #4857

mina-sebastian opened this issue Jan 5, 2024 · 4 comments
Assignees
@mifi
Labels
Bug

Comments

@mina-sebastian
Copy link

Initial checklist

  • I understand this is a bug report and questions should be posted in the Community Forum
  • I searched issues and couldn’t find anything (or linked relevant results below)

Link to runnable example

No response

Steps to reproduce

I have a limit of 100MB per request on my server.
Uppy init:

const mb = 1024*1024;
new Uppy({ id: 'uppy1', autoProceed: true, debug: true})
  .use(Tus, { endpoint: 'https://cloud.mysite.com/files', chunkSize: 100 * mb })
  .use(Webcam)
  .use(RemoteSources, { companionUrl: 'https://companion.mysite.com/', sources: ['GoogleDrive']})

Companion standalone config:

{
    "providerOptions": {
		"drive": {
			"key": GOOGLE_KEY,
			"secret": GOOGLE_SECRET
		}
	},
	"server": {
		"host": "companion.mysite.com",
		"protocol": "https",
		"path": ""
	},
	"filePath": "/ssd/temp_uploads/"
}

Tusd server command:
tusd -hooks-http https://mysite.com/api/cloud/upload_check -disable-cors -behind-proxy -upload-dir=/ssd/uploads -host=192.168.0.122 -port=3022

Expected behavior

When I upload something from a device(below or above 100MB), everything is ok. It should be able to do the same when a file is uploaded from Google Drive.

Actual behavior

If i upload something bigger than 100MB, from Google Drive, i receive 413 http error. When a file lower than 100MB is uploaded, everything is ok again.
@mifi
Copy link
Contributor

mifi commented Jan 9, 2024 •
edited
Loading

what's your companion configration? i just tried to upload a 100gb file from google drive to tus and at 500+mb the uploaded it's still going strong..

i receive 413 http error

do you see anything in companion logs? also when do you get the 413 error? do you get it immediately after the upload starts, or only right after the upload progress has exceeded 100mb?

also are you running newest version of uppy and companion?

@mina-sebastian
Copy link
Author

mina-sebastian commented Jan 9, 2024 •
edited
Loading

I start companion(standalone) by companion --config companion.json.
companion.json:
{ "providerOptions": { "drive": { "key": GOOGLE_KEY, "secret": GOOGLE_SECRET } }, "server": { "host": "companion.mysite.com", "protocol": "https", "path": "" }, "filePath": "/ssd/temp_uploads/" }

I checked uppy/core, uppy/google-drive and uppy/tus, and they are up-to-date.
3a641699f4514aed7104d02ee6d6d40d

The error is thrown only right after the upload progress has exceeded 100MB(cloudflare limit).
Companion log after i uploaded a ~200MB file:

companion: 2024-01-09T14:42:32.820Z [debug] 7ea3a73e-eef2-4c5a-93f9-c67aff0bccc1null Socket connection received. Starting remote download/upload.
companion: 2024-01-09T14:42:32.820Z [debug] cd52baf5 controller.get.provider.size need to download the whole file first
companion: 2024-01-09T14:42:32.820Z [debug] cd52baf5 uploader.download fully downloading file
companion: 2024-01-09T14:42:32.821Z [debug] cd52baf5 uploader.total.progress 8810 228586446 0.00%
companion: 2024-01-09T14:42:33.010Z [debug] cd52baf5 uploader Received from client: resume
companion: 2024-01-09T14:42:33.822Z [debug] cd52baf5 uploader.total.progress 1326338 228586446 0.58%
companion: 2024-01-09T14:42:34.823Z [debug] cd52baf5 uploader.total.progress 2571522 228586446 1.12%
companion: 2024-01-09T14:42:35.823Z [debug] cd52baf5 uploader.total.progress 18649040 228586446 8.16%
companion: 2024-01-09T14:42:36.823Z [debug] cd52baf5 uploader.total.progress 54029361 228586446 23.64%
companion: 2024-01-09T14:42:37.823Z [debug] cd52baf5 uploader.total.progress 86720512 228586446 37.94%
companion: 2024-01-09T14:42:38.824Z [debug] cd52baf5 uploader.total.progress 108535808 228586446 47.48%
companion: 2024-01-09T14:42:39.824Z [debug] cd52baf5 uploader.total.progress 113516544 228586446 49.66%
companion: 2024-01-09T14:42:39.987Z [debug] cd52baf5 uploader.download finished fully downloading file
companion: 2024-01-09T14:42:40.648Z [error] uploader.tus.error DetailedError: tus: unexpected response while uploading chunk, originated from request (method: PATCH, url: https://cloud.site.org/files/1a8feab6a0a2fedb7100c28d4c249ded, response code: 413, response text: <html>
<head><title>413 Request Entity Too Large</title></head>
<body>
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>cloudflare</center>
</body>
</html>
, request id: 7d4d7e61-31a2-4b0c-9876-71bdce4c1cf9)
    at Upload._emitHttpError (/home/devusr/.nvm/versions/node/v20.10.0/lib/node_modules/@uppy/companion/node_modules/tus-js-client/lib.es5/upload.js:872:23)
    at /home/devusr/.nvm/versions/node/v20.10.0/lib/node_modules/@uppy/companion/node_modules/tus-js-client/lib.es5/upload.js:1127:18
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  originalRequest: Request {
    _method: 'PATCH',
    _url: 'https://cloud.site.org/files/1a8feab6a0a2fedb7100c28d4c249ded',
    _headers: {
      'Tus-Resumable': '1.0.0',
      'X-Request-ID': '7d4d7e61-31a2-4b0c-9876-71bdce4c1cf9',
      'Upload-Offset': 0,
      'Content-Type': 'application/offset+octet-stream'
    },
    _request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 6,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: false,
      _last: false,
      chunkedEncoding: false,
      shouldKeepAlive: true,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 228586446,
      _hasBody: true,
      _trailer: '',
      finished: false,
      _headerSent: true,
      _closed: false,
      socket: [TLSSocket],
      _header: 'PATCH /files/1a8feab6a0a2fedb7100c28d4c249ded HTTP/1.1\r\n' +
        'Tus-Resumable: 1.0.0\r\n' +
        'X-Request-ID: 7d4d7e61-31a2-4b0c-9876-71bdce4c1cf9\r\n' +
        'Upload-Offset: 0\r\n' +
        'Content-Type: application/offset+octet-stream\r\n' +
        'Content-Length: 228586446\r\n' +
        'Host: cloud.site.org\r\n' +
        'Connection: keep-alive\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'PATCH',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/files/1a8feab6a0a2fedb7100c28d4c249ded',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: [Function: emitRequestTimeout],
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: true,
      host: 'cloud.site.org',
      protocol: 'https:',
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: true,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    _progressHandler: [Function (anonymous)],
    _requestOptions: {}
  },
  originalResponse: Response {
    _response: IncomingMessage {
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      socket: [TLSSocket],
      httpVersionMajor: 1,
      httpVersionMinor: 1,
      httpVersion: '1.1',
      complete: true,
      rawHeaders: [Array],
      rawTrailers: [],
      joinDuplicateHeaders: undefined,
      aborted: false,
      upgrade: false,
      url: '',
      method: null,
      statusCode: 413,
      statusMessage: 'Request Entity Too Large',
      client: [TLSSocket],
      _consuming: true,
      _dumped: false,
      req: [ClientRequest],
      [Symbol(kCapture)]: false,
      [Symbol(kHeaders)]: null,
      [Symbol(kHeadersCount)]: 22,
      [Symbol(kTrailers)]: null,
      [Symbol(kTrailersCount)]: 0
    },
    _body: '<html>\r\n' +
      '<head><title>413 Request Entity Too Large</title></head>\r\n' +
      '<body>\r\n' +
      '<center><h1>413 Request Entity Too Large</h1></center>\r\n' +
      '<hr><center>cloudflare</center>\r\n' +
      '</html>\r\n'
  },
  causingError: null
}

@mifi
Copy link
Contributor

mifi commented Jan 14, 2024

thx. which companion version are you running?

@mifi
Copy link
Contributor

mifi commented Jan 15, 2024 •
edited
Loading

I think the problem is that you're not setting COMPANION_CHUNK_SIZE for Companion. The Tus option chunkSize that you set to 100MB only affects (browser) local uploads.

I just tested companion remote uploads using Tus and COMPANION_CHUNK_SIZE=1000000, and when I modify the companion code and set

        onChunkComplete(...args) {
          console.log('chunk complete', ...args)
        }

and run an upload, I see

companion: 2024-01-15T04:37:56.803Z [debug] 74095ab4 uploader Received from client: resume
companion: 2024-01-15T04:37:56.803Z [debug] 74095ab4 uploader.socket.wait socket connection received
companion: 2024-01-15T04:37:56.804Z [debug] 62fd6e90-91b9-497c-ab47-95517c12b20anull Socket connection received. Starting remote download/upload.
companion: 2024-01-15T04:37:58.187Z [debug] 74095ab4 uploader.total.progress 0 1073741824 0.00%
companion: 2024-01-15T04:37:59.212Z [debug] 74095ab4 uploader.total.progress 147456 1073741824 0.01%
companion: 2024-01-15T04:38:00.263Z [debug] 74095ab4 uploader.total.progress 327680 1073741824 0.03%
companion: 2024-01-15T04:38:01.414Z [debug] 74095ab4 uploader.total.progress 819200 1073741824 0.08%
companion: 2024-01-15T04:38:02.717Z [debug] 74095ab4 uploader.total.progress 1000000 1073741824 0.09%
chunk complete 1000000 1000000 null
companion: 2024-01-15T04:38:03.884Z [debug] 74095ab4 uploader.total.progress 1147456 1073741824 0.11%
companion: 2024-01-15T04:38:04.944Z [debug] 74095ab4 uploader.total.progress 1229376 1073741824 0.11%
companion: 2024-01-15T04:38:07.935Z [debug] 74095ab4 uploader.total.progress 2000000 1073741824 0.19%
chunk complete 1000000 2000000 null
companion: 2024-01-15T04:38:09.267Z [debug] 74095ab4 uploader.total.progress 2163840 1073741824 0.20%
companion: 2024-01-15T04:38:10.278Z [debug] 74095ab4 uploader.total.progress 2720896 1073741824 0.25%
companion: 2024-01-15T04:38:11.405Z [debug] 74095ab4 uploader.total.progress 3000000 1073741824 0.28%
chunk complete 1000000 3000000 null
companion: 2024-01-15T04:38:12.616Z [debug] 74095ab4 uploader.total.progress 3147456 1073741824 0.29%
companion: 2024-01-15T04:38:13.781Z [debug] 74095ab4 uploader.total.progress 3294912 1073741824 0.31%
companion: 2024-01-15T04:38:15.563Z [debug] 74095ab4 uploader.total.progress 4000000 1073741824 0.37%
chunk complete 1000000 4000000 null

so I think you just need to set the companion variable too

@mifi mifi closed this as completed Jan 15, 2024
mifi added a commit to transloadit/uppy.io that referenced this issue Jan 15, 2024
@mifi
add note about chunkSize
284b94a 
see transloadit/uppy#4857
@mifi mifi mentioned this issue Jan 15, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mifi mifi

Labels
Bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mifi @mina-sebastian