Long processing times while uploading to S3

[solracsf]

Running Nextcloud 17.0.3 behind Cloudflare works fine, unless the upload processing time takes longer than 100s (in our case, uploading files bigger than 2GB trough WebUI to a S3 primary backend). In that case, Cloudflare returns a 524.

Error when assembling chunks, status code 524

Contacting Cloudflare about this hits on a:

... modify your process so it sends a keep alive request...

Cloudflare will typically wait for a HTTP response from origin server for 100 seconds. If no response (or <8Kb) is sent by the origin in that time, Cloudflare will close the connection and serve a 524 error page.

So the question is, can (should?) Nextcloud detect those long running processing times and send a keepalive request every (say) 60 seconds to avoid these problems on some proxies like Cloudflare?

[FlorentCoppint]

By the way, to complete this issue, sometimes some chunks are "missing" on S3 storage (404 Not found) ! So the assembling step is failing. Maybe there should be some checks when sending chunks to S3, if they are not correctly saved, maybe resend them...

[dwehrmann]

I can confirm this happens for me too (error 524) without cloudflare – only when uploading files larger than 1GB. Note that the uploading process finishes fine, the error raises on processing (presumably chunks) on the s3 storage.

`Error: fclose(): supplied resource is not a valid stream resource at /var/www/nextcloud/3rdparty/icewind/streams/src/Wrapper.php#96

<<closure>>

OC\Log\ErrorHandler::onError()

/var/www/nextcloud/3rdparty/icewind/streams/src/Wrapper.php - line 96:

fclose()

/var/www/nextcloud/3rdparty/icewind/streams/src/CallbackWrapper.php - line 117:

Icewind\Streams\Wrapper->stream_close()

<<closure>>

Icewind\Streams\CallbackWrapper->stream_close()

/var/www/nextcloud/3rdparty/guzzlehttp/psr7/src/Stream.php - line 108:

fclose()

/var/www/nextcloud/3rdparty/guzzlehttp/psr7/src/Stream.php - line 74:

GuzzleHttp\Psr7\Stream->close()

<<closure>>

GuzzleHttp\Psr7\Stream->__destruct()`

I'm using latest nextcloud version to date.

[weasam]

My Nextcloud snyc windows client also have this issue, in my windows nextcloud logs says 524, which is cloudflare issue, check the server load and response both are ok, what I know is the Sync client sync a large file, it stop for a while at the sync process. so I blame nextcloud sync client, lol. why it stop, keep going

[joshtrichards]

https://community.cloudflare.com/t/community-tip-fixing-error-524-a-timeout-occurred/42342

https://developers.cloudflare.com/support/troubleshooting/cloudflare-errors/troubleshooting-cloudflare-5xx-errors/#error-524-a-timeout-occurred