Timeout requests to dandidav that take too long to resolve

[jwodder]

The CLI should have a --timeout option that takes an integer number of seconds. When specified, requests to the server that take more than that many seconds to complete should be timed out; when not specified, no timeout should be in effect.

This can be done via either the TimeoutLayer middleware or ServiceBuilder::timeout().

[jwodder]

@yarikoptic Do you think this could help with the DoS problem?

[jwodder]

For the record, Heroku's recommendation regarding timeouts is:

Heroku recommends setting a timeout within your application and keeping the value well under 30 seconds, such as 10 or 15 seconds.

[yarikoptic]

That is what I am observing ATM when e.g. navigating to https://webdav.dandiarchive.org/dandisets/000026/draft/sub-I46/ses-SPIM/micr/ -- eventually heroku times out with

such error message in browser

and log

2025-01-16T02:01:22.197768+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path="/dandisets/000026/draft/sub-I46/ses-SPIM/micr/" host=webdav.dandiarchive.org request_id=453a347d-be05-4bb4-992f-0ae17ae3ea4e fwd="73.218.136.210" dyno=web.1 connect=2ms service=30001ms status=503 bytes=0 protocol=https

and before then dandidav going very busy asking those extra attributes on the assets in the folder e.g.

2025-01-16T02:04:32.961717+00:00 app[web.1]: {"timestamp":"2025-01-16T02:04:32.961580534Z","level":"DEBUG","fields":{"message":"Making HTTP request"},"target":"dandidav::httputil","span":{"method":"GET","url":"https://api.dandiarchive.org/api/dandisets/000026/versions/draft/assets/aedc4ae4-d72d-402a-ac2e-a4422be14b7a/info/","name":"outgoing-request"},"spans":[{"method":"GET","request_id":"c4811fa1-d50c-4cd8-9b9a-300b5a5b646c","uri":"/dandisets/000026/draft/sub-I46/ses-SPIM/micr/","name":"request"},{"method":"GET","url":"https://api.dandiarchive.org/api/dandisets/000026/versions/draft/assets/aedc4ae4-d72d-402a-ac2e-a4422be14b7a/info/","name":"outgoing-request"}]}
2025-01-16T02:04:32.978120+00:00 app[web.1]: {"timestamp":"2025-01-16T02:04:32.977900859Z","level":"DEBUG","fields":{"message":"Response received","status":"200 OK"},"target":"dandidav::httputil","span":{"method":"GET","url":"https://api.dandiarchive.org/api/dandisets/000026/versions/draft/assets/aedc4ae4-d72d-402a-ac2e-a4422be14b7a/info/","name":"outgoing-request"},"spans":[{"method":"GET","request_id":"c4811fa1-d50c-4cd8-9b9a-300b5a5b646c","uri":"/dandisets/000026/draft/sub-I46/ses-SPIM/micr/","name":"request"},{"method":"GET","url":"https://api.dandiarchive.org/api/dandisets/000026/versions/draft/assets/aedc4ae4-d72d-402a-ac2e-a4422be14b7a/info/","name":"outgoing-request"}]}
2025-01-16T02:04:32.978122+00:00 app[web.1]: {"timestamp":"2025-01-16T02:04:32.978010253Z","level":"DEBUG","fields":{"message":"Making HTTP request"},"target":"dandidav::httputil","span":{"method":"GET","url":"https://api.dandiarchive.org/api/dandisets/000026/versions/draft/assets/fc3847a8-e956-4bb7-aa06-ca5740d70e41/info/","name":"outgoing-request"},"spans":[{"method":"GET","request_id":"c4811fa1-d50c-4cd8-9b9a-300b5a5b646c","uri":"/dandisets/000026/draft/sub-I46/ses-SPIM/micr/","name":"request"},{"method":"GET","url":"https://api.dandiarchive.org/api/dandisets/000026/versions/draft/assets/fc3847a8-e956-4bb7-aa06-ca5740d70e41/info/","name":"outgoing-request"}]}
2025-01-16T02:04:32.995645+00:00 app[web.1]: {"timestamp":"2025-01-16T02:04:32.995384612Z","level":"DEBUG","fields":{"message":"Response received","status":"200 OK"},"target":"dandidav::httputil","span":{"method":"GET","url":"https://api.dandiarchive.org/api/dandisets/000026/versions/draft/assets/fc3847a8-e956-4bb7-aa06-ca5740d70e41/info/","name":"outgoing-request"},"spans":[{"method":"GET","request_id":"c4811fa1-d50c-4cd8-9b9a-300b5a5b646c","uri":"/dandisets/000026/draft/sub-I46/ses-SPIM/micr/","name":"request"},{"method":"GET","url":"https://api.dandiarchive.org/api/dandisets/000026/versions/draft/assets/fc3847a8-e956-4bb7-aa06-ca5740d70e41/info/","name":"outgoing-request"}]}

so, isn't that "Request timeout" from heroku would still happen at 30 seconds mark even if dandidav itself decides that it is to time out after longer period of time?

I might be not grasping what good the timeouts you mention would do for us. Feels like we need to solve problem of in efficient listing there + may be overall adding "paging" support to dandidav ?

[jwodder]

@yarikoptic

even if dandidav itself decides that it is to time out after longer period of time?

The idea is that dandidav would time out before Heroku. See the quote in my previous comment.

In addition, when Heroku times out, nothing is communicated back to dandidav, so the server continues wasting resources on requests to the Archive. If dandidav had its own timeout for processing incoming requests, then timing out would (I believe) mean that Archive requests stop.

[yarikoptic]

ok, agree. Let's set timeout of 25 seconds or so.