HTTP: Support for other operations

[gaby]

Currently the HTTP service only supports GET and HEAD. Are there plans to add support for POST and PUT. It would be great to be able to also send data, instead of only reading.

[Xuanwo]

Most plain HTTP Serve doesn't define the POST and PUT behavior. Maybe you want to take a look over our webdav service?

[gaby]

@Xuanwo I will take a look. I was looking into using OpenDAL to replace some code running with aiohttp in Python. Still not sure if it's the the right tool though.

[Xuanwo]

I was looking into using OpenDAL to replace some code running with aiohttp in Python.

Very happy to hear that! I'm willing to provide help for this!

[gaby]

@Xuanwo We were able to use HTTP/WebDAV to do GET/HEAD, one thing that was confusing is that these services only support the AsyncOperator.

Another thing I noticed is that WebDAV is missing POST, even though it's mentioned in the RFC (https://datatracker.ietf.org/doc/html/rfc4918)

[Xuanwo]

one thing that was confusing is that these services only support the AsyncOperator.

Yes. We depends on reqwest now and still seeking ways to support blocking operations.

Another thing I noticed is that WebDAV is missing POST,

OpenDAL is not a one-to-one mapping of storage services' API. Can you share more details on the operation you want to do over storage service? For POST on webdav, you want to create a new dir?

[gaby]

one thing that was confusing is that these services only support the AsyncOperator.

Yes. We depends on reqwest now and still seeking ways to support blocking operations.

That makes sense, it was only confusing at first since libs like Request let do blocking HTTP operations.

Another thing I noticed is that WebDAV is missing POST,

OpenDAL is not a one-to-one mapping of storage services' API. Can you share more details on the operation you want to do over storage service? For POST on webdav, you want to create a new dir?

I'm trying to upload files, the server has a POST route. I could add a PUT route because I manage the server code, but if it was a third party server, I wouldn't be able to POST data.

Another thing I want to try is Streaming Upload big files. Mainly so that they don't get loaded into memory. I noticed some of the methods support FilePath's but I havent tested if they get loaded into memory or streamed.

I currently to this with aiohttp using this: https://docs.aiohttp.org/en/stable/client_quickstart.html#streaming-uploads

Or with requests using this: https://docs.python-requests.org/en/v1.2.3/user/advanced/#streaming-uploads

Main difference being that requests is a blocking operation, while aiohttp doesn't block.

[Xuanwo]

Another thing I want to try is Streaming Upload big files. Mainly so that they don't get loaded into memory. I noticed some of the methods support FilePath's but I havent tested if they get loaded into memory or streamed.

Thanks for pointing out! We are working on this now: #2084. By adding sink support, we can upload a big file without reading all it's content into memory first. I'm hoping the new copy_from() API will address this.

Besides, OpenDAL rust core already supports Writer and we will expose it to python binding so that users can streaming data too.

[gaby]

Another thing I want to try is Streaming Upload big files. Mainly so that they don't get loaded into memory. I noticed some of the methods support FilePath's but I havent tested if they get loaded into memory or streamed.

Thanks for pointing out! We are working on this now: #2084. By adding sink support, we can upload a big file without reading all it's content into memory first. I'm hoping the new copy_from() API will address this.

Good to know, I will hold off testing streaming files for now.

Not sure if adding POST is in the timeline or out of scope for WebDAV

[Xuanwo]

Not sure if adding POST is in the timeline or out of scope for WebDAV

There are many ways to upload a file via POST. Could you share the detailed method or service for posting it? OpenDAL is willing to add support for a service or API Specs (like webdav or Upload API).

[xyjixyjixyji]

Yes. We depends on reqwest now and still seeking ways to support blocking operations.

Maybe we can use block_on(async_op) to support blocking operations?

[Xuanwo]

Maybe we can use block_on(async_op) to support blocking operations?

Yes, it's possible. However, the process is more complex as we also need to convert AsyncRead to Read.

[gaby]

Not sure if adding POST is in the timeline or out of scope for WebDAV

There are many ways to upload a file via POST. Could you share the detailed method or service for posting it? OpenDAL is willing to add support for a service or API Specs (like webdav or Upload API).

I just have a custom REST server that supports uploading files. Currently I do so using the aiohttp/requests example posted above.

aiohttp: https://docs.aiohttp.org/en/stable/client_quickstart.html#streaming-uploads

requests: https://docs.python-requests.org/en/v1.2.3/user/advanced/#streaming-uploads

Small data is not streamed, just send as a payload.

[Xuanwo]

I just have a custom REST server that supports uploading files.

Sorry, OpenDAL cannot be used in this case. It is designed to connect with storage services instead of a custom API that may not be general enough and could be difficult to work with accurately.

[gaby]

I just have a custom REST server that supports uploading files.

Sorry, OpenDAL cannot be used in this case. It is designed to connect with storage services instead of a custom API that may not be general enough and could be difficult to work with accurately.

How about adding a service named "REST", and having OpenDAL be able to do rest operations like GET, HEAD, PUT, POST, DELETE ?

[Xuanwo]

How about adding a service named "REST", and having OpenDAL be able to do rest operations like GET, HEAD, PUT, POST, DELETE ?

Thanks for the advice. However, file uploading involves more than a simple POST or PUT. There are edge cases that need to be handled, such as:

• Does this service support content-type?
• What status code will the service return for successful operation?
• What status code and error response will the service return for errors?

Therefore, OpenDAL can only connect with services that have clear API specifications covering all these details. We are willing to implement such a service if you need it.

[gaby]

How about adding a service named "REST", and having OpenDAL be able to do rest operations like GET, HEAD, PUT, POST, DELETE ?

Thanks for the advice. However, file uploading involves more than a simple POST or PUT. There are edge cases that need to be handled, such as:

• Does this service support content-type?
• What status code will the service return for successful operation?
• What status code and error response will the service return for errors?

Therefore, OpenDAL can only connect with services that have clear API specifications covering all these details. We are willing to implement such a service if you need it.

i see what you mean, thanks for your help. I will close this as solved. For now aiohttp/requests work fine