Get random access to the rows

[severo]

Currently, only the first rows can be obtained with /rows. We want to get access to slices of the rows through pagination, eg /rows?from=40000&rows=10

[severo]

Currently, some datasets are fully on the disk (those that cannot be streamed, and have a small size) and could be queried for ranges of rows, or could produce statistics.
But it's a fallback, not the normal mode, and if we implement this, we would want to have it to all the datasets if possible.

[severo]

Another Hub that gives random access to the data: https://app.activeloop.ai/activeloop/coco-train (right panel)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[severo]

Design thoughts (thanks to discussion with @lhoestq):

• the endpoint will be GET /rows, with required parameters dataset, config, split and optional parameters to control the number of results, eg: from and limit? offset and length (see Get random access to the rows #13 (comment))
• the API service will request the parquet files (if they exist), and return the formatted JSON. Beware: it might take some time, in particular if the rows are very big (every row can contain several MB of JSON) so we should not impose a timeout on the response.
• to be discussed: what to do with the audio files and images? They are stored as a struct with fields: bytes (the raw bytes) and path (the filename). We could return a temporary URL in the /rows JSON, and then generate a temporary static file if the URL is called from the client (again: without a timeout, because generating the temporary file can take some time). Fo images, maybe depend on an external service like cloudinary.

[lhoestq]

the endpoint will be GET /rows, with required parameters dataset, config, split and optional parameters to control the number of results, eg: from and limit?

for consistency with the arrow terminology you can use offset and length

[mariosasko]

the API service will request the parquet files (if they exist), and return the formatted JSON. Beware: it might take some time, in particular if the rows are very big (every row can contain several MB of JSON) so we should not impose a timeout on the response.

Maybe we could eventually switch to Apache Flight to make this performant.

A nice article that compares Apache Flight and the standard approach based on returning a binary format over HTTP: https://voltrondata.com/resources/data-transfer-at-the-speed-of-flight

[severo]

We could provide two response formats for the /rows endpoint, depending on what the client wants:

• JSON
• Arrow Flight

We could use the Accept header, or a query parameter, to select the format of the response.

[severo]

Another option would be to create an Arrow Flight server that can be accessed on a dedicated endpoint (it's RPC, not REST API); and the /rows endpoint would use that RPC endpoint to serve a JSON serialized response.

[severo]

Done