feat: query data from S3 location or stage

[BohuTANG]

Summary

Make databend as a query engine, query data from S3 location or stage directly.

Refer:
https://docs.snowflake.com/en/user-guide/querying-stage.html

[Xuanwo]

The potancial that I can recongenize from this feature:

• Query data from dropbox/google drive: we can empower personal/enterprise users without complex infra
• Load data in background: Users query as normal but copy data to databend cloud at the same time. Once load are ready, users can query in a more efficient way.

[doki23]

/assignme

[doki23]

Hmm, is /assignme invalid?

[BohuTANG]

• Load data in background: Users query as normal but copy data to databend cloud at the same time. Once load are ready, users can query in a more efficient way.

There is no COPY here, we can transform the parquet files to fuse engine files directly, for example:

Users can create a table:

CREATE table xx ... location='s3://<user-bucket-path>'  CONNECTION=...

If the location is parquet files and not created by fuse engine, we can query them in normal way:

1. list all the parquet files
2. query them without any optimization (Since it does not have fuse indexes)

If the user does some optimization like:

optimize table xx; -- this statement syntax is a demo

We can:

1. create min/max and other all fuse indexes for the parquet files without loading them
2. convert all parquet files as the fuse engine files, and store some metadata to metasrv

I think @dantengsky have some ideas on it.

[BohuTANG]

Hi @doki23 ,

This feature is related to much databend-query internal mod refractory(Such as planner bind_sql and schema infer), so it's hard to do it now.
So, let's re-assign this issue to @youngsofun , he will start the task and complete the first phase: querying the parquet file from stage/location.

cc @Xuanwo @sundy-li @dantengsky

[doki23]

Get it