Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Zero-copy converting for a location with many parquet files to fuse engine table #7381

Open
BohuTANG opened this issue Aug 30, 2022 · 3 comments
Labels
A-storage Area: databend storage C-feature Category: feature

Comments

@BohuTANG
Copy link
Member

  • Load data in background: Users query as normal but copy data to databend cloud at the same time. Once load are ready, users can query in a more efficient way.

There is no COPY here, we can transform the parquet files to fuse engine files directly, for example:

Users can create a table:

CREATE table xx ... location='s3://<user-bucket-path>'  CONNECTION=...

If the location is parquet files and not created by fuse engine, we can query them in normal way:

  1. list all the parquet files
  2. query them without any optimization (Since it does not have fuse indexes)

If the user does some optimization like:

optimize table xx; -- this statement syntax is a demo

We can:

  1. create min/max and other all fuse indexes for the parquet files without loading them
  2. convert all parquet files as the fuse engine files, and store some metadata to metasrv

I think @dantengsky have some ideas on it.

Originally posted by @BohuTANG in #7211 (comment)

@BohuTANG BohuTANG added the C-feature Category: feature label Aug 30, 2022
@BohuTANG
Copy link
Member Author

Note: This task should wait until #7211 is finished.

@BohuTANG BohuTANG added the A-storage Area: databend storage label Aug 30, 2022
@kesavkolla
Copy link

It would be awesome to support generate/load indices from fuse and actual data is stored under remote storage. This will give very good power for most of the anaytics solutions. The data changes in analytics is rather slow as compared to OLTP system. So even if we compute the indices at a periodic interval that would be a tremendous improvement.

@BohuTANG
Copy link
Member Author

BohuTANG commented Nov 19, 2022

That's absolutely right, thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-storage Area: databend storage C-feature Category: feature
Projects
None yet
Development

No branches or pull requests

2 participants