Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stream COPY from a directory #6338

Closed
4 tasks done
BohuTANG opened this issue Jun 29, 2022 · 5 comments
Closed
4 tasks done

Stream COPY from a directory #6338

BohuTANG opened this issue Jun 29, 2022 · 5 comments
Assignees
@lichuang
Labels
A-query Area: databend query C-feature Category: feature

Comments

@BohuTANG
Copy link
Member

BohuTANG commented Jun 29, 2022 •
edited
Loading

Summary

Like:
https://docs.snowflake.com/en/user-guide/data-load-considerations-load.html#loading-older-files

Draft:
Generate a digest from metadata, including:

  1. database id
  2. table id
  3. File etag(md5?)
  4. File name
@BohuTANG BohuTANG added C-feature Category: feature A-query Area: databend query labels Jun 29, 2022
@BohuTANG

This comment was marked as resolved.

Sign in to view
@Xuanwo

This comment was marked as resolved.

Sign in to view
@junnplus

This comment was marked as resolved.

Sign in to view
@BohuTANG
Copy link
Member Author

This is related to metasrv, such as the unique identify ID can be stored in the metasrv.
cc @lichuang

@lichuang lichuang self-assigned this Jul 27, 2022
This was referenced Sep 2, 2022
Closed
Closed
@BohuTANG
Copy link
Member Author

BohuTANG commented Sep 4, 2022

Because databend can COPY from remote files, so the file may not be an object from s3.

@lichuang lichuang mentioned this issue Sep 8, 2022
Closed
@Xuanwo Xuanwo moved this from 📋 Backlog to 🪤 In Waiting in Xuanwo's Work Sep 20, 2022
@Xuanwo Xuanwo moved this from 🪤 In Waiting to 📦 Done in Xuanwo's Work Sep 21, 2022
@Xuanwo Xuanwo moved this from 📦 Done to 📥 Archived in Xuanwo's Work Sep 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lichuang lichuang

Labels
A-query Area: databend query C-feature Category: feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: save copy into table stage file meta, avoid duplicate copy file
4 participants
@BohuTANG @lichuang @Xuanwo @junnplus