Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PartitionedFile abstraction for flexible table scan #946

Closed
Tracked by #944
yjshen opened this issue Aug 25, 2021 · 0 comments · Fixed by #932
Closed
Tracked by #944

PartitionedFile abstraction for flexible table scan #946

yjshen opened this issue Aug 25, 2021 · 0 comments · Fixed by #932
Labels
enhancement New feature or request

Comments

@yjshen
Copy link
Member

yjshen commented Aug 25, 2021

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Currently, we can only handle table input on a file basis. A more flexible and extensible abstraction is desirable.

Describe the solution you'd like

pub struct PartitionedFile {
    /// Path for the file (e.g. URL, filesystem path, etc)
    pub file_path: String,
    /// Statistics of the file
    pub statistics: Statistics,

    // Several feasible extensions

    // 1. partitioned table
    /// Values of partition columns to be appended to each row
    pub partition_value: Option<Vec<ScalarValue>>,

    // 2. sub-file processing
    /// A range of file to process in the current task
    pub start: u64,
    pub end: u64,
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant