Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

POC: Add ParquetMetaDataReader #6392

Draft
wants to merge 27 commits into
base: master
Choose a base branch
from
Draft

POC: Add ParquetMetaDataReader #6392

wants to merge 27 commits into from
+1,009 −102

Commits on Sep 12, 2024

  1. add ParquetMetaDataReader

    @etseidl
    etseidl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    0c5087f View commit details
    Browse the repository at this point in the history

  2. add todo

    @etseidl
    etseidl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    d5b60ab View commit details
    Browse the repository at this point in the history

  3. add more todos

    @etseidl
    etseidl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    d462fda View commit details
    Browse the repository at this point in the history

  4. take a stab at reading metadata without file size provided

    @etseidl
    etseidl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    6b9dd1c View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2024

  1. temporarily comment out MetadataLoader

    @etseidl
    etseidl committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    0a2c4b2 View commit details
    Browse the repository at this point in the history

  2. remove debug print

    @etseidl
    etseidl committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    58f2463 View commit details
    Browse the repository at this point in the history

  3. clippy

    @etseidl
    etseidl committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    08b985a View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2024

  1. Merge remote-tracking branch 'origin/master' into metadata_reader

    @etseidl
    etseidl committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    96062e1 View commit details
    Browse the repository at this point in the history

  2. add more todos

    @etseidl
    etseidl committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    cdf6ac5 View commit details
    Browse the repository at this point in the history

  3. uncomment MetadataLoader

    @etseidl
    etseidl committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    25e23d7 View commit details
    Browse the repository at this point in the history

  4. silence doc warnings

    @etseidl
    etseidl committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    03bc663 View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2024

  1. fix size check

    @etseidl
    etseidl committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    51a5a72 View commit details
    Browse the repository at this point in the history

  2. add try_parse_range

    @etseidl
    etseidl committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    8a3f496 View commit details
    Browse the repository at this point in the history

  3. start on documentation

    @etseidl
    etseidl committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    f8450e2 View commit details
    Browse the repository at this point in the history

  4. make sure docs compile

    @etseidl
    etseidl committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    180e3e6 View commit details
    Browse the repository at this point in the history

  5. attempt recovery in test

    @etseidl
    etseidl committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    9d1147d View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. implement some suggestions from review

    @etseidl
    etseidl committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    1a1d3aa View commit details
    Browse the repository at this point in the history

  2. remove suffix reading for now

    @etseidl
    etseidl committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    d450ab8 View commit details
    Browse the repository at this point in the history

  3. add new error types to aid recovery

    @etseidl
    etseidl committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    3c340b7 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. remove parquet_metadata_from_file and add ParquetMetaDataReader::parse

    @etseidl
    etseidl committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    0d13599 View commit details
    Browse the repository at this point in the history

  2. remove todo

    @etseidl
    etseidl committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    d300cf3 View commit details
    Browse the repository at this point in the history

  3. point to with_prefetch_hint from try_load docstring

    @etseidl
    etseidl committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    4ee162f View commit details
    Browse the repository at this point in the history

  4. refactor the retry logic 

    replace try_parse_range() with try_parse_sized()
    @etseidl
    etseidl committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    2d65c3f View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'origin/master' into metadata_reader

    @etseidl
    etseidl committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    2a2cf81 View commit details
    Browse the repository at this point in the history

  6. add some more tests

    @etseidl
    etseidl committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    faff575 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. add load() and bring over tests from async_reader/metadata.rs

    @etseidl
    etseidl committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    c9e5ea6 View commit details
    Browse the repository at this point in the history

  2. only run new tests if async is enabled

    @etseidl
    etseidl committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    4214909 View commit details
    Browse the repository at this point in the history