Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added support to write metadata files in parquet #5105

Merged
merged 50 commits into from
Apr 2, 2024
Merged

Added support to write metadata files in parquet #5105

merged 50 commits into from
Apr 2, 2024

Commits on Jan 31, 2024

  1. Copied methods from inside parquet hadoop to write metadata files

    @malhotrashivam
    malhotrashivam committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    f8ae4ef View commit details
    Browse the repository at this point in the history

  2. Calling methods from inside ParquetHadoop for writing metadata files

    @malhotrashivam
    malhotrashivam committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    0ef7f79 View commit details
    Browse the repository at this point in the history

Commits on Feb 13, 2024

  1. Cleaned up the code and added a lot of TODOs for review

    @malhotrashivam
    malhotrashivam committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    44d031f View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into sm-pq-metadata

    @malhotrashivam
    malhotrashivam committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    6394bac View commit details
    Browse the repository at this point in the history

  3. Some more changes

    @malhotrashivam
    malhotrashivam committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    6c6b075 View commit details
    Browse the repository at this point in the history

Commits on Feb 14, 2024

  1. WIP commit

    @malhotrashivam
    malhotrashivam committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    9263aa0 View commit details
    Browse the repository at this point in the history

Commits on Feb 16, 2024

  1. Added a custom metadata file writer

    @malhotrashivam
    malhotrashivam committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    8c887c4 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into sm-pq-metadata

    @malhotrashivam
    malhotrashivam committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    05400c5 View commit details
    Browse the repository at this point in the history

  3. Minor fix

    @malhotrashivam
    malhotrashivam committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    b696649 View commit details
    Browse the repository at this point in the history

Commits on Feb 17, 2024

  1. Fixed failing test

    @malhotrashivam
    malhotrashivam committed Feb 17, 2024
    Configuration menu
    Copy the full SHA
    797b3dc View commit details
    Browse the repository at this point in the history

Commits on Feb 20, 2024

  1. Moved some code around

    @malhotrashivam
    malhotrashivam committed Feb 20, 2024
    Configuration menu
    Copy the full SHA
    d987a06 View commit details
    Browse the repository at this point in the history

  2. Minor change

    @malhotrashivam
    malhotrashivam committed Feb 20, 2024
    Configuration menu
    Copy the full SHA
    d4da175 View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2024

  1. Review comments

    @malhotrashivam
    malhotrashivam committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    f33d712 View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2024

  1. Merge branch 'main' into sm-pq-metadata

    @malhotrashivam
    malhotrashivam committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    7d66445 View commit details
    Browse the repository at this point in the history

  2. Read offset index from column chunk on demand

    @malhotrashivam
    malhotrashivam committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    238f5f5 View commit details
    Browse the repository at this point in the history

  3. Fixed failing test

    @malhotrashivam
    malhotrashivam committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    7a3652d View commit details
    Browse the repository at this point in the history

Commits on Feb 27, 2024

  1. Added support for partitioned parquet writing

    @malhotrashivam
    malhotrashivam committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    1f41eef View commit details
    Browse the repository at this point in the history

  2. Added some more tests

    @malhotrashivam
    malhotrashivam committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    c8aa764 View commit details
    Browse the repository at this point in the history

  3. Added some more tests

    @malhotrashivam
    malhotrashivam committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    ff99a36 View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2024

  1. Added a new API for writing a partitioned table directly

    @malhotrashivam
    malhotrashivam committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    3e48937 View commit details
    Browse the repository at this point in the history

Commits on Feb 29, 2024

  1. Improved the tests

    @malhotrashivam
    malhotrashivam committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    e584343 View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2024

  1. Review with Ryan part 1

    @malhotrashivam
    malhotrashivam committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    f659f3c View commit details
    Browse the repository at this point in the history

  2. Added more tests

    @malhotrashivam
    malhotrashivam committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    3651e5f View commit details
    Browse the repository at this point in the history

  3. Iterating using chunked iterators

    @malhotrashivam
    malhotrashivam committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    51eefe1 View commit details
    Browse the repository at this point in the history

  4. Removed some unnecessary includes

    @malhotrashivam
    malhotrashivam committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    60ee1f5 View commit details
    Browse the repository at this point in the history

  5. Added support for {index} and {partition} in file basename

    @malhotrashivam
    malhotrashivam committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    5c7353f View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2024

  1. Review comments

    @malhotrashivam
    malhotrashivam committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    c174452 View commit details
    Browse the repository at this point in the history

  2. Minor touchups

    @malhotrashivam
    malhotrashivam committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    ab73df0 View commit details
    Browse the repository at this point in the history

Commits on Mar 6, 2024

  1. Added fix and tests for big decimals 

    Also fixed the instruction for writing metadata file
    @malhotrashivam
    malhotrashivam committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    1cb8b81 View commit details
    Browse the repository at this point in the history

  2. Updated a comment

    @malhotrashivam
    malhotrashivam committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    20e8204 View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2024

  1. Review with Ryan part 1

    @malhotrashivam
    malhotrashivam committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    9892d14 View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2024

  1. Review with Ryan part 2 

    Tests are failing
    @malhotrashivam
    malhotrashivam committed Mar 11, 2024
    Configuration menu
    Copy the full SHA
    1d98927 View commit details
    Browse the repository at this point in the history

  2. Minor touchups

    @malhotrashivam
    malhotrashivam committed Mar 11, 2024
    Configuration menu
    Copy the full SHA
    5018feb View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2024

  1. Fixed failing tests

    @malhotrashivam
    malhotrashivam committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    a529fc2 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into sm-pq-metadata

    @malhotrashivam
    malhotrashivam committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    48906dd View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2024

  1. Added python APIs and improved comments 

    Also, moved from File to String
    @malhotrashivam
    malhotrashivam committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    819a1b9 View commit details
    Browse the repository at this point in the history

Commits on Mar 19, 2024

  1. Added more fixes for python

    @malhotrashivam
    malhotrashivam committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    10b7e0d View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2024

  1. Review with Ryan and Chip

    @malhotrashivam
    malhotrashivam committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    9f7c55e View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into sm-pq-metadata

    @malhotrashivam
    malhotrashivam committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    b62abb7 View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2024

  1. Review with Chip part 2

    @malhotrashivam
    malhotrashivam committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    0d7c62e View commit details
    Browse the repository at this point in the history

  2. Review with Chip and Jianfeng Part 3

    @malhotrashivam
    malhotrashivam committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    5a3de8e View commit details
    Browse the repository at this point in the history

  3. Review with Chip and Jianfeng continued

    @malhotrashivam
    malhotrashivam committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    f68551f View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2024

  1. Added new APIs for managing indexes

    @malhotrashivam
    malhotrashivam committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    b11ebd1 View commit details
    Browse the repository at this point in the history

  2. Trigger CI jobs

    @malhotrashivam
    malhotrashivam committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    3bdc92b View commit details
    Browse the repository at this point in the history

Commits on Mar 26, 2024

  1. Review with Ryan

    @malhotrashivam
    malhotrashivam committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    3dd6097 View commit details
    Browse the repository at this point in the history

Commits on Mar 27, 2024

  1. Added python support for writing indexes

    @malhotrashivam
    malhotrashivam committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    d8bc0a5 View commit details
    Browse the repository at this point in the history

  2. Reordered comments

    @malhotrashivam
    malhotrashivam committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    f700883 View commit details
    Browse the repository at this point in the history

  3. Added more details to python comments

    @malhotrashivam
    malhotrashivam committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    6246289 View commit details
    Browse the repository at this point in the history

  4. Moved from list to sequence

    @malhotrashivam
    malhotrashivam committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    fa2b0c5 View commit details
    Browse the repository at this point in the history

Commits on Apr 1, 2024

  1. Added fixes and tests for windows

    @malhotrashivam
    malhotrashivam committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    7909ea0 View commit details
    Browse the repository at this point in the history