Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Icechunk Support #256

Merged
merged 90 commits into from
Oct 22, 2024
Merged

Add Icechunk Support #256

merged 90 commits into from
Oct 22, 2024

Commits on Sep 27, 2024

  1. move vds_with_manifest_arrays fixture up

    @TomNicholas
    TomNicholas committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    7b00e41 View commit details
    Browse the repository at this point in the history

  2. sketch implementation

    @TomNicholas
    TomNicholas committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    c82221c View commit details
    Browse the repository at this point in the history

  3. test that we can create an icechunk store

    @TomNicholas
    TomNicholas committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    d29362b View commit details
    Browse the repository at this point in the history

  4. fixture to create icechunk filestore in temporary directory

    @TomNicholas
    TomNicholas committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    2aa3cb5 View commit details
    Browse the repository at this point in the history

  5. get the async fixture working properly

    @TomNicholas
    TomNicholas committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    f2c095c View commit details
    Browse the repository at this point in the history

  6. split into more functions

    @TomNicholas
    TomNicholas committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    6abe32d View commit details
    Browse the repository at this point in the history

  7. change mode

    @TomNicholas
    TomNicholas committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    93080b3 View commit details
    Browse the repository at this point in the history

  8. try creating zarr group and arrays explicitly

    @TomNicholas
    TomNicholas committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    bebf370 View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2024

  1. create root group from store

    @TomNicholas
    TomNicholas committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    833e5f0 View commit details
    Browse the repository at this point in the history

  2. todos

    @TomNicholas
    TomNicholas committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    9853140 View commit details
    Browse the repository at this point in the history

  3. do away with the async pytest fixtures/functions

    @TomNicholas
    TomNicholas committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    030a96e View commit details
    Browse the repository at this point in the history

  4. successfully writes root group attrs

    @TomNicholas
    TomNicholas committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    90ca5cf View commit details
    Browse the repository at this point in the history

  5. check array metadata is correct

    @TomNicholas
    TomNicholas committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    b138dde View commit details
    Browse the repository at this point in the history

  6. try to write array attributes

    @TomNicholas
    TomNicholas committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    6631102 View commit details
    Browse the repository at this point in the history

  7. sketch test for checking virtual references have been set correctly

    @TomNicholas
    TomNicholas committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    e92b56c View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. test setting single virtual ref

    @TomNicholas
    TomNicholas committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    2c8c0ee View commit details
    Browse the repository at this point in the history

  2. use async properly

    @TomNicholas
    TomNicholas committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    a2ce1ed View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. better separation of handling of loadable variables

    @TomNicholas
    TomNicholas committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    9393995 View commit details
    Browse the repository at this point in the history

  2. fix chunk key format

    @TomNicholas
    TomNicholas committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    956e324 View commit details
    Browse the repository at this point in the history

  3. use require_array

    @TomNicholas
    TomNicholas committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    2d7d5f6 View commit details
    Browse the repository at this point in the history

  4. check that store supports writes

    @TomNicholas
    TomNicholas committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    8726e23 View commit details
    Browse the repository at this point in the history

  5. removed outdated note about awaiting

    @TomNicholas
    TomNicholas committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    387f345 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. fix incorrect chunk key in test

    @TomNicholas
    TomNicholas committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    b2a0700 View commit details
    Browse the repository at this point in the history

  2. absolute path

    @TomNicholas
    TomNicholas committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    4ffb55e View commit details
    Browse the repository at this point in the history

  3. convert to file URI before handing to icechunk

    @TomNicholas
    TomNicholas committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    f929fcb View commit details
    Browse the repository at this point in the history

  4. test that without encoding we can definitely read one chunk

    @TomNicholas
    TomNicholas committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    e9c1287 View commit details
    Browse the repository at this point in the history

  5. Work on encoding test

    @mpiannucci
    mpiannucci committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    2fe3012 View commit details
    Browse the repository at this point in the history

  6. Merge remote-tracking branch 'origin/icechunk' into matt/icechunk-enc… 

    …oding
    @mpiannucci
    mpiannucci committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    33d8ce8 View commit details
    Browse the repository at this point in the history

  7. Update test to match

    @mpiannucci
    mpiannucci committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    8aa6034 View commit details
    Browse the repository at this point in the history

  8. Quick comment

    @mpiannucci
    mpiannucci committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    aa2d415 View commit details
    Browse the repository at this point in the history

  9. more comprehensive

    @mpiannucci
    mpiannucci committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    7e4e2ce View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2024

  1. add attrtirbute encoding

    @mpiannucci
    mpiannucci committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    9a03245 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2024

  1. Merge pull request zarr-developers#2 from earth-mover/matt/icechunk-e… 

    …ncoding

Matt/icechunk encoding
    @TomNicholas
    TomNicholas authored Oct 4, 2024
    Configuration menu
    Copy the full SHA
    9676485 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2024

  1. Fix array dimensions

    @mpiannucci
    mpiannucci committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    bbaf3ba View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2024

  1. Merge pull request zarr-developers#3 from earth-mover/matt/array-dims

    @mpiannucci
    mpiannucci authored Oct 11, 2024
    Configuration menu
    Copy the full SHA
    31945cd View commit details
    Browse the repository at this point in the history

  2. Fix v3 codec pipeline

    @mpiannucci
    mpiannucci committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    49daa7e View commit details
    Browse the repository at this point in the history

  3. Put xarray dep back

    @mpiannucci
    mpiannucci committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    756ff92 View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2024

  1. Handle codecs, but get bad results

    @mpiannucci
    mpiannucci committed Oct 12, 2024
    Configuration menu
    Copy the full SHA
    8c7242e View commit details
    Browse the repository at this point in the history

  2. Gzip an d zlib are not directly working

    @mpiannucci
    mpiannucci committed Oct 12, 2024
    Configuration menu
    Copy the full SHA
    666b676 View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2024

  1. Get up working with numcodecs zarr 3 codecs

    @mpiannucci
    mpiannucci committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    9076ad7 View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2024

  1. Update codec pipeline

    @mpiannucci
    mpiannucci committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    7a160fd View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2024

  1. Merge pull request zarr-developers#4 from earth-mover/matt/v3-codecs 

    Fix v3 codec pipeline
    @mpiannucci
    mpiannucci authored Oct 15, 2024
    Configuration menu
    Copy the full SHA
    286a9b5 View commit details
    Browse the repository at this point in the history

  2. oUdpate to latest icechunk using sync api

    @mpiannucci
    mpiannucci committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    8f1f96e View commit details
    Browse the repository at this point in the history

  3. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    b59060d View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2024

  1. Merge remote-tracking branch 'origin/main' into icechunk

    @mpiannucci
    mpiannucci committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    4f3bafa View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'mpiannucci/icechunk' into icechunk

    @mpiannucci
    mpiannucci committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    26db575 View commit details
    Browse the repository at this point in the history

  3. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    01d261c View commit details
    Browse the repository at this point in the history

  4. Some type stuff

    @mpiannucci
    mpiannucci committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    52e53f9 View commit details
    Browse the repository at this point in the history

  5. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    d10de6b View commit details
    Browse the repository at this point in the history

  6. Update zarr and icechunk tests, fix zarr v3 metadata

    @mpiannucci
    mpiannucci committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    d0b6bfb View commit details
    Browse the repository at this point in the history

  7. Update import we dont need

    @mpiannucci
    mpiannucci committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    b7dc5f5 View commit details
    Browse the repository at this point in the history

  8. Update kerhcunk tests

    @mpiannucci
    mpiannucci committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    1e8ba7d View commit details
    Browse the repository at this point in the history

  9. Check for v3 metadata import in zarr test

    @mpiannucci
    mpiannucci committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    6b5b8b5 View commit details
    Browse the repository at this point in the history

  10. More tests

    @mpiannucci
    mpiannucci committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    00ecae3 View commit details
    Browse the repository at this point in the history

  11. type checker

    @mpiannucci
    mpiannucci committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    14ee6f9 View commit details
    Browse the repository at this point in the history

  12. types

    @mpiannucci
    mpiannucci committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    23d98de View commit details
    Browse the repository at this point in the history

  13. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    c633f13 View commit details
    Browse the repository at this point in the history

  14. More types

    @mpiannucci
    mpiannucci committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    308a508 View commit details
    Browse the repository at this point in the history

  15. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    60cb43e View commit details
    Browse the repository at this point in the history

  16. ooops

    @mpiannucci
    mpiannucci committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    d72e9d8 View commit details
    Browse the repository at this point in the history

  17. Merge remote-tracking branch 'mpiannucci/icechunk' into icechunk

    @mpiannucci
    mpiannucci committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    0e0e5ac View commit details
    Browse the repository at this point in the history

  18. One left

    @mpiannucci
    mpiannucci committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    3873fde View commit details
    Browse the repository at this point in the history

  19. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    19bc9ae View commit details
    Browse the repository at this point in the history

  20. Finally done being dumb

    @mpiannucci
    mpiannucci committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    4b6aefc View commit details
    Browse the repository at this point in the history

  21. Merge remote-tracking branch 'mpiannucci/icechunk' into icechunk

    @mpiannucci
    mpiannucci committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    9fa9b38 View commit details
    Browse the repository at this point in the history

  22. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    4d5c46a View commit details
    Browse the repository at this point in the history

  23. Support loadables without tests

    @mpiannucci
    mpiannucci committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    5cb3d21 View commit details
    Browse the repository at this point in the history

  24. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    3e31f21 View commit details
    Browse the repository at this point in the history

  25. Add test for multiple chunks to check order

    @mpiannucci
    mpiannucci committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    e105b78 View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2024

  1. Add loadable varaible test

    @mpiannucci
    mpiannucci committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    ea52003 View commit details
    Browse the repository at this point in the history

  2. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    54d9bea View commit details
    Browse the repository at this point in the history

  3. Add accessor, simple docs

    @mpiannucci
    mpiannucci committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    303e5cb View commit details
    Browse the repository at this point in the history

  4. Update icechunk.py 

    Co-authored-by: Tom Nicholas <tom@cworthy.org>
    @mpiannucci @TomNicholas
    mpiannucci and TomNicholas authored Oct 22, 2024
    Configuration menu
    Copy the full SHA
    8838a1d View commit details
    Browse the repository at this point in the history

  5. Update accessor.py 

    Co-authored-by: Tom Nicholas <tom@cworthy.org>
    @mpiannucci @TomNicholas
    mpiannucci and TomNicholas authored Oct 22, 2024
    Configuration menu
    Copy the full SHA
    dd6c118 View commit details
    Browse the repository at this point in the history

  6. Fix attributes when loadables are available

    @mpiannucci
    mpiannucci committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    85de689 View commit details
    Browse the repository at this point in the history

  7. Merge remote-tracking branch 'mpiannucci/icechunk' into icechunk

    @mpiannucci
    mpiannucci committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    9b471ec View commit details
    Browse the repository at this point in the history

  8. Protect zarr import

    @mpiannucci
    mpiannucci committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    8cd5237 View commit details
    Browse the repository at this point in the history

  9. Fix import errors in icechunk writer

    @mpiannucci
    mpiannucci committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    542a953 View commit details
    Browse the repository at this point in the history

  10. More protection

    @mpiannucci
    mpiannucci committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    305b0c6 View commit details
    Browse the repository at this point in the history

  11. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    2f8e270 View commit details
    Browse the repository at this point in the history

  12. i am bad at this

    @mpiannucci
    mpiannucci committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    7ce9f67 View commit details
    Browse the repository at this point in the history

  13. Add xarray roundtrip asserts

    @mpiannucci
    mpiannucci committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    8b4863e View commit details
    Browse the repository at this point in the history

  14. Add icechunk to api.rst

    @mpiannucci
    mpiannucci committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    b072535 View commit details
    Browse the repository at this point in the history

  15. Update virtualizarr/tests/test_writers/test_icechunk.py 

    Co-authored-by: Tom Nicholas <tom@cworthy.org>
    @mpiannucci @TomNicholas
    mpiannucci and TomNicholas authored Oct 22, 2024
    Configuration menu
    Copy the full SHA
    45ae850 View commit details
    Browse the repository at this point in the history

  16. More test improvements, update realeses.rst

    @mpiannucci
    mpiannucci committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    36eaad1 View commit details
    Browse the repository at this point in the history

  17. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    9f4e978 View commit details
    Browse the repository at this point in the history

  18. tmore testing

    @mpiannucci
    mpiannucci committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    3b496f5 View commit details
    Browse the repository at this point in the history

  19. Merge remote-tracking branch 'mpiannucci/icechunk' into icechunk

    @mpiannucci
    mpiannucci committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    1e580a5 View commit details
    Browse the repository at this point in the history

  20. Figure out tests for real this time

    @mpiannucci
    mpiannucci committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    117479c View commit details
    Browse the repository at this point in the history

  21. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    46c41d0 View commit details
    Browse the repository at this point in the history