Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sharded manifests for tarred datasets #6395

Merged
merged 30 commits into from
Apr 18, 2023
Merged

Sharded manifests for tarred datasets #6395

merged 30 commits into from
Apr 18, 2023

Commits on Apr 6, 2023

  1. testing sharded manifests 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    37b43f5 View commit details
    Browse the repository at this point in the history

  2. compatibility 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    955bb0d View commit details
    Browse the repository at this point in the history

  3. proper fixes 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    7be7860 View commit details
    Browse the repository at this point in the history

  4. adding flag tot convert_to_tarred_audio_dataset 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    ce027f5 View commit details
    Browse the repository at this point in the history

Commits on Apr 7, 2023

  1. shard_manifests conf param 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    a4fd990 View commit details
    Browse the repository at this point in the history

  2. propagating the shard_manifests param 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    eac0324 View commit details
    Browse the repository at this point in the history

  3. propagating the shard_manifests param 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    fc4ccba View commit details
    Browse the repository at this point in the history

  4. distributed checks 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    d5f4898 View commit details
    Browse the repository at this point in the history

  5. typo 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    cc762e7 View commit details
    Browse the repository at this point in the history

  6. typo 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    1f78a49 View commit details
    Browse the repository at this point in the history

  7. fixes 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    483901d View commit details
    Browse the repository at this point in the history

  8. fixes 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    ac3f5ad View commit details
    Browse the repository at this point in the history

  9. fixes 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    9dacbdd View commit details
    Browse the repository at this point in the history

  10. fixes 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    dc81d26 View commit details
    Browse the repository at this point in the history

  11. fixes 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    18e8b99 View commit details
    Browse the repository at this point in the history

  12. fixes 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    bd5cc3b View commit details
    Browse the repository at this point in the history

  13. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    0f572b1 View commit details
    Browse the repository at this point in the history

Commits on Apr 8, 2023

  1. fixes based on PR comments and tests 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 8, 2023
    Configuration menu
    Copy the full SHA
    f3cd8ff View commit details
    Browse the repository at this point in the history

  2. fixes based on PR comments and tests 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 8, 2023
    Configuration menu
    Copy the full SHA
    b1aac87 View commit details
    Browse the repository at this point in the history

  3. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Apr 8, 2023
    Configuration menu
    Copy the full SHA
    ded5462 View commit details
    Browse the repository at this point in the history

Commits on Apr 13, 2023

  1. fixes to convert_to_tarred_audio_dataset.py 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    a37f794 View commit details
    Browse the repository at this point in the history

  2. reversing manifest shards flag 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    f788c30 View commit details
    Browse the repository at this point in the history

  3. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    e2ac42a View commit details
    Browse the repository at this point in the history

  4. tests 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    4fba9fc View commit details
    Browse the repository at this point in the history

  5. Merge branch 'shrdmnf' of https://github.com/nvidia/nemo into shrdmnf

    @bmwshop
    bmwshop committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    c27c57a View commit details
    Browse the repository at this point in the history

Commits on Apr 14, 2023

  1. excluding manifests from webdataset url expansion 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    15a5d5a View commit details
    Browse the repository at this point in the history

  2. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    559581c View commit details
    Browse the repository at this point in the history

  3. expand manifest paths before attempting to cache from datastore 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    b712738 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'shrdmnf' of https://github.com/nvidia/nemo into shrdmnf

    @bmwshop
    bmwshop committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    55a3ace View commit details
    Browse the repository at this point in the history

Commits on Apr 18, 2023

  1. explicit use of UTF-8 for manifest i/o 

    Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
    @bmwshop
    bmwshop committed Apr 18, 2023
    Configuration menu
    Copy the full SHA
    e66b216 View commit details
    Browse the repository at this point in the history