Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix ICL race conditions #1978

Merged
merged 13 commits into from
Feb 21, 2023
Merged

Fix ICL race conditions #1978

merged 13 commits into from
Feb 21, 2023

Conversation

dakinggg
Copy link
Contributor

@dakinggg dakinggg commented Feb 17, 2023

What does this PR do?

Fixes two issues with ICL

  • only downloads the dataset file once per node
  • requires destination_path arg (and makes it possible to pass through the dataloader helper) to avoid accidentally overwriting files and loading the wrong ones

What issue(s) does this change relate to?

CO-1796

Before submitting

  • Have you read the contributor guidelines?
  • Was this change discussed/approved in a GitHub issue first? It is much more likely to be merged if so.
  • Did you update any related docs and document your change?
  • Did you update any related tests and add any new tests related to your change? (see testing)
  • Did you run the tests locally to make sure they pass?
  • Did you run pre-commit on your change? (see the pre-commit section of prerequisites)

@dakinggg dakinggg marked this pull request as ready for review February 17, 2023 05:15
composer/utils/dist.py Outdated Show resolved Hide resolved
composer/utils/dist.py Outdated Show resolved Hide resolved
Co-authored-by: Mihir Patel <mihir.v.patel7@gmail.com>
@dakinggg dakinggg merged commit 7fb85bf into mosaicml:dev Feb 21, 2023
@dakinggg dakinggg deleted the icl-hang branch June 1, 2023 00:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants