Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

adding hugging face dataset download class #1970

Closed
wants to merge 4 commits into from
Closed

adding hugging face dataset download class #1970

wants to merge 4 commits into from

Conversation

deepanker13
Copy link
Contributor

What this PR does / why we need it:

  1. Adding hugging face dataset download class.
  2. Abstract class download_model_and_tokenizer name fix

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Partially Fixes #1945

Checklist:

  • Docs included if any changes are user facing

@google-oss-prow Google OSS Prow
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: deepanker13
Once this PR has been reviewed and has the lgtm label, please assign jinchihe for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@deepanker13
Copy link
Contributor Author

@johnugeorge please check this

@coveralls
Copy link

coveralls commented Dec 21, 2023 •
edited
Loading

Pull Request Test Coverage Report for Build 7415177127

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage decreased (-0.06%) to 42.839%
Totals Coverage Status
Change from base Build 7285460954: -0.06%
Covered Lines: 3751
Relevant Lines: 8756
💛 - Coveralls

andreyvelich
andreyvelich reviewed Dec 21, 2023
View reviewed changes
Copy link
Member

@andreyvelich andreyvelich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this!
I left a few comments @deepanker13

sdk/python/kubeflow/storage_init_container/hugging_face.py Outdated Show resolved Hide resolved
sdk/python/kubeflow/storage_init_container/hugging_face.py Show resolved Hide resolved
sdk/python/kubeflow/storage_init_container/storage.py Show resolved Hide resolved
sdk/python/kubeflow/storage_init_container/hugging_face.py Outdated Show resolved Hide resolved
sdk/python/kubeflow/storage_init_container/hugging_face.py Outdated

def __post_init__(self):
# Custom checks or validations can be added here
if self.repo_id is None:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we verify that access token exists ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

access_token is not needed for public datasets

sdk/python/kubeflow/storage_init_container/requirements.txt Outdated Show resolved Hide resolved
@google-oss-prow google-oss-prow bot added size/L and removed size/M labels Jan 4, 2024
@deepanker13
Copy link
Contributor Author

deepanker13 commented Jan 8, 2024 •
edited
Loading

@andreyvelich @tenzen-y @johnugeorge if it is good to go, can we merge this?

@deepanker13
Copy link
Contributor Author

changes were merged in #1962 so closing this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andreyvelich andreyvelich andreyvelich left review comments

@tenzen-y tenzen-y tenzen-y left review comments

@jinchihe jinchihe Awaiting requested review from jinchihe

@kuizhiqing kuizhiqing Awaiting requested review from kuizhiqing

Assignees
No one assigned
Labels
size/L
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@deepanker13 @coveralls @andreyvelich @tenzen-y