feature request pipeline to fetch upload data to hugging face #1239
Conversation
There was a problem hiding this comment.
Thanks @koch3092 , left some initial comments, we need to support creating dataset card for camel, refer: https://huggingface.co/docs/datasets/en/dataset_card
Wendong-Fan
requested review from
willshang76,
zjrwtx and
CaelumF
and removed request for
willshang76
…oad-data-to-hugging-face
|
Can we support picture-type data and examples？ |
|
…oad-data-to-hugging-face # Conflicts: # poetry.lock
1.throw NotImplementedError in abstract methods 2.update logging to camel.logger 3.Check the validity of json in upload_file() 4.update poetry.lock 5.Update the token assignment method in test
camel/datahubs/clients/base.py
Outdated
| Returns: | ||
| str: The URL of the created dataset. | ||
| """ | ||
| raise NotImplementedError("Method not implemented.") |
| class RepoType(str, Enum): | ||
| DATASET = "dataset" | ||
| MODEL = "model" | ||
| SPACE = "space" |
| if not authors: | ||
| authors = [] | ||
|
|
||
| if not tags: | ||
| tags = [] |
There was a problem hiding this comment.
authors and tags could be unified, set within metadata
| metadata = { | ||
| "license": license, | ||
| "task_categories": task_categories if task_categories else [], | ||
| "language": language if language else [], | ||
| "tags": tags, | ||
| "pretty_name": dataset_name, | ||
| "size_categories": [size_category] if size_category else [], | ||
| } |
There was a problem hiding this comment.
could we remove keys with None values by using metadata = {k: v for k, v in metadata.items() if v}?
| if not existing_records: | ||
| raise ValueError( | ||
| f"Dataset '{dataset_name}' does not have an existing file to " | ||
| f"update. Use `add_records` first." | ||
| ) |
There was a problem hiding this comment.
add records directly and give user a warming msg?
Thanks @Wendong-Fan , It looks good to me. |
…oad-data-to-hugging-face
Wendong-Fan
deleted the
1213-feature-request-pipeline-to-fetch-upload-data-to-hugging-face
branch
Description
add CRUD of hugging face's datasets/records
Motivation and Context
close #1213
Types of changes
What types of changes does your code introduce? Put an
xin all the boxes that apply:Implemented Tasks
Checklist
Go over all the following points, and put an
xin all the boxes that apply.If you are unsure about any of these, don't hesitate to ask. We are here to help!