Support streaming Beam datasets from HF GCS preprocessed data #5689
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The documentation is not available anymore as the PR was closed or merged. |
In [1]: from datasets import load_dataset
In [2]: ds = load_dataset("wikipedia", "20220301.en", split="train", streaming=True); item = next(iter(ds)); item
Out[2]:
{'id': '12',
'url': 'https://en.wikipedia.org/wiki/Anarchism',
'title': 'Anarchism',
'text': 'Anarchism is a political philosophy and movement that is sceptical of authority and rejects all involuntary, coercive forms of hierarchy. Anarchism calls for the abolition of the state, which it holds to be unnecessary, undesirable, and harmful. As a historically left-wing movement, placed on the farthest left of the political spectrum, it is usually described alongside communalism and libertarian Marxism as the libertarian wing (libertarian socialism) of the socialist movement,...} |
albertvillanova
changed the title
albertvillanova
changed the title
|
I love your example 🏴 |
lhoestq
reviewed
Apr 4, 2023
lhoestq
reviewed
Apr 4, 2023
src/datasets/builder.py
Outdated
| return ExamplesIterable(self._generate_examples_from_hf_gcs, {"split": split}) | ||
|
|
||
| def _generate_examples_from_hf_gcs(self, split): | ||
| remote_prepared_filename = f"{self._remote_cache_dir_from_hf_gcs}/{self.name}-{split}.arrow" |
There was a problem hiding this comment.
(nit) Beam builders may create sharded arrow files now.
It's the case for none of the datasets on GCP but if we regenerate a new one it may en up sharded.
You can check the dataset info.shard_lengths to know if it's sharded or not and how many shards there are.
if self.info.splits[split].shard_lengths:
num_shards = len(self.info.splits[split].shard_lengths)
urls = [
f"{self._remote_cache_dir_from_hf_gcs}/{self.name}-{split}-{shard_id:05d}-of-{num_shards:05d}.arrow"
for shard_id in range(num_shards)
]
else:
urls = [f"{self._remote_cache_dir_from_hf_gcs}/{self.name}-{split}.arrow"]edit: fixed self.info.splits[split].shard_lengths
There was a problem hiding this comment.
I see... Thanks.
Let me refactor the code.
There was a problem hiding this comment.
As the attribute shard_lengths belongs to SplitInfo (and not DatasetInfo), I have refactored the method so that split is now of type SplitInfo instead of str.
Show benchmarksPyArrow==8.0.0 Show updated benchmarks!Benchmark: benchmark_array_xd.json
Benchmark: benchmark_getitem_100B.json
Benchmark: benchmark_indices_mapping.json
Benchmark: benchmark_iterating.json
Benchmark: benchmark_map_filter.json
Show updated benchmarks!Benchmark: benchmark_array_xd.json
Benchmark: benchmark_getitem_100B.json
Benchmark: benchmark_indices_mapping.json
Benchmark: benchmark_iterating.json
Benchmark: benchmark_map_filter.json
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements streaming Apache Beam datasets that are already preprocessed by us and stored in the HF Google Cloud Storage:
This is done by streaming from the prepared Arrow files in HF Google Cloud Storage.
This will fix their corresponding dataset viewers. Related to:
Related to:
CC: @severo