Skip to content

Commit

Permalink
TF 2.13 has a specific optimization for
Browse files Browse the repository at this point in the history
dataset.shuffle(dataset.cardinality()), so use that
instead of dataset.shuffle(len(dataset))
  • Loading branch information
Rocketknight1 committed May 24, 2023
1 parent 757f192 commit 824f96c
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/datasets/utils/tf_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,7 @@ def scan_random_indices(state, indices):

tf_dataset = tf_dataset.scan(base_seed, scan_random_indices)
elif shuffle:
tf_dataset = tf_dataset.shuffle(len(tf_dataset))
tf_dataset = tf_dataset.shuffle(tf_dataset.cardinality())
tf_dataset = tf_dataset.batch(batch_size, drop_remainder=drop_remainder)
else:
tf_dataset = tf_dataset.batch(batch_size, drop_remainder=drop_remainder)
Expand Down

0 comments on commit 824f96c

Please sign in to comment.