Better tqdm wrapper

[mariosasko]

This PR aligns the tqdm logic with huggingface_hub (without introducing breaking changes), as the current one is error-prone.

Additionally, it improves the doc page about the datasets' utilities, and the handling of local fsspec paths in cached_path.

Fix #6409

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005070 / 0.011353 (-0.006283)	0.003251 / 0.011008 (-0.007757)	0.061528 / 0.038508 (0.023020)	0.055386 / 0.023109 (0.032276)	0.248536 / 0.275898 (-0.027362)	0.272346 / 0.323480 (-0.051134)	0.003875 / 0.007986 (-0.004111)	0.002396 / 0.004328 (-0.001933)	0.047659 / 0.004250 (0.043409)	0.037448 / 0.037052 (0.000396)	0.251101 / 0.258489 (-0.007388)	0.282353 / 0.293841 (-0.011488)	0.027784 / 0.128546 (-0.100762)	0.010534 / 0.075646 (-0.065113)	0.206025 / 0.419271 (-0.213246)	0.035410 / 0.043533 (-0.008123)	0.250626 / 0.255139 (-0.004513)	0.266801 / 0.283200 (-0.016399)	0.017704 / 0.141683 (-0.123979)	1.089970 / 1.452155 (-0.362185)	1.171683 / 1.492716 (-0.321033)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.092700 / 0.018006 (0.074694)	0.301314 / 0.000490 (0.300824)	0.000212 / 0.000200 (0.000012)	0.000044 / 0.000054 (-0.000011)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.018385 / 0.037411 (-0.019026)	0.062364 / 0.014526 (0.047838)	0.075887 / 0.176557 (-0.100670)	0.119484 / 0.737135 (-0.617651)	0.074490 / 0.296338 (-0.221849)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.283893 / 0.215209 (0.068684)	2.746772 / 2.077655 (0.669118)	1.486568 / 1.504120 (-0.017552)	1.376451 / 1.541195 (-0.164744)	1.377928 / 1.468490 (-0.090562)	0.572393 / 4.584777 (-4.012384)	2.383282 / 3.745712 (-1.362430)	2.791614 / 5.269862 (-2.478248)	1.753373 / 4.565676 (-2.812303)	0.063539 / 0.424275 (-0.360736)	0.005014 / 0.007607 (-0.002593)	0.341300 / 0.226044 (0.115256)	3.376960 / 2.268929 (1.108032)	1.914162 / 55.444624 (-53.530462)	1.590188 / 6.876477 (-5.286289)	1.618420 / 2.142072 (-0.523652)	0.648723 / 4.805227 (-4.156504)	0.117745 / 6.500664 (-6.382919)	0.048858 / 0.075469 (-0.026611)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.944422 / 1.841788 (-0.897366)	11.603590 / 8.074308 (3.529282)	10.707000 / 10.191392 (0.515608)	0.130779 / 0.680424 (-0.549645)	0.015126 / 0.534201 (-0.519075)	0.284869 / 0.579283 (-0.294414)	0.266778 / 0.434364 (-0.167585)	0.320646 / 0.540337 (-0.219691)	0.417167 / 1.386936 (-0.969769)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005384 / 0.011353 (-0.005969)	0.003311 / 0.011008 (-0.007698)	0.049933 / 0.038508 (0.011425)	0.052791 / 0.023109 (0.029681)	0.277061 / 0.275898 (0.001162)	0.302149 / 0.323480 (-0.021331)	0.004006 / 0.007986 (-0.003979)	0.002394 / 0.004328 (-0.001934)	0.049020 / 0.004250 (0.044770)	0.040168 / 0.037052 (0.003116)	0.278625 / 0.258489 (0.020136)	0.308641 / 0.293841 (0.014800)	0.029808 / 0.128546 (-0.098738)	0.010873 / 0.075646 (-0.064774)	0.058040 / 0.419271 (-0.361231)	0.032706 / 0.043533 (-0.010827)	0.277254 / 0.255139 (0.022115)	0.295208 / 0.283200 (0.012008)	0.017769 / 0.141683 (-0.123914)	1.126416 / 1.452155 (-0.325739)	1.169046 / 1.492716 (-0.323670)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.094776 / 0.018006 (0.076770)	0.306262 / 0.000490 (0.305772)	0.000223 / 0.000200 (0.000023)	0.000043 / 0.000054 (-0.000011)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.022279 / 0.037411 (-0.015132)	0.086784 / 0.014526 (0.072258)	0.082268 / 0.176557 (-0.094289)	0.120131 / 0.737135 (-0.617004)	0.082862 / 0.296338 (-0.213476)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.300565 / 0.215209 (0.085356)	2.923424 / 2.077655 (0.845769)	1.594836 / 1.504120 (0.090716)	1.504323 / 1.541195 (-0.036872)	1.498495 / 1.468490 (0.030005)	0.570969 / 4.584777 (-4.013808)	2.476966 / 3.745712 (-1.268746)	2.785190 / 5.269862 (-2.484672)	1.749839 / 4.565676 (-2.815837)	0.062809 / 0.424275 (-0.361466)	0.004908 / 0.007607 (-0.002699)	0.361513 / 0.226044 (0.135469)	3.587135 / 2.268929 (1.318207)	1.952030 / 55.444624 (-53.492595)	1.661552 / 6.876477 (-5.214925)	1.678673 / 2.142072 (-0.463399)	0.645083 / 4.805227 (-4.160144)	0.117098 / 6.500664 (-6.383566)	0.041630 / 0.075469 (-0.033839)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.987883 / 1.841788 (-0.853904)	12.300764 / 8.074308 (4.226456)	10.962068 / 10.191392 (0.770675)	0.143200 / 0.680424 (-0.537224)	0.015743 / 0.534201 (-0.518458)	0.289733 / 0.579283 (-0.289550)	0.276384 / 0.434364 (-0.157979)	0.328542 / 0.540337 (-0.211795)	0.552175 / 1.386936 (-0.834761)

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005110 / 0.011353 (-0.006243)	0.003311 / 0.011008 (-0.007697)	0.061962 / 0.038508 (0.023454)	0.050250 / 0.023109 (0.027140)	0.245313 / 0.275898 (-0.030585)	0.268748 / 0.323480 (-0.054732)	0.004693 / 0.007986 (-0.003293)	0.002465 / 0.004328 (-0.001863)	0.047698 / 0.004250 (0.043447)	0.037314 / 0.037052 (0.000262)	0.250370 / 0.258489 (-0.008119)	0.286023 / 0.293841 (-0.007818)	0.027891 / 0.128546 (-0.100655)	0.010574 / 0.075646 (-0.065072)	0.204895 / 0.419271 (-0.214376)	0.036014 / 0.043533 (-0.007519)	0.250959 / 0.255139 (-0.004180)	0.266710 / 0.283200 (-0.016489)	0.018492 / 0.141683 (-0.123191)	1.115340 / 1.452155 (-0.336815)	1.176488 / 1.492716 (-0.316229)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.099409 / 0.018006 (0.081402)	0.310151 / 0.000490 (0.309661)	0.000223 / 0.000200 (0.000023)	0.000044 / 0.000054 (-0.000011)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.018132 / 0.037411 (-0.019279)	0.061820 / 0.014526 (0.047294)	0.074960 / 0.176557 (-0.101596)	0.119793 / 0.737135 (-0.617342)	0.074132 / 0.296338 (-0.222206)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.286388 / 0.215209 (0.071179)	2.830791 / 2.077655 (0.753137)	1.514588 / 1.504120 (0.010468)	1.376514 / 1.541195 (-0.164681)	1.405080 / 1.468490 (-0.063410)	0.555297 / 4.584777 (-4.029480)	2.364838 / 3.745712 (-1.380874)	2.806050 / 5.269862 (-2.463812)	1.756114 / 4.565676 (-2.809562)	0.062254 / 0.424275 (-0.362022)	0.005020 / 0.007607 (-0.002588)	0.346272 / 0.226044 (0.120227)	3.453195 / 2.268929 (1.184266)	1.837810 / 55.444624 (-53.606814)	1.577984 / 6.876477 (-5.298493)	1.560821 / 2.142072 (-0.581251)	0.633930 / 4.805227 (-4.171297)	0.116414 / 6.500664 (-6.384250)	0.042007 / 0.075469 (-0.033462)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.941322 / 1.841788 (-0.900466)	11.740927 / 8.074308 (3.666618)	10.450543 / 10.191392 (0.259151)	0.128820 / 0.680424 (-0.551604)	0.014856 / 0.534201 (-0.519345)	0.285636 / 0.579283 (-0.293647)	0.270051 / 0.434364 (-0.164313)	0.321244 / 0.540337 (-0.219093)	0.415486 / 1.386936 (-0.971450)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005333 / 0.011353 (-0.006020)	0.003370 / 0.011008 (-0.007638)	0.049046 / 0.038508 (0.010538)	0.055767 / 0.023109 (0.032658)	0.273463 / 0.275898 (-0.002435)	0.292909 / 0.323480 (-0.030571)	0.004102 / 0.007986 (-0.003883)	0.002460 / 0.004328 (-0.001868)	0.048025 / 0.004250 (0.043775)	0.040342 / 0.037052 (0.003290)	0.275114 / 0.258489 (0.016625)	0.295988 / 0.293841 (0.002147)	0.029461 / 0.128546 (-0.099085)	0.010654 / 0.075646 (-0.064992)	0.057196 / 0.419271 (-0.362076)	0.033238 / 0.043533 (-0.010295)	0.275885 / 0.255139 (0.020746)	0.288566 / 0.283200 (0.005366)	0.018058 / 0.141683 (-0.123625)	1.130513 / 1.452155 (-0.321642)	1.173608 / 1.492716 (-0.319108)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.097751 / 0.018006 (0.079745)	0.312106 / 0.000490 (0.311616)	0.000232 / 0.000200 (0.000032)	0.000044 / 0.000054 (-0.000010)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.021201 / 0.037411 (-0.016211)	0.070150 / 0.014526 (0.055624)	0.081073 / 0.176557 (-0.095484)	0.119520 / 0.737135 (-0.617615)	0.084479 / 0.296338 (-0.211859)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.292322 / 0.215209 (0.077113)	2.844070 / 2.077655 (0.766415)	1.581838 / 1.504120 (0.077718)	1.462665 / 1.541195 (-0.078530)	1.483013 / 1.468490 (0.014523)	0.558705 / 4.584777 (-4.026072)	2.422368 / 3.745712 (-1.323344)	2.772274 / 5.269862 (-2.497587)	1.725901 / 4.565676 (-2.839775)	0.062993 / 0.424275 (-0.361282)	0.004982 / 0.007607 (-0.002625)	0.344336 / 0.226044 (0.118292)	3.425230 / 2.268929 (1.156302)	1.947199 / 55.444624 (-53.497425)	1.670362 / 6.876477 (-5.206115)	1.674112 / 2.142072 (-0.467961)	0.633857 / 4.805227 (-4.171370)	0.114837 / 6.500664 (-6.385827)	0.042558 / 0.075469 (-0.032911)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.979474 / 1.841788 (-0.862314)	12.110856 / 8.074308 (4.036548)	10.605998 / 10.191392 (0.414606)	0.130769 / 0.680424 (-0.549654)	0.016057 / 0.534201 (-0.518144)	0.296448 / 0.579283 (-0.282835)	0.278078 / 0.434364 (-0.156286)	0.320809 / 0.540337 (-0.219528)	0.570756 / 1.386936 (-0.816180)

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005181 / 0.011353 (-0.006172)	0.003434 / 0.011008 (-0.007574)	0.062333 / 0.038508 (0.023825)	0.058544 / 0.023109 (0.035435)	0.233794 / 0.275898 (-0.042104)	0.258774 / 0.323480 (-0.064706)	0.003869 / 0.007986 (-0.004117)	0.002478 / 0.004328 (-0.001850)	0.047871 / 0.004250 (0.043620)	0.037997 / 0.037052 (0.000945)	0.241269 / 0.258489 (-0.017220)	0.270103 / 0.293841 (-0.023738)	0.027710 / 0.128546 (-0.100836)	0.010683 / 0.075646 (-0.064963)	0.213204 / 0.419271 (-0.206067)	0.036156 / 0.043533 (-0.007377)	0.240061 / 0.255139 (-0.015078)	0.253627 / 0.283200 (-0.029573)	0.017880 / 0.141683 (-0.123803)	1.102965 / 1.452155 (-0.349189)	1.176919 / 1.492716 (-0.315797)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.093206 / 0.018006 (0.075200)	0.300960 / 0.000490 (0.300470)	0.000214 / 0.000200 (0.000014)	0.000042 / 0.000054 (-0.000012)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.019417 / 0.037411 (-0.017994)	0.061948 / 0.014526 (0.047422)	0.073560 / 0.176557 (-0.102997)	0.120682 / 0.737135 (-0.616453)	0.074925 / 0.296338 (-0.221413)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.280157 / 0.215209 (0.064948)	2.760648 / 2.077655 (0.682994)	1.482129 / 1.504120 (-0.021991)	1.364091 / 1.541195 (-0.177104)	1.415680 / 1.468490 (-0.052810)	0.564697 / 4.584777 (-4.020080)	2.364080 / 3.745712 (-1.381633)	2.794018 / 5.269862 (-2.475844)	1.752157 / 4.565676 (-2.813520)	0.062234 / 0.424275 (-0.362041)	0.004927 / 0.007607 (-0.002680)	0.337835 / 0.226044 (0.111790)	3.313819 / 2.268929 (1.044891)	1.834095 / 55.444624 (-53.610530)	1.559964 / 6.876477 (-5.316513)	1.598489 / 2.142072 (-0.543584)	0.636829 / 4.805227 (-4.168399)	0.116436 / 6.500664 (-6.384228)	0.042506 / 0.075469 (-0.032963)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.951508 / 1.841788 (-0.890280)	11.599532 / 8.074308 (3.525224)	10.492355 / 10.191392 (0.300963)	0.151582 / 0.680424 (-0.528842)	0.014356 / 0.534201 (-0.519845)	0.288448 / 0.579283 (-0.290835)	0.265607 / 0.434364 (-0.168757)	0.324455 / 0.540337 (-0.215883)	0.416718 / 1.386936 (-0.970218)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005489 / 0.011353 (-0.005864)	0.003481 / 0.011008 (-0.007527)	0.048952 / 0.038508 (0.010444)	0.054650 / 0.023109 (0.031540)	0.280853 / 0.275898 (0.004955)	0.298089 / 0.323480 (-0.025391)	0.004762 / 0.007986 (-0.003224)	0.002500 / 0.004328 (-0.001828)	0.048503 / 0.004250 (0.044253)	0.042048 / 0.037052 (0.004995)	0.281729 / 0.258489 (0.023240)	0.303625 / 0.293841 (0.009785)	0.028887 / 0.128546 (-0.099659)	0.010687 / 0.075646 (-0.064960)	0.058093 / 0.419271 (-0.361178)	0.032366 / 0.043533 (-0.011167)	0.281987 / 0.255139 (0.026848)	0.295554 / 0.283200 (0.012355)	0.019242 / 0.141683 (-0.122441)	1.127760 / 1.452155 (-0.324395)	1.166868 / 1.492716 (-0.325848)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.092367 / 0.018006 (0.074361)	0.300195 / 0.000490 (0.299706)	0.000222 / 0.000200 (0.000022)	0.000043 / 0.000054 (-0.000011)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.022062 / 0.037411 (-0.015349)	0.069955 / 0.014526 (0.055429)	0.081224 / 0.176557 (-0.095333)	0.120183 / 0.737135 (-0.616953)	0.082846 / 0.296338 (-0.213492)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.295880 / 0.215209 (0.080671)	2.902508 / 2.077655 (0.824853)	1.616311 / 1.504120 (0.112191)	1.491320 / 1.541195 (-0.049875)	1.517333 / 1.468490 (0.048843)	0.566824 / 4.584777 (-4.017953)	2.428397 / 3.745712 (-1.317315)	2.807241 / 5.269862 (-2.462620)	1.786364 / 4.565676 (-2.779312)	0.065253 / 0.424275 (-0.359022)	0.004971 / 0.007607 (-0.002636)	0.350095 / 0.226044 (0.124051)	3.422226 / 2.268929 (1.153297)	1.972955 / 55.444624 (-53.471670)	1.686142 / 6.876477 (-5.190335)	1.694539 / 2.142072 (-0.447533)	0.645709 / 4.805227 (-4.159518)	0.117774 / 6.500664 (-6.382890)	0.041679 / 0.075469 (-0.033790)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.976835 / 1.841788 (-0.864952)	12.358039 / 8.074308 (4.283730)	10.774304 / 10.191392 (0.582912)	0.130442 / 0.680424 (-0.549982)	0.016071 / 0.534201 (-0.518130)	0.289911 / 0.579283 (-0.289372)	0.280693 / 0.434364 (-0.153671)	0.325598 / 0.540337 (-0.214739)	0.549618 / 1.386936 (-0.837318)

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005176 / 0.011353 (-0.006177)	0.003297 / 0.011008 (-0.007711)	0.061673 / 0.038508 (0.023165)	0.052174 / 0.023109 (0.029065)	0.245897 / 0.275898 (-0.030001)	0.273102 / 0.323480 (-0.050377)	0.003870 / 0.007986 (-0.004115)	0.002385 / 0.004328 (-0.001943)	0.047675 / 0.004250 (0.043424)	0.037722 / 0.037052 (0.000670)	0.250780 / 0.258489 (-0.007709)	0.279464 / 0.293841 (-0.014376)	0.028107 / 0.128546 (-0.100439)	0.010460 / 0.075646 (-0.065187)	0.205522 / 0.419271 (-0.213750)	0.035781 / 0.043533 (-0.007752)	0.246526 / 0.255139 (-0.008613)	0.263919 / 0.283200 (-0.019281)	0.018634 / 0.141683 (-0.123049)	1.103845 / 1.452155 (-0.348310)	1.175536 / 1.492716 (-0.317181)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.091696 / 0.018006 (0.073690)	0.301284 / 0.000490 (0.300794)	0.000213 / 0.000200 (0.000013)	0.000051 / 0.000054 (-0.000004)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.019153 / 0.037411 (-0.018258)	0.063846 / 0.014526 (0.049320)	0.073635 / 0.176557 (-0.102922)	0.119625 / 0.737135 (-0.617511)	0.075161 / 0.296338 (-0.221177)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.285637 / 0.215209 (0.070428)	2.751787 / 2.077655 (0.674132)	1.465098 / 1.504120 (-0.039022)	1.341676 / 1.541195 (-0.199519)	1.390636 / 1.468490 (-0.077854)	0.567663 / 4.584777 (-4.017114)	2.378183 / 3.745712 (-1.367529)	2.801830 / 5.269862 (-2.468032)	1.750125 / 4.565676 (-2.815551)	0.063705 / 0.424275 (-0.360570)	0.004967 / 0.007607 (-0.002640)	0.373302 / 0.226044 (0.147258)	3.301847 / 2.268929 (1.032918)	1.830117 / 55.444624 (-53.614508)	1.564360 / 6.876477 (-5.312117)	1.551766 / 2.142072 (-0.590306)	0.654424 / 4.805227 (-4.150803)	0.120656 / 6.500664 (-6.380008)	0.042383 / 0.075469 (-0.033086)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.931815 / 1.841788 (-0.909973)	11.755904 / 8.074308 (3.681596)	10.571707 / 10.191392 (0.380315)	0.131118 / 0.680424 (-0.549306)	0.015241 / 0.534201 (-0.518960)	0.287137 / 0.579283 (-0.292146)	0.265651 / 0.434364 (-0.168713)	0.329083 / 0.540337 (-0.211254)	0.417501 / 1.386936 (-0.969435)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005355 / 0.011353 (-0.005998)	0.003305 / 0.011008 (-0.007703)	0.048289 / 0.038508 (0.009781)	0.059223 / 0.023109 (0.036114)	0.267213 / 0.275898 (-0.008685)	0.290151 / 0.323480 (-0.033329)	0.004683 / 0.007986 (-0.003303)	0.002413 / 0.004328 (-0.001916)	0.047982 / 0.004250 (0.043732)	0.040943 / 0.037052 (0.003891)	0.270967 / 0.258489 (0.012478)	0.297644 / 0.293841 (0.003803)	0.029309 / 0.128546 (-0.099237)	0.010624 / 0.075646 (-0.065023)	0.057359 / 0.419271 (-0.361913)	0.032716 / 0.043533 (-0.010816)	0.268602 / 0.255139 (0.013463)	0.286016 / 0.283200 (0.002817)	0.018578 / 0.141683 (-0.123105)	1.120275 / 1.452155 (-0.331880)	1.195514 / 1.492716 (-0.297202)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.092590 / 0.018006 (0.074584)	0.302589 / 0.000490 (0.302099)	0.000217 / 0.000200 (0.000017)	0.000043 / 0.000054 (-0.000012)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.022439 / 0.037411 (-0.014972)	0.070914 / 0.014526 (0.056388)	0.084927 / 0.176557 (-0.091629)	0.123154 / 0.737135 (-0.613981)	0.085527 / 0.296338 (-0.210812)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.292652 / 0.215209 (0.077443)	2.843736 / 2.077655 (0.766081)	1.561289 / 1.504120 (0.057169)	1.439500 / 1.541195 (-0.101695)	1.485074 / 1.468490 (0.016584)	0.570520 / 4.584777 (-4.014257)	2.436611 / 3.745712 (-1.309102)	2.925600 / 5.269862 (-2.344261)	1.796518 / 4.565676 (-2.769159)	0.065075 / 0.424275 (-0.359200)	0.004995 / 0.007607 (-0.002612)	0.349976 / 0.226044 (0.123932)	3.442535 / 2.268929 (1.173607)	1.919002 / 55.444624 (-53.525622)	1.659222 / 6.876477 (-5.217255)	1.648370 / 2.142072 (-0.493703)	0.643119 / 4.805227 (-4.162108)	0.118015 / 6.500664 (-6.382649)	0.041229 / 0.075469 (-0.034240)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.986226 / 1.841788 (-0.855562)	12.302487 / 8.074308 (4.228179)	10.528848 / 10.191392 (0.337456)	0.143911 / 0.680424 (-0.536513)	0.015265 / 0.534201 (-0.518936)	0.287692 / 0.579283 (-0.291591)	0.277011 / 0.434364 (-0.157353)	0.327650 / 0.540337 (-0.212688)	0.552951 / 1.386936 (-0.833985)

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005234 / 0.011353 (-0.006119)	0.003324 / 0.011008 (-0.007684)	0.062429 / 0.038508 (0.023921)	0.051619 / 0.023109 (0.028510)	0.256850 / 0.275898 (-0.019048)	0.260566 / 0.323480 (-0.062914)	0.002914 / 0.007986 (-0.005071)	0.003093 / 0.004328 (-0.001235)	0.047947 / 0.004250 (0.043696)	0.038753 / 0.037052 (0.001701)	0.246810 / 0.258489 (-0.011679)	0.275128 / 0.293841 (-0.018713)	0.027171 / 0.128546 (-0.101375)	0.010290 / 0.075646 (-0.065356)	0.206069 / 0.419271 (-0.213203)	0.035514 / 0.043533 (-0.008019)	0.240645 / 0.255139 (-0.014494)	0.259693 / 0.283200 (-0.023507)	0.019722 / 0.141683 (-0.121961)	1.128534 / 1.452155 (-0.323620)	1.139602 / 1.492716 (-0.353115)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.095837 / 0.018006 (0.077830)	0.304754 / 0.000490 (0.304264)	0.000204 / 0.000200 (0.000004)	0.000043 / 0.000054 (-0.000012)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.018349 / 0.037411 (-0.019063)	0.062763 / 0.014526 (0.048237)	0.074443 / 0.176557 (-0.102113)	0.120607 / 0.737135 (-0.616528)	0.077721 / 0.296338 (-0.218617)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.281852 / 0.215209 (0.066643)	2.770806 / 2.077655 (0.693151)	1.466255 / 1.504120 (-0.037864)	1.349611 / 1.541195 (-0.191584)	1.385463 / 1.468490 (-0.083027)	0.566489 / 4.584777 (-4.018288)	2.420932 / 3.745712 (-1.324780)	2.809397 / 5.269862 (-2.460464)	1.749734 / 4.565676 (-2.815942)	0.063407 / 0.424275 (-0.360868)	0.005038 / 0.007607 (-0.002569)	0.379121 / 0.226044 (0.153077)	3.500938 / 2.268929 (1.232010)	1.852207 / 55.444624 (-53.592417)	1.570474 / 6.876477 (-5.306002)	1.555222 / 2.142072 (-0.586850)	0.657198 / 4.805227 (-4.148030)	0.119835 / 6.500664 (-6.380829)	0.042453 / 0.075469 (-0.033016)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.949953 / 1.841788 (-0.891835)	11.736811 / 8.074308 (3.662503)	10.558049 / 10.191392 (0.366657)	0.146230 / 0.680424 (-0.534194)	0.014922 / 0.534201 (-0.519279)	0.289100 / 0.579283 (-0.290183)	0.267130 / 0.434364 (-0.167234)	0.320055 / 0.540337 (-0.220282)	0.417244 / 1.386936 (-0.969692)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005309 / 0.011353 (-0.006044)	0.003329 / 0.011008 (-0.007679)	0.048576 / 0.038508 (0.010068)	0.055219 / 0.023109 (0.032110)	0.271522 / 0.275898 (-0.004376)	0.294435 / 0.323480 (-0.029045)	0.004018 / 0.007986 (-0.003968)	0.002456 / 0.004328 (-0.001873)	0.047939 / 0.004250 (0.043689)	0.041195 / 0.037052 (0.004143)	0.274819 / 0.258489 (0.016330)	0.299407 / 0.293841 (0.005566)	0.029145 / 0.128546 (-0.099401)	0.010680 / 0.075646 (-0.064966)	0.057238 / 0.419271 (-0.362034)	0.032722 / 0.043533 (-0.010810)	0.272066 / 0.255139 (0.016927)	0.289223 / 0.283200 (0.006023)	0.017826 / 0.141683 (-0.123857)	1.119079 / 1.452155 (-0.333076)	1.179109 / 1.492716 (-0.313608)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.095662 / 0.018006 (0.077656)	0.307652 / 0.000490 (0.307162)	0.000213 / 0.000200 (0.000013)	0.000051 / 0.000054 (-0.000004)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.022263 / 0.037411 (-0.015149)	0.070224 / 0.014526 (0.055698)	0.081477 / 0.176557 (-0.095079)	0.120763 / 0.737135 (-0.616372)	0.083152 / 0.296338 (-0.213187)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.295780 / 0.215209 (0.080571)	2.926623 / 2.077655 (0.848968)	1.605901 / 1.504120 (0.101781)	1.482874 / 1.541195 (-0.058321)	1.501467 / 1.468490 (0.032977)	0.569566 / 4.584777 (-4.015211)	2.474948 / 3.745712 (-1.270764)	2.831877 / 5.269862 (-2.437985)	1.761229 / 4.565676 (-2.804448)	0.064129 / 0.424275 (-0.360147)	0.004964 / 0.007607 (-0.002643)	0.350081 / 0.226044 (0.124037)	3.446766 / 2.268929 (1.177837)	1.974998 / 55.444624 (-53.469627)	1.683381 / 6.876477 (-5.193095)	1.711543 / 2.142072 (-0.430530)	0.648695 / 4.805227 (-4.156532)	0.118224 / 6.500664 (-6.382440)	0.040895 / 0.075469 (-0.034574)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.960208 / 1.841788 (-0.881580)	12.164941 / 8.074308 (4.090633)	10.860573 / 10.191392 (0.669181)	0.133525 / 0.680424 (-0.546899)	0.015643 / 0.534201 (-0.518558)	0.290898 / 0.579283 (-0.288386)	0.289612 / 0.434364 (-0.144752)	0.325836 / 0.540337 (-0.214501)	0.565592 / 1.386936 (-0.821344)

[lhoestq]

Cool ! just some comment:

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006097 / 0.011353 (-0.005256)	0.004386 / 0.011008 (-0.006622)	0.064481 / 0.038508 (0.025973)	0.059983 / 0.023109 (0.036873)	0.268177 / 0.275898 (-0.007721)	0.296207 / 0.323480 (-0.027273)	0.002986 / 0.007986 (-0.005000)	0.002923 / 0.004328 (-0.001406)	0.048798 / 0.004250 (0.044547)	0.039945 / 0.037052 (0.002893)	0.271234 / 0.258489 (0.012745)	0.295461 / 0.293841 (0.001620)	0.028771 / 0.128546 (-0.099775)	0.011104 / 0.075646 (-0.064542)	0.207471 / 0.419271 (-0.211800)	0.036955 / 0.043533 (-0.006578)	0.254761 / 0.255139 (-0.000378)	0.275933 / 0.283200 (-0.007267)	0.021232 / 0.141683 (-0.120451)	1.170771 / 1.452155 (-0.281384)	1.188900 / 1.492716 (-0.303816)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.092328 / 0.018006 (0.074322)	0.302591 / 0.000490 (0.302102)	0.000220 / 0.000200 (0.000020)	0.000052 / 0.000054 (-0.000003)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.019207 / 0.037411 (-0.018204)	0.070247 / 0.014526 (0.055721)	0.074963 / 0.176557 (-0.101593)	0.124301 / 0.737135 (-0.612834)	0.077356 / 0.296338 (-0.218982)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.283321 / 0.215209 (0.068112)	2.800448 / 2.077655 (0.722793)	1.510278 / 1.504120 (0.006158)	1.390353 / 1.541195 (-0.150842)	1.387881 / 1.468490 (-0.080609)	0.563927 / 4.584777 (-4.020850)	2.387753 / 3.745712 (-1.357959)	2.776655 / 5.269862 (-2.493207)	1.767383 / 4.565676 (-2.798293)	0.064864 / 0.424275 (-0.359411)	0.004999 / 0.007607 (-0.002608)	0.351173 / 0.226044 (0.125129)	3.459446 / 2.268929 (1.190517)	1.873078 / 55.444624 (-53.571547)	1.602831 / 6.876477 (-5.273646)	1.595612 / 2.142072 (-0.546460)	0.648786 / 4.805227 (-4.156441)	0.118720 / 6.500664 (-6.381944)	0.042821 / 0.075469 (-0.032649)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.970738 / 1.841788 (-0.871049)	12.273548 / 8.074308 (4.199240)	11.191375 / 10.191392 (0.999983)	0.131903 / 0.680424 (-0.548521)	0.014512 / 0.534201 (-0.519689)	0.289382 / 0.579283 (-0.289901)	0.269449 / 0.434364 (-0.164915)	0.327557 / 0.540337 (-0.212781)	0.427052 / 1.386936 (-0.959884)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005472 / 0.011353 (-0.005881)	0.003380 / 0.011008 (-0.007628)	0.050677 / 0.038508 (0.012169)	0.059606 / 0.023109 (0.036497)	0.275798 / 0.275898 (-0.000100)	0.303733 / 0.323480 (-0.019747)	0.004187 / 0.007986 (-0.003799)	0.002657 / 0.004328 (-0.001672)	0.048713 / 0.004250 (0.044463)	0.043501 / 0.037052 (0.006449)	0.278845 / 0.258489 (0.020356)	0.305322 / 0.293841 (0.011481)	0.030665 / 0.128546 (-0.097881)	0.010600 / 0.075646 (-0.065047)	0.058923 / 0.419271 (-0.360349)	0.032936 / 0.043533 (-0.010596)	0.272835 / 0.255139 (0.017696)	0.293975 / 0.283200 (0.010775)	0.018193 / 0.141683 (-0.123490)	1.144903 / 1.452155 (-0.307251)	1.192220 / 1.492716 (-0.300497)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.094519 / 0.018006 (0.076513)	0.305591 / 0.000490 (0.305101)	0.000221 / 0.000200 (0.000021)	0.000056 / 0.000054 (0.000001)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.022108 / 0.037411 (-0.015303)	0.070184 / 0.014526 (0.055658)	0.081640 / 0.176557 (-0.094916)	0.124661 / 0.737135 (-0.612474)	0.082229 / 0.296338 (-0.214110)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.303710 / 0.215209 (0.088501)	2.966478 / 2.077655 (0.888824)	1.646066 / 1.504120 (0.141946)	1.551454 / 1.541195 (0.010259)	1.557995 / 1.468490 (0.089505)	0.577723 / 4.584777 (-4.007054)	2.510321 / 3.745712 (-1.235391)	2.951343 / 5.269862 (-2.318519)	1.857550 / 4.565676 (-2.708127)	0.064079 / 0.424275 (-0.360196)	0.004971 / 0.007607 (-0.002636)	0.359022 / 0.226044 (0.132978)	3.628716 / 2.268929 (1.359788)	2.011380 / 55.444624 (-53.433245)	1.710407 / 6.876477 (-5.166070)	1.756235 / 2.142072 (-0.385838)	0.659185 / 4.805227 (-4.146042)	0.120245 / 6.500664 (-6.380419)	0.042751 / 0.075469 (-0.032718)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.026794 / 1.841788 (-0.814993)	12.695125 / 8.074308 (4.620816)	10.864908 / 10.191392 (0.673516)	0.136128 / 0.680424 (-0.544295)	0.016824 / 0.534201 (-0.517377)	0.289717 / 0.579283 (-0.289567)	0.282919 / 0.434364 (-0.151445)	0.323345 / 0.540337 (-0.216992)	0.556375 / 1.386936 (-0.830561)

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005407 / 0.011353 (-0.005946)	0.003464 / 0.011008 (-0.007544)	0.062084 / 0.038508 (0.023576)	0.052582 / 0.023109 (0.029472)	0.251239 / 0.275898 (-0.024659)	0.276675 / 0.323480 (-0.046805)	0.002894 / 0.007986 (-0.005092)	0.003850 / 0.004328 (-0.000479)	0.047789 / 0.004250 (0.043538)	0.038955 / 0.037052 (0.001903)	0.258333 / 0.258489 (-0.000156)	0.290103 / 0.293841 (-0.003738)	0.027291 / 0.128546 (-0.101256)	0.010575 / 0.075646 (-0.065071)	0.207208 / 0.419271 (-0.212063)	0.035848 / 0.043533 (-0.007685)	0.253918 / 0.255139 (-0.001221)	0.269870 / 0.283200 (-0.013330)	0.019830 / 0.141683 (-0.121853)	1.085332 / 1.452155 (-0.366823)	1.171385 / 1.492716 (-0.321331)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.094956 / 0.018006 (0.076950)	0.301104 / 0.000490 (0.300614)	0.000204 / 0.000200 (0.000004)	0.000049 / 0.000054 (-0.000005)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.019045 / 0.037411 (-0.018367)	0.070815 / 0.014526 (0.056289)	0.073763 / 0.176557 (-0.102794)	0.120668 / 0.737135 (-0.616467)	0.075197 / 0.296338 (-0.221141)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.286072 / 0.215209 (0.070863)	2.762868 / 2.077655 (0.685213)	1.504481 / 1.504120 (0.000361)	1.390301 / 1.541195 (-0.150894)	1.449571 / 1.468490 (-0.018919)	0.555598 / 4.584777 (-4.029179)	2.404975 / 3.745712 (-1.340737)	2.864359 / 5.269862 (-2.405503)	1.764913 / 4.565676 (-2.800763)	0.062956 / 0.424275 (-0.361320)	0.005116 / 0.007607 (-0.002491)	0.344027 / 0.226044 (0.117983)	3.426781 / 2.268929 (1.157852)	1.891040 / 55.444624 (-53.553584)	1.599972 / 6.876477 (-5.276505)	1.603464 / 2.142072 (-0.538608)	0.638136 / 4.805227 (-4.167091)	0.117808 / 6.500664 (-6.382857)	0.043740 / 0.075469 (-0.031730)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.934654 / 1.841788 (-0.907133)	12.243698 / 8.074308 (4.169390)	10.566791 / 10.191392 (0.375399)	0.130440 / 0.680424 (-0.549983)	0.014019 / 0.534201 (-0.520182)	0.285453 / 0.579283 (-0.293831)	0.266121 / 0.434364 (-0.168243)	0.325962 / 0.540337 (-0.214375)	0.422181 / 1.386936 (-0.964755)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005151 / 0.011353 (-0.006202)	0.003704 / 0.011008 (-0.007304)	0.049483 / 0.038508 (0.010975)	0.055147 / 0.023109 (0.032038)	0.277589 / 0.275898 (0.001691)	0.301274 / 0.323480 (-0.022206)	0.004031 / 0.007986 (-0.003955)	0.002568 / 0.004328 (-0.001760)	0.048830 / 0.004250 (0.044580)	0.040391 / 0.037052 (0.003339)	0.281031 / 0.258489 (0.022541)	0.304263 / 0.293841 (0.010422)	0.029237 / 0.128546 (-0.099309)	0.010598 / 0.075646 (-0.065048)	0.058089 / 0.419271 (-0.361182)	0.032529 / 0.043533 (-0.011004)	0.275761 / 0.255139 (0.020622)	0.294427 / 0.283200 (0.011227)	0.017227 / 0.141683 (-0.124456)	1.138036 / 1.452155 (-0.314119)	1.201946 / 1.492716 (-0.290770)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.094241 / 0.018006 (0.076234)	0.301622 / 0.000490 (0.301132)	0.000229 / 0.000200 (0.000029)	0.000054 / 0.000054 (-0.000001)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.022731 / 0.037411 (-0.014680)	0.071217 / 0.014526 (0.056691)	0.082619 / 0.176557 (-0.093937)	0.123308 / 0.737135 (-0.613827)	0.083552 / 0.296338 (-0.212787)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.295770 / 0.215209 (0.080561)	2.886069 / 2.077655 (0.808414)	1.597686 / 1.504120 (0.093566)	1.458612 / 1.541195 (-0.082583)	1.501171 / 1.468490 (0.032680)	0.575653 / 4.584777 (-4.009124)	2.444021 / 3.745712 (-1.301691)	2.860192 / 5.269862 (-2.409669)	1.758896 / 4.565676 (-2.806780)	0.063334 / 0.424275 (-0.360941)	0.004913 / 0.007607 (-0.002694)	0.341828 / 0.226044 (0.115783)	3.420310 / 2.268929 (1.151381)	1.996099 / 55.444624 (-53.448525)	1.680112 / 6.876477 (-5.196365)	1.693418 / 2.142072 (-0.448654)	0.697321 / 4.805227 (-4.107906)	0.120474 / 6.500664 (-6.380190)	0.042192 / 0.075469 (-0.033277)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.975876 / 1.841788 (-0.865912)	12.174933 / 8.074308 (4.100625)	10.400906 / 10.191392 (0.209514)	0.162244 / 0.680424 (-0.518180)	0.016443 / 0.534201 (-0.517758)	0.293430 / 0.579283 (-0.285853)	0.285664 / 0.434364 (-0.148700)	0.332322 / 0.540337 (-0.208015)	0.609815 / 1.386936 (-0.777121)

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005155 / 0.011353 (-0.006198)	0.003226 / 0.011008 (-0.007782)	0.062651 / 0.038508 (0.024143)	0.051314 / 0.023109 (0.028205)	0.246075 / 0.275898 (-0.029823)	0.266859 / 0.323480 (-0.056621)	0.003895 / 0.007986 (-0.004091)	0.002462 / 0.004328 (-0.001866)	0.048097 / 0.004250 (0.043846)	0.037313 / 0.037052 (0.000261)	0.253208 / 0.258489 (-0.005281)	0.280255 / 0.293841 (-0.013585)	0.027052 / 0.128546 (-0.101494)	0.010276 / 0.075646 (-0.065370)	0.205663 / 0.419271 (-0.213608)	0.035111 / 0.043533 (-0.008422)	0.253757 / 0.255139 (-0.001382)	0.265466 / 0.283200 (-0.017733)	0.017873 / 0.141683 (-0.123810)	1.118906 / 1.452155 (-0.333249)	1.176384 / 1.492716 (-0.316332)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.094921 / 0.018006 (0.076914)	0.300459 / 0.000490 (0.299970)	0.000214 / 0.000200 (0.000014)	0.000042 / 0.000054 (-0.000012)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.018430 / 0.037411 (-0.018981)	0.062690 / 0.014526 (0.048165)	0.074215 / 0.176557 (-0.102342)	0.119969 / 0.737135 (-0.617166)	0.075846 / 0.296338 (-0.220493)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.273492 / 0.215209 (0.058283)	2.667937 / 2.077655 (0.590282)	1.405912 / 1.504120 (-0.098208)	1.269041 / 1.541195 (-0.272153)	1.313461 / 1.468490 (-0.155029)	0.554633 / 4.584777 (-4.030144)	2.325552 / 3.745712 (-1.420160)	2.825580 / 5.269862 (-2.444282)	1.745432 / 4.565676 (-2.820245)	0.062497 / 0.424275 (-0.361778)	0.004935 / 0.007607 (-0.002673)	0.337045 / 0.226044 (0.111001)	3.246360 / 2.268929 (0.977432)	1.775329 / 55.444624 (-53.669296)	1.491812 / 6.876477 (-5.384665)	1.499783 / 2.142072 (-0.642290)	0.636768 / 4.805227 (-4.168459)	0.116471 / 6.500664 (-6.384193)	0.041838 / 0.075469 (-0.033631)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.937388 / 1.841788 (-0.904400)	11.950930 / 8.074308 (3.876622)	10.532062 / 10.191392 (0.340670)	0.129490 / 0.680424 (-0.550934)	0.013907 / 0.534201 (-0.520294)	0.287503 / 0.579283 (-0.291780)	0.270548 / 0.434364 (-0.163816)	0.324321 / 0.540337 (-0.216016)	0.427639 / 1.386936 (-0.959297)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005272 / 0.011353 (-0.006081)	0.003413 / 0.011008 (-0.007595)	0.049800 / 0.038508 (0.011292)	0.055978 / 0.023109 (0.032868)	0.274365 / 0.275898 (-0.001533)	0.293414 / 0.323480 (-0.030066)	0.003994 / 0.007986 (-0.003992)	0.002480 / 0.004328 (-0.001848)	0.048787 / 0.004250 (0.044537)	0.040520 / 0.037052 (0.003468)	0.276198 / 0.258489 (0.017709)	0.301085 / 0.293841 (0.007244)	0.028352 / 0.128546 (-0.100194)	0.010631 / 0.075646 (-0.065015)	0.057103 / 0.419271 (-0.362168)	0.032277 / 0.043533 (-0.011256)	0.274472 / 0.255139 (0.019333)	0.289953 / 0.283200 (0.006754)	0.018048 / 0.141683 (-0.123635)	1.120329 / 1.452155 (-0.331826)	1.175784 / 1.492716 (-0.316932)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.102519 / 0.018006 (0.084512)	0.322030 / 0.000490 (0.321540)	0.000234 / 0.000200 (0.000034)	0.000045 / 0.000054 (-0.000009)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.023084 / 0.037411 (-0.014327)	0.069592 / 0.014526 (0.055066)	0.081293 / 0.176557 (-0.095264)	0.119546 / 0.737135 (-0.617589)	0.083249 / 0.296338 (-0.213090)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.294997 / 0.215209 (0.079788)	2.925517 / 2.077655 (0.847863)	1.607824 / 1.504120 (0.103705)	1.469586 / 1.541195 (-0.071608)	1.492350 / 1.468490 (0.023860)	0.561351 / 4.584777 (-4.023426)	2.446741 / 3.745712 (-1.298972)	2.842588 / 5.269862 (-2.427273)	1.789189 / 4.565676 (-2.776487)	0.064064 / 0.424275 (-0.360211)	0.005011 / 0.007607 (-0.002597)	0.351059 / 0.226044 (0.125015)	3.485277 / 2.268929 (1.216348)	1.981821 / 55.444624 (-53.462803)	1.671846 / 6.876477 (-5.204631)	1.702014 / 2.142072 (-0.440058)	0.645205 / 4.805227 (-4.160023)	0.117358 / 6.500664 (-6.383306)	0.041633 / 0.075469 (-0.033836)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.963281 / 1.841788 (-0.878506)	12.141256 / 8.074308 (4.066947)	10.595207 / 10.191392 (0.403815)	0.130401 / 0.680424 (-0.550023)	0.015490 / 0.534201 (-0.518710)	0.284201 / 0.579283 (-0.295082)	0.280244 / 0.434364 (-0.154120)	0.323545 / 0.540337 (-0.216792)	0.561246 / 1.386936 (-0.825690)