add allow_primitive_to_str and allow_decimal_to_str instead of allow_number_to_str

[Modexus]

Fix #6805

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[Modexus]

@mariosasko pytest seems to be missing on windows?

[mariosasko]

CI is not behaving well today 🙂

[mariosasko]

Thanks! A few comments:

[mariosasko]

LGTM! Also, thanks for adding tests!

[mariosasko]

I couldn't find an instance of the allow_number_to_str parameter (or array_cast/cast_array_to_feature more generally) being used in the wild. So, I think simply removing allow_number_to_str instead of deprecating it should be fine, considering array_cast/cast_array_to_feature are somewhat hidden. Do you agree @lhoestq?

[lhoestq]

Yup we can remove without any deprecation cycle

[lhoestq]

Nice fix !

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005253 / 0.011353 (-0.006100)	0.003767 / 0.011008 (-0.007241)	0.064599 / 0.038508 (0.026091)	0.030758 / 0.023109 (0.007649)	0.237437 / 0.275898 (-0.038461)	0.277580 / 0.323480 (-0.045900)	0.004220 / 0.007986 (-0.003766)	0.002738 / 0.004328 (-0.001591)	0.049393 / 0.004250 (0.045143)	0.045283 / 0.037052 (0.008231)	0.249907 / 0.258489 (-0.008582)	0.283301 / 0.293841 (-0.010540)	0.027722 / 0.128546 (-0.100825)	0.010842 / 0.075646 (-0.064804)	0.219197 / 0.419271 (-0.200074)	0.036449 / 0.043533 (-0.007084)	0.237774 / 0.255139 (-0.017365)	0.257981 / 0.283200 (-0.025218)	0.018098 / 0.141683 (-0.123585)	1.161778 / 1.452155 (-0.290376)	1.212707 / 1.492716 (-0.280010)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.096462 / 0.018006 (0.078456)	0.305322 / 0.000490 (0.304832)	0.000218 / 0.000200 (0.000018)	0.000048 / 0.000054 (-0.000006)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.018438 / 0.037411 (-0.018973)	0.061633 / 0.014526 (0.047107)	0.073678 / 0.176557 (-0.102879)	0.122033 / 0.737135 (-0.615103)	0.074846 / 0.296338 (-0.221493)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.279564 / 0.215209 (0.064355)	2.756984 / 2.077655 (0.679330)	1.486525 / 1.504120 (-0.017595)	1.366474 / 1.541195 (-0.174721)	1.370192 / 1.468490 (-0.098298)	0.576940 / 4.584777 (-4.007837)	2.414088 / 3.745712 (-1.331624)	2.788423 / 5.269862 (-2.481439)	1.738695 / 4.565676 (-2.826982)	0.064456 / 0.424275 (-0.359819)	0.005536 / 0.007607 (-0.002071)	0.337266 / 0.226044 (0.111222)	3.327140 / 2.268929 (1.058212)	1.837553 / 55.444624 (-53.607072)	1.538955 / 6.876477 (-5.337521)	1.575624 / 2.142072 (-0.566448)	0.639960 / 4.805227 (-4.165267)	0.117607 / 6.500664 (-6.383057)	0.042077 / 0.075469 (-0.033393)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.960488 / 1.841788 (-0.881300)	11.565280 / 8.074308 (3.490972)	9.702633 / 10.191392 (-0.488759)	0.139106 / 0.680424 (-0.541318)	0.013601 / 0.534201 (-0.520600)	0.291499 / 0.579283 (-0.287784)	0.277433 / 0.434364 (-0.156930)	0.325700 / 0.540337 (-0.214637)	0.421036 / 1.386936 (-0.965900)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005405 / 0.011353 (-0.005948)	0.003816 / 0.011008 (-0.007192)	0.050422 / 0.038508 (0.011914)	0.030473 / 0.023109 (0.007364)	0.275975 / 0.275898 (0.000077)	0.298002 / 0.323480 (-0.025478)	0.004280 / 0.007986 (-0.003706)	0.002746 / 0.004328 (-0.001583)	0.049649 / 0.004250 (0.045398)	0.040675 / 0.037052 (0.003623)	0.287496 / 0.258489 (0.029007)	0.315140 / 0.293841 (0.021299)	0.029835 / 0.128546 (-0.098711)	0.010443 / 0.075646 (-0.065204)	0.058299 / 0.419271 (-0.360972)	0.032944 / 0.043533 (-0.010588)	0.279468 / 0.255139 (0.024329)	0.296336 / 0.283200 (0.013136)	0.018572 / 0.141683 (-0.123111)	1.177622 / 1.452155 (-0.274532)	1.238240 / 1.492716 (-0.254477)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.091867 / 0.018006 (0.073861)	0.299982 / 0.000490 (0.299492)	0.000217 / 0.000200 (0.000017)	0.000043 / 0.000054 (-0.000011)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.022649 / 0.037411 (-0.014762)	0.074948 / 0.014526 (0.060422)	0.087949 / 0.176557 (-0.088607)	0.125875 / 0.737135 (-0.611261)	0.089295 / 0.296338 (-0.207044)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.290387 / 0.215209 (0.075178)	2.820969 / 2.077655 (0.743315)	1.614607 / 1.504120 (0.110487)	1.496959 / 1.541195 (-0.044236)	1.526475 / 1.468490 (0.057985)	0.570087 / 4.584777 (-4.014690)	2.423106 / 3.745712 (-1.322606)	2.825321 / 5.269862 (-2.444540)	1.765580 / 4.565676 (-2.800097)	0.063289 / 0.424275 (-0.360986)	0.005456 / 0.007607 (-0.002151)	0.344100 / 0.226044 (0.118055)	3.395733 / 2.268929 (1.126804)	1.951794 / 55.444624 (-53.492830)	1.677689 / 6.876477 (-5.198787)	1.684448 / 2.142072 (-0.457624)	0.644343 / 4.805227 (-4.160885)	0.115796 / 6.500664 (-6.384868)	0.041052 / 0.075469 (-0.034417)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.031487 / 1.841788 (-0.810301)	12.116156 / 8.074308 (4.041848)	10.472247 / 10.191392 (0.280855)	0.142934 / 0.680424 (-0.537490)	0.015470 / 0.534201 (-0.518731)	0.290402 / 0.579283 (-0.288882)	0.272594 / 0.434364 (-0.161770)	0.328311 / 0.540337 (-0.212027)	0.424694 / 1.386936 (-0.962242)