Deprecate Pandas builder

[mariosasko]

The Pandas packaged builder is undocumented and relies on pickle to read the data, making it unsafe. Moreover, I haven't seen a single instance of this builder being used (not even using the GH/Hub search), so we should deprecate it.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[lhoestq]

Sounds good to me

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005301 / 0.011353 (-0.006052)	0.003701 / 0.011008 (-0.007307)	0.065830 / 0.038508 (0.027322)	0.029791 / 0.023109 (0.006682)	0.251676 / 0.275898 (-0.024222)	0.283824 / 0.323480 (-0.039655)	0.003083 / 0.007986 (-0.004903)	0.004144 / 0.004328 (-0.000185)	0.053670 / 0.004250 (0.049419)	0.042020 / 0.037052 (0.004968)	0.266389 / 0.258489 (0.007899)	0.296740 / 0.293841 (0.002900)	0.028320 / 0.128546 (-0.100226)	0.010604 / 0.075646 (-0.065042)	0.219881 / 0.419271 (-0.199390)	0.036216 / 0.043533 (-0.007317)	0.255718 / 0.255139 (0.000579)	0.275808 / 0.283200 (-0.007392)	0.018407 / 0.141683 (-0.123276)	1.140007 / 1.452155 (-0.312148)	1.174005 / 1.492716 (-0.318711)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.091230 / 0.018006 (0.073224)	0.300704 / 0.000490 (0.300215)	0.000207 / 0.000200 (0.000007)	0.000043 / 0.000054 (-0.000011)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.018950 / 0.037411 (-0.018461)	0.062177 / 0.014526 (0.047651)	0.073968 / 0.176557 (-0.102589)	0.122161 / 0.737135 (-0.614974)	0.075001 / 0.296338 (-0.221338)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.285675 / 0.215209 (0.070466)	2.794176 / 2.077655 (0.716522)	1.478666 / 1.504120 (-0.025454)	1.361843 / 1.541195 (-0.179351)	1.383847 / 1.468490 (-0.084643)	0.568610 / 4.584777 (-4.016167)	2.402351 / 3.745712 (-1.343361)	2.860772 / 5.269862 (-2.409089)	1.768588 / 4.565676 (-2.797089)	0.063257 / 0.424275 (-0.361018)	0.004998 / 0.007607 (-0.002609)	0.340897 / 0.226044 (0.114853)	3.340238 / 2.268929 (1.071310)	1.836434 / 55.444624 (-53.608190)	1.556844 / 6.876477 (-5.319633)	1.610685 / 2.142072 (-0.531388)	0.644941 / 4.805227 (-4.160286)	0.117593 / 6.500664 (-6.383072)	0.042803 / 0.075469 (-0.032666)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.979181 / 1.841788 (-0.862607)	11.901365 / 8.074308 (3.827057)	9.587943 / 10.191392 (-0.603449)	0.139648 / 0.680424 (-0.540776)	0.013904 / 0.534201 (-0.520297)	0.291249 / 0.579283 (-0.288034)	0.260737 / 0.434364 (-0.173627)	0.326000 / 0.540337 (-0.214338)	0.433459 / 1.386936 (-0.953477)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005503 / 0.011353 (-0.005850)	0.003738 / 0.011008 (-0.007270)	0.049137 / 0.038508 (0.010629)	0.031484 / 0.023109 (0.008374)	0.265783 / 0.275898 (-0.010115)	0.295125 / 0.323480 (-0.028354)	0.004074 / 0.007986 (-0.003911)	0.002707 / 0.004328 (-0.001622)	0.048340 / 0.004250 (0.044089)	0.045453 / 0.037052 (0.008401)	0.276500 / 0.258489 (0.018011)	0.312002 / 0.293841 (0.018162)	0.029139 / 0.128546 (-0.099408)	0.010445 / 0.075646 (-0.065201)	0.057486 / 0.419271 (-0.361785)	0.052386 / 0.043533 (0.008853)	0.267099 / 0.255139 (0.011960)	0.283193 / 0.283200 (-0.000007)	0.018368 / 0.141683 (-0.123315)	1.136207 / 1.452155 (-0.315948)	1.178418 / 1.492716 (-0.314298)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.089270 / 0.018006 (0.071264)	0.301087 / 0.000490 (0.300598)	0.000208 / 0.000200 (0.000008)	0.000050 / 0.000054 (-0.000005)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.021991 / 0.037411 (-0.015421)	0.075357 / 0.014526 (0.060831)	0.087781 / 0.176557 (-0.088775)	0.126923 / 0.737135 (-0.610212)	0.088491 / 0.296338 (-0.207847)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.293653 / 0.215209 (0.078444)	2.872156 / 2.077655 (0.794501)	1.559229 / 1.504120 (0.055109)	1.441201 / 1.541195 (-0.099993)	1.472642 / 1.468490 (0.004152)	0.588463 / 4.584777 (-3.996314)	2.447685 / 3.745712 (-1.298028)	2.755752 / 5.269862 (-2.514110)	1.796591 / 4.565676 (-2.769086)	0.068024 / 0.424275 (-0.356252)	0.005148 / 0.007607 (-0.002459)	0.343572 / 0.226044 (0.117528)	3.347856 / 2.268929 (1.078927)	1.945977 / 55.444624 (-53.498647)	1.648953 / 6.876477 (-5.227524)	1.804468 / 2.142072 (-0.337604)	0.651034 / 4.805227 (-4.154193)	0.118130 / 6.500664 (-6.382534)	0.041019 / 0.075469 (-0.034450)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.020461 / 1.841788 (-0.821327)	12.514237 / 8.074308 (4.439929)	10.696276 / 10.191392 (0.504884)	0.154549 / 0.680424 (-0.525874)	0.015964 / 0.534201 (-0.518237)	0.290392 / 0.579283 (-0.288891)	0.276074 / 0.434364 (-0.158290)	0.326253 / 0.540337 (-0.214085)	0.440383 / 1.386936 (-0.946553)