Add CLI function to convert script-dataset to Parquet

[albertvillanova]

Close #6690.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[albertvillanova]

@huggingface/datasets once this PR is merged, I would suggest making a release. Do you agree?

[lhoestq]

LGTM :)

[lhoestq]

Maybe add trust_remote_code ? And maybe a --trust_remote_code arg to the cli command

(also in get_dataset_config_names and get_dataset_default_config_name)

[albertvillanova]

Yes, otherwise we are raising warnings now and eventually raise an error.

[albertvillanova]

Done.

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005367 / 0.011353 (-0.005986)	0.003161 / 0.011008 (-0.007847)	0.063259 / 0.038508 (0.024751)	0.030550 / 0.023109 (0.007441)	0.243789 / 0.275898 (-0.032109)	0.262474 / 0.323480 (-0.061006)	0.003157 / 0.007986 (-0.004829)	0.002586 / 0.004328 (-0.001742)	0.049336 / 0.004250 (0.045085)	0.046434 / 0.037052 (0.009382)	0.249142 / 0.258489 (-0.009347)	0.282953 / 0.293841 (-0.010888)	0.027881 / 0.128546 (-0.100666)	0.010069 / 0.075646 (-0.065578)	0.207937 / 0.419271 (-0.211334)	0.036005 / 0.043533 (-0.007528)	0.251850 / 0.255139 (-0.003288)	0.265156 / 0.283200 (-0.018044)	0.019780 / 0.141683 (-0.121903)	1.124301 / 1.452155 (-0.327853)	1.177392 / 1.492716 (-0.315324)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.091045 / 0.018006 (0.073039)	0.301258 / 0.000490 (0.300769)	0.000214 / 0.000200 (0.000014)	0.000048 / 0.000054 (-0.000006)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.018726 / 0.037411 (-0.018686)	0.061623 / 0.014526 (0.047097)	0.073905 / 0.176557 (-0.102651)	0.119444 / 0.737135 (-0.617692)	0.074614 / 0.296338 (-0.221725)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.287313 / 0.215209 (0.072104)	2.772864 / 2.077655 (0.695209)	1.465267 / 1.504120 (-0.038853)	1.343666 / 1.541195 (-0.197528)	1.329390 / 1.468490 (-0.139100)	0.570222 / 4.584777 (-4.014555)	2.421835 / 3.745712 (-1.323877)	2.747282 / 5.269862 (-2.522579)	1.728733 / 4.565676 (-2.836943)	0.063671 / 0.424275 (-0.360604)	0.005343 / 0.007607 (-0.002264)	0.335078 / 0.226044 (0.109033)	3.334305 / 2.268929 (1.065376)	1.779496 / 55.444624 (-53.665129)	1.496475 / 6.876477 (-5.380002)	1.507848 / 2.142072 (-0.634224)	0.653653 / 4.805227 (-4.151575)	0.118373 / 6.500664 (-6.382291)	0.041727 / 0.075469 (-0.033742)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.981985 / 1.841788 (-0.859803)	11.290978 / 8.074308 (3.216670)	9.499217 / 10.191392 (-0.692175)	0.131353 / 0.680424 (-0.549071)	0.014416 / 0.534201 (-0.519785)	0.288381 / 0.579283 (-0.290902)	0.265483 / 0.434364 (-0.168880)	0.323438 / 0.540337 (-0.216900)	0.417946 / 1.386936 (-0.968990)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005272 / 0.011353 (-0.006081)	0.003551 / 0.011008 (-0.007457)	0.050173 / 0.038508 (0.011665)	0.031291 / 0.023109 (0.008182)	0.278658 / 0.275898 (0.002760)	0.301812 / 0.323480 (-0.021668)	0.004237 / 0.007986 (-0.003748)	0.002713 / 0.004328 (-0.001615)	0.049483 / 0.004250 (0.045233)	0.039995 / 0.037052 (0.002943)	0.293101 / 0.258489 (0.034612)	0.319956 / 0.293841 (0.026116)	0.029127 / 0.128546 (-0.099419)	0.010247 / 0.075646 (-0.065400)	0.057929 / 0.419271 (-0.361342)	0.032942 / 0.043533 (-0.010591)	0.281677 / 0.255139 (0.026538)	0.297937 / 0.283200 (0.014737)	0.018285 / 0.141683 (-0.123398)	1.272858 / 1.452155 (-0.179297)	1.213375 / 1.492716 (-0.279342)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.091110 / 0.018006 (0.073104)	0.302589 / 0.000490 (0.302099)	0.000214 / 0.000200 (0.000014)	0.000070 / 0.000054 (0.000015)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.021520 / 0.037411 (-0.015891)	0.075013 / 0.014526 (0.060487)	0.088695 / 0.176557 (-0.087862)	0.128281 / 0.737135 (-0.608854)	0.090611 / 0.296338 (-0.205727)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.297457 / 0.215209 (0.082248)	2.928612 / 2.077655 (0.850957)	1.613245 / 1.504120 (0.109125)	1.485263 / 1.541195 (-0.055931)	1.496885 / 1.468490 (0.028395)	0.570120 / 4.584777 (-4.014657)	2.487532 / 3.745712 (-1.258180)	2.761552 / 5.269862 (-2.508309)	1.731864 / 4.565676 (-2.833812)	0.062989 / 0.424275 (-0.361286)	0.005428 / 0.007607 (-0.002179)	0.354932 / 0.226044 (0.128888)	3.524475 / 2.268929 (1.255547)	1.977684 / 55.444624 (-53.466941)	1.692568 / 6.876477 (-5.183909)	1.673003 / 2.142072 (-0.469069)	0.643976 / 4.805227 (-4.161251)	0.116499 / 6.500664 (-6.384165)	0.040772 / 0.075469 (-0.034697)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.020354 / 1.841788 (-0.821434)	12.143991 / 8.074308 (4.069683)	10.354058 / 10.191392 (0.162666)	0.145460 / 0.680424 (-0.534964)	0.015356 / 0.534201 (-0.518845)	0.307190 / 0.579283 (-0.272093)	0.276664 / 0.434364 (-0.157699)	0.350068 / 0.540337 (-0.190269)	0.440824 / 1.386936 (-0.946112)