Split first rows from parquet new Job Runner

[AndreaFrancis]

Final part of #755
Based on #875 for parquet reading logic

[codecov-commenter]

Codecov Report

Patch coverage: 100.00% and project coverage change: -1.77 ⚠️

Comparison is base (4788650) 89.58% compared to head (0e87d4f) 87.81%.

❗ Current head 0e87d4f differs from pull request most recent head b2f428b. Consider uploading reports for the commit b2f428b to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #988      +/-   ##
==========================================
- Coverage   89.58%   87.81%   -1.77%     
==========================================
  Files         147       94      -53     
  Lines        7854     4121    -3733     
==========================================
- Hits         7036     3619    -3417     
+ Misses        818      502     -316     

Flag	Coverage Δ
jobs_cache_refresh	98.50% <ø> (ø)
jobs_mongodb_migration	80.57% <ø> (ø)
libs_libcommon	93.57% <100.00%> (+0.03%)	⬆️
services_admin	87.32% <ø> (ø)
services_api	84.70% <100.00%> (ø)
services_worker	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
libs/libcommon/src/libcommon/config.py	78.33% <ø> (ø)
services/api/src/api/config.py	100.00% <ø> (ø)
libs/libcommon/src/libcommon/constants.py	100.00% <100.00%> (ø)
libs/libcommon/tests/test_processing_steps.py	100.00% <100.00%> (ø)
services/api/tests/routes/test_endpoint.py	100.00% <100.00%> (ø)

... and 53 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[HuggingFaceDocBuilder]

The documentation is not available anymore as the PR was closed or merged.

[AndreaFrancis]

Not sure why pip audit fails with worker project, it is now different from other folders that use pymongo and mongoengine

ERROR:pip_audit._virtual_env:internal pip failure: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    pymongo<5.0,>=3.4 from https://files.pythonhosted.org/packages/38/68/928d7ce22719cfa255fb973b34aed6f04ac3ea89049ce69e3b092c30a60f/pymongo-4.3.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (from mongoengine==0.24.2->-r /tmp/tmpmnbps2mm (line 1032))

[severo]

Not sure why pip audit fails with worker project, it is now different from other folders that use pymongo and mongoengine

ERROR:pip_audit._virtual_env:internal pip failure: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    pymongo<5.0,>=3.4 from https://files.pythonhosted.org/packages/38/68/928d7ce22719cfa255fb973b34aed6f04ac3ea89049ce69e3b092c30a60f/pymongo-4.3.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (from mongoengine==0.24.2->-r /tmp/tmpmnbps2mm (line 1032))

In 12289d8, you updated all the dependencies of the project, including some that were not related to the matter of this PR. It's better to do it in a separate PR, where we update the dependencies and ensure nothing has broken.

In this case, you updated the dev dependency pip-audit to https://github.com/pypa/pip-audit/releases/tag/v2.5.3, which introduced the issue (I tried with 2.5.2, and it works).

So, my recommendation for this PR would be to:

• copy/paste the version of main of files pypoetry.toml and poetry.lock
• run make install
• then poetry remove apache-beam
• then restore the new version (https://github.com/huggingface/datasets-server/pull/988/files#diff-5def9e9ad886d364c3edccbe42f21d5aafd1b936d6756a71fa9be4f86ecf573e) of pyproject.toml and call "update" for all the updated dependencies, one by one:
  • poetry update datasets
  • poetry update hffs
  • poetry update pyarrow

and I think that your poetry.lock should be good (in particular, you should be using pip-audit v2.4.14)

And then, in another PR, we could upgrade the dependencies, one by one, and checking that we don't break anything.

[AndreaFrancis]

And then, in another PR, we could upgrade the dependencies, one by one, and checking that we don't break anything.

I introduced dependencies for this PR on #996

[severo]

Very nice. This, along with #995, should help reduce a lot the load on the workers!

[AndreaFrancis]

I will introduce same logic as #995 for first rows job runners

[severo]

You're right, #995 was for split-names. Let's do it for first-rows now!