Updating dependencies in worker to support first rows from parquet #996
Conversation
|
The documentation is not available anymore as the PR was closed or merged. |
|
Weird but some tests for parquet-and-dataset-info job runner are failing, it looks like it is caused by pyarrow 11. I will investigate locally. |
| CSV_PARQUET_SIZE = 1_865 | ||
| AUDIO_PARQUET_SIZE = 1_383 | ||
| CSV_PARQUET_SIZE = 1_866 | ||
| AUDIO_PARQUET_SIZE = 1_384 |
There was a problem hiding this comment.
Increasing size to have test running successful. Looks like pyarrow 11 increased the size of computed parquet file.
| @@ -328,7 +328,7 @@ def create_dataset_info_response_for_csv(dataset: str, config: str) -> Any: | |||
| } | |||
|
|
|||
|
|
|||
| def create_dataset_info_response_for_audio(dataset: str, config: str) -> Any: | |||
| def create_dataset_info_response_for_audio() -> Any: | |||
There was a problem hiding this comment.
I did this little refactor to make git actions alive again on this PR, nothing related to the logic.
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## main #996 +/- ##
==========================================
+ Coverage 89.58% 91.55% +1.96%
==========================================
Files 147 53 -94
Lines 7854 3740 -4114
==========================================
- Hits 7036 3424 -3612
+ Misses 818 316 -502
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 94 files with indirect coverage changes Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report in Codecov by Sentry. |
Preparation for #988 implementation
As suggested in #988 (comment)
I followed these steps: