Convert the /backfill endpoint to a kubernetes Job run periodically

[severo]

See #708 (comment) and following comments.

Maybe we should not have this endpoint as such. See the related issue: #736. Maybe #736 should be fixed first, then see how we implement a backfill trigger.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[severo]

Changed the title: what we need is to convert this endpoint to a "cron" job

[AndreaFrancis]

I would suggest to split this issue in small ones:

1. Make /backfill a one time job (like cache_refresh and mongo_migration https://github.com/huggingface/datasets-server/tree/main/jobs) to run each deploy (keep /backfill logic as it is initially).
2. Make backfill k8s job to only append Jobs in order to "complete" missing cache entries (According to current processing graph). This could be done by iterating each supported dataset and verifying its missing cache by processing step (Could be heavy) or we could store somewhere the processing graph steps, and compare it with the current one, in case of new steps or newer job runner version, backfill by step (This could probably work as well to replace cache-refresh job).
3. Make backfill cron job that will run every x time

[AndreaFrancis]

Tasks:

• Make /backfill a one time job (like cache_refresh and mongo_migration https://github.com/huggingface/datasets-server/tree/main/jobs) to run each deploy (keep /backfill logic as it is initially).
• Make backfill k8s job to only append Jobs in order to "complete" missing cache entries (According to current processing graph). This could be done by iterating each supported dataset and verifying its missing cache by processing step (Could be heavy) or we could store somewhere the processing graph steps, and compare it with the current one, in case of new steps or newer job runner version, backfill by step (This could probably work as well to replace cache-refresh job).
• Make backfill cron job that will run every x time

[severo]

related issue: #844
Ideally, the backfill should take care of removing the dangling assets/cache entries/jobs/parquet files

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[severo]

Done