fix: 🐛 fix migration script

[severo]

See #705 (comment)

[codecov-commenter]

Codecov Report

Base: 90.42% // Head: 83.46% // Decreases project coverage by -6.97% ⚠️

Coverage data is based on head (b35d08d) compared to base (8d890ea).
Patch coverage: 76.66% of modified lines in pull request are covered.

❗ Current head b35d08d differs from pull request most recent head 48dee06. Consider uploading reports for the commit 48dee06 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #707      +/-   ##
==========================================
- Coverage   90.42%   83.46%   -6.97%     
==========================================
  Files          65       14      -51     
  Lines        3163      526    -2637     
==========================================
- Hits         2860      439    -2421     
+ Misses        303       87     -216     

Flag	Coverage Δ
jobs_mongodb_migration	83.46% <76.66%> (?)
services_admin	?
services_api	?
workers_datasets_based	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...grations/_20230126164900_queue_job_add_priority.py	76.27% <76.27%> (ø)
...ngodb_migration/src/mongodb_migration/collector.py	100.00% <100.00%> (ø)
services/admin/src/admin/routes/force_refresh.py
services/api/src/api/routes/processing_step.py
services/api/tests/conftest.py
services/api/tests/routes/test_valid.py
services/api/tests/test_app.py
...orkers/datasets_based/src/datasets_based/config.py
...orkers/datasets_based/src/datasets_based/worker.py
...atasets_based/src/datasets_based/worker_factory.py
... and 70 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[severo]

Migration logs:

INFO: 2023-01-26 18:47:05,041 - root - Start migrations
INFO: 2023-01-26 18:47:05,138 - root - 3 migrations have already been applied. They will be skipped.
INFO: 2023-01-26 18:47:05,139 - root - Migrate 20230126164900: add to the migrations collection
INFO: 2023-01-26 18:47:05,145 - root - Migrate 20230126164900: apply
INFO: 2023-01-26 18:47:05,145 - root - Add the priority field, with the default value ('normal'), to all the jobs

[severo]

Hmmm, it took too much time to process the migration:

$ make upgrade-prod
make[1]: Entering directory '/home/slesage/hf/datasets-server/chart'
helm upgrade --install datasets-server-prod . --values docker-images.yaml --values env/prod.yaml -n datasets-server
Error: UPGRADE FAILED: pre-upgrade hooks failed: timed out waiting for the condition
make[1]: *** [Makefile:18: upgrade] Error 1
make[1]: Leaving directory '/home/slesage/hf/datasets-server/chart'
make: *** [Makefile:42: upgrade-prod] Error 2

But the migration job finished successfully anyway.

I could launch again, successfully:

 ✘ slesage@aws-dev-sylvain-lesage  ~/hf/datasets-server/chart   main  make upgrade-prod
make[1]: Entering directory '/home/slesage/hf/datasets-server/chart'
helm upgrade --install datasets-server-prod . --values docker-images.yaml --values env/prod.yaml -n datasets-server
Release "datasets-server-prod" has been upgraded. Happy Helming!
NAME: datasets-server-prod
LAST DEPLOYED: Thu Jan 26 18:53:56 2023
NAMESPACE: datasets-server
STATUS: deployed
REVISION: 309
TEST SUITE: None
make[1]: Leaving directory '/home/slesage/hf/datasets-server/chart'

The migration logs in the second deploy show that the migration had already been applied:

$ k logs -f datasets-server-prod-job-mongodb-migration-wnxqx
INFO: 2023-01-26 18:54:00,974 - root - Start migrations
INFO: 2023-01-26 18:54:01,075 - root - 4 migrations have already been applied. They will be skipped.
INFO: 2023-01-26 18:54:01,075 - root - All migrations have already been applied.
INFO: 2023-01-26 18:54:01,075 - root - All migrations have been applied

[severo]

The only issue is that maybe some jobs have been created without the "priority" field meanwhile, but it's not crucial at all.

[severo]

That's the case indeed, 12 jobs are stuck in waiting mode because they lack a priority field.

I relaunched them by hand (only four datasets)