chore: Configure Redis priority queues and task routes for Celery

[StanGirard]

This commit configures Redis priority queues and task routes for Celery. It sets up the queue order strategy as "priority" and defines 10 priority levels (0-9). The task routes are also defined, with "process_file_and_notify" having the highest priority (9), "process_crawl_and_notify" having priority 8, and all other tasks having a default priority of 5.

Note: This commit message does not include any issue references, tags, or author names.

Description

Please include a summary of the changes and the related issue. Please also include relevant motivation and context.

Checklist before requesting a review

Please delete options that are not relevant.

• My code follows the style guidelines of this project
• I have performed a self-review of my code
• I have commented hard-to-understand areas
• I have ideally added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged

Screenshots (if appropriate):

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
quivrapp	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 24, 2024 9:11pm

[AmineDiro]

LGTM ! 👍🏼

I read the celery guide on redis message priorities : https://docs.celeryq.dev/en/stable/userguide/routing.html#redis-message-priorities . This note seems important :
"note that this will never be as good as priorities implemented at the broker server level, and may be approximate at best. But it may still be good enough for your application."

Redis doesn't have any understanding of message priority so I hope we don't encounter any unexpected behavior... Also this is pretty hard to test systematically.

[StanGirard]

I think we should postpone a bit but yeah this is a solution :)