Add backfill functionality to check admin #7094

[xmunoz]

• Add backfill task
• Change lookup of checks to check_name instead of id
• Load checks that are also in "evaluation" state

[xmunoz]

Failing on coverage, but I wanted get input about my implementation of the backfill function. There are two ways that this could run:

• Calling the run_task method directly, which causes the backfill task to be long-running, synchronously executing each run_check call.
• Having the backfill task enqueue a bunch of run_check tasks.

I have the first implementation in this PR for a few reasons. On my local machine, there appears to be a huge overhead incurred by enqueue thousands of tasks. To execute a backfill on 10,000 items took about 15 seconds with the first method, and about 5 minutes with the second. This overhead possibly goes away on production. My other reasons for preferring the first method:

• It's easy to kill a single, rogue long-running task, and more difficult to play task whack-a-mole.
• Each run_check executes a few read and at least one write db query. This could potentially cause lock contention on the database, requiring task whack-a-mole...
• There may be a concern about the rate at which certain types of checks run, e.g. ones that query external APIs. Backfill checks executing synchronously means we get rate-limiting for free!

Let me know what you think!

[ewdurbin]

Looks good overall, just one concern.

[ewdurbin]

Sorry, I missed this comment, and the approach in the backfill task itself...

Failing on coverage, but I wanted get input about my implementation of the backfill function. There are two ways that this could run:

• Calling the run_task method directly, which causes the backfill task to be long-running, synchronously executing each run_check call.
• Having the backfill task enqueue a bunch of run_check tasks.

I have the first implementation in this PR for a few reasons. On my local machine, there appears to be a huge overhead incurred by enqueue thousands of tasks. To execute a backfill on 10,000 items took about 15 seconds with the first method, and about 5 minutes with the second. This overhead possibly goes away on production. My other reasons for preferring the first method:

I think the primary reason why the first approach is significantly more performant is that we currently aren't doing anything in any of the checks. I suspect that if a check were preforming more substantial work it would likely be problematic.

The second approach is likely more feasible in the long run, though perhaps not via the web UI due to timeouts. I doubt the overhead would disappear in prod, as enqueueing tasks still require network calls to the queue.

• It's easy to kill a single, rogue long-running task, and more difficult to play task whack-a-mole.
• Each run_check executes a few read and at least one write db query. This could potentially cause lock contention on the database, requiring task whack-a-mole...
• There may be a concern about the rate at which certain types of checks run, e.g. ones that query external APIs. Backfill checks executing synchronously means we get rate-limiting for free!

These are all reasonable benefits of the first approach, but it looks to me like a better approach would be to strip out the ability to do a full backfill from the web and make it a CLI entrypoint under the malware module. Perhaps this view should only allow kicking off evaluations with very conservative corpus sizes.

[xmunoz]

The second approach is likely more feasible in the long run, though perhaps not via the web UI due to timeouts.

I don't understand the concern about the web UI. The admin simply enqueues the backfill task and returns a view to the user. The backfill execution happens asynchronously. I don't think timeouts are a concern.

[ewdurbin]

I don't understand the concern about the web UI. The admin simply enqueues the backfill task and returns a view to the user. The backfill execution happens asynchronously. I don't think timeouts are a concern.

I got mixed up where the iteration was occurring. You're right.

[xmunoz]

After more discussion around scalability, we are removing the ability to run full backfills from the project scope, and limiting the admin triggered backfills to a fixed size for check evaluation purposes. Update pending.