Persistence Agent keep trying to delete workflows that do not exist

[Bobgy]

In #4482, a different problem is that

There are many error messages like

Transient failure while syncing resource (default/train-pipeline-ltd84): CustomError (code: 0): Syncing Workflow (train-pipeline-ltd84): transient failure: CustomError (code: 0): Error while reporting workflow resource (code: Internal, message: Report workflow failed.: InternalServerError: Failed to delete the completed workflow for run 3e649710-8b97-4c64-9f39-a0536d8abf1b: workflows.argoproj.io "train-pipeline-ltd84" not found):

I suspect the workqueue is filled with workflows that have already been deleted, so it cannot move forward with others.

The Fix

KFP API server should mark delete workflow not found errors as permanent, so persistence agent won't need to keep retrying them.