Delete a job with dependencies between task will cause the controller to report an error repeatedly

[Christina935]

What happened:
Delete a job with dependencies between task will cause the controller to report an error repeatedly. The error is Failed to get pod taskA-pod-name pod "taskA-pod-name" not found
Then resumit the job, the job will lost the status.

There has actually been a completely diffrent job that also lost status when controller repeatedly reported this error, but I'm not sure if it's the same reason.

What you expected to happen:
job doesn't lose status and controller doesn't report error repeatedly.

How to reproduce it (as minimally and precisely as possible):

1. submit a job with two tasks that are taskA and taskB, taskB depends on taskA.
2. podA of taskA has been created but can't run because of ImagePullBackOff error.
3. delete the job.
4. then resubmit a same job.

Anything else we need to know?:

Environment:

• Volcano Version: 1.5.1
• Kubernetes version (use kubectl version): 1.20.15
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[Thor-wl]

/assign @hwdef

[hwdef]

This is a bug.
repeat: #2284
Temporary workaround: delete volcano-controller pod