Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The job task resync logic is not right #384

Closed
hzxuzhonghu opened this issue Jul 24, 2019 · 0 comments · Fixed by #385
Closed

The job task resync logic is not right #384

hzxuzhonghu opened this issue Jul 24, 2019 · 0 comments · Fixed by #385
Labels
area/controllers kind/bug Categorizes issue or PR as related to a bug. priority/high

Comments

@hzxuzhonghu
Copy link
Collaborator

/kind bug

When i deleted several pods of volcano job, i see too many errors

E0724 01:27:37.980526       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:37.984007       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
I0724 01:27:37.985337       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-4> of Job <default/test-job3>
I0724 01:27:37.985356       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-5> of Job <default/test-job3>
E0724 01:27:37.986466       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
I0724 01:27:37.986642       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-0> of Job <default/test-job3>
I0724 01:27:37.989292       1 job_controller_actions.go:349] Finished Job <default/test-job3> sync up
E0724 01:27:37.989859       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:37.991320       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:37.992678       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:37.994137       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:37.995630       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:37.996875       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:37.998123       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:37.999642       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.001281       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.002655       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.005832       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.007604       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.009749       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.012752       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.014255       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.018370       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.021022       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.022929       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.024827       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.028570       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.030376       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.032411       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.054733       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.056256       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.059800       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.062486       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.064431       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.066314       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.070034       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.071687       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.073770       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.136685       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.138391       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.141394       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.143954       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.145752       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.147767       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.151643       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.152966       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.155177       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.298918       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.300686       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.302956       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.305450       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.307124       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.309255       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.313123       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.314624       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.316636       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
I0724 01:27:38.360519       1 job_controller.go:295] Try to handle request <Job: default/test-job3, Task:default-nginx, Event:PodEvicted, ExitCode:0, Action:, JobVersion: 0>
I0724 01:27:38.362306       1 job_controller_util.go:138] Request Job: default/test-job3, Task:default-nginx, Event:PodEvicted, ExitCode:0, Action:, JobVersion: 0 is outdated, will perform sync instead.
I0724 01:27:38.362367       1 job_controller.go:312] Execute <SyncJob> on Job <default/test-job3> in <Inqueue> by <*state.inqueueState>.
I0724 01:27:38.362388       1 job_controller_actions.go:200] Starting to sync up Job <default/test-job3>
I0724 01:27:38.362413       1 job_controller_actions.go:204] Current Version is: 2 of job: default/test-job3
I0724 01:27:38.362582       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:38.362651       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:38.362711       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:38.362759       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:38.362793       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:38.362812       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:38.362831       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:38.362847       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:38.362879       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:38.367274       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-5> of Job <default/test-job3>
I0724 01:27:38.369116       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-0> of Job <default/test-job3>
I0724 01:27:38.369309       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-4> of Job <default/test-job3>
E0724 01:27:38.375256       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
I0724 01:27:38.375573       1 job_controller_actions.go:349] Finished Job <default/test-job3> sync up
I0724 01:27:38.376551       1 job_controller.go:295] Try to handle request <Job: default/test-job3, Task:default-nginx, Event:PodEvicted, ExitCode:0, Action:, JobVersion: 0>
I0724 01:27:38.376599       1 job_controller_util.go:138] Request Job: default/test-job3, Task:default-nginx, Event:PodEvicted, ExitCode:0, Action:, JobVersion: 0 is outdated, will perform sync instead.
I0724 01:27:38.376617       1 job_controller.go:312] Execute <SyncJob> on Job <default/test-job3> in <Inqueue> by <*state.inqueueState>.
I0724 01:27:38.376634       1 job_controller_actions.go:200] Starting to sync up Job <default/test-job3>
I0724 01:27:38.376670       1 job_controller_actions.go:204] Current Version is: 2 of job: default/test-job3
I0724 01:27:38.376699       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:38.376714       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:38.376720       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:38.376731       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:38.376740       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:38.376745       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:38.376773       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:38.376782       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:38.376787       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
E0724 01:27:38.377810       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.380311       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
I0724 01:27:38.382917       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-4> of Job <default/test-job3>
I0724 01:27:38.382992       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-0> of Job <default/test-job3>
I0724 01:27:38.383030       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-5> of Job <default/test-job3>
I0724 01:27:38.385520       1 job_controller_actions.go:349] Finished Job <default/test-job3> sync up
E0724 01:27:38.386814       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.389489       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.390835       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.392569       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.394562       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.396442       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.402527       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.403972       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.406115       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.408370       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.410995       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.417899       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.423943       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.425382       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.427496       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.449767       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.452509       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.459443       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.465388       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.466756       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.468831       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.531792       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.534176       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.540979       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.547070       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.548394       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.552692       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.620892       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.622410       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.624744       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.626849       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.628675       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.630753       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.634712       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.636217       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.638155       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.693954       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:38.696049       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.702660       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.708647       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:38.710255       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:38.714836       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
I0724 01:27:38.959630       1 job_controller.go:295] Try to handle request <Job: default/test-job3, Task:default-nginx, Event:OutOfSync, ExitCode:0, Action:, JobVersion: 2>
I0724 01:27:38.959736       1 job_controller.go:312] Execute <SyncJob> on Job <default/test-job3> in <Inqueue> by <*state.inqueueState>.
I0724 01:27:38.959753       1 job_controller_actions.go:200] Starting to sync up Job <default/test-job3>
I0724 01:27:38.959849       1 job_controller_actions.go:204] Current Version is: 2 of job: default/test-job3
I0724 01:27:38.959883       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:38.959908       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:38.960327       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:38.960362       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:38.960403       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:38.960419       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:38.960437       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:38.960443       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:38.960446       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:38.974735       1 queue_controller.go:141] Begin sync queue default
I0724 01:27:38.974759       1 queue_controller.go:184] queue default jobs pending 0, running 1, unknown 0
I0724 01:27:38.981630       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-0> of Job <default/test-job3>
I0724 01:27:38.985997       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-4> of Job <default/test-job3>
I0724 01:27:38.986191       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-5> of Job <default/test-job3>
I0724 01:27:38.993089       1 job_controller_actions.go:349] Finished Job <default/test-job3> sync up
I0724 01:27:38.993118       1 job_controller.go:295] Try to handle request <Job: default/test-job3, Task:, Event:, ExitCode:0, Action:, JobVersion: 0>
I0724 01:27:38.993134       1 job_controller_util.go:138] Request Job: default/test-job3, Task:, Event:, ExitCode:0, Action:, JobVersion: 0 is outdated, will perform sync instead.
I0724 01:27:38.993221       1 job_controller.go:312] Execute <SyncJob> on Job <default/test-job3> in <Inqueue> by <*state.inqueueState>.
I0724 01:27:38.993328       1 job_controller_actions.go:200] Starting to sync up Job <default/test-job3>
I0724 01:27:38.993421       1 job_controller_actions.go:204] Current Version is: 2 of job: default/test-job3
I0724 01:27:38.993502       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:38.993547       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:38.993621       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:38.993970       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:38.993984       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:38.993991       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:38.993999       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:38.994056       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:38.994067       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:39.001406       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-4> of Job <default/test-job3>
I0724 01:27:39.002708       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-0> of Job <default/test-job3>
I0724 01:27:39.002924       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-5> of Job <default/test-job3>
I0724 01:27:39.005152       1 job_controller_actions.go:349] Finished Job <default/test-job3> sync up
I0724 01:27:39.005176       1 job_controller.go:295] Try to handle request <Job: default/test-job3, Task:default-nginx, Event:OutOfSync, ExitCode:0, Action:, JobVersion: 2>
I0724 01:27:39.005190       1 job_controller.go:312] Execute <SyncJob> on Job <default/test-job3> in <Inqueue> by <*state.inqueueState>.
I0724 01:27:39.005210       1 job_controller_actions.go:200] Starting to sync up Job <default/test-job3>
I0724 01:27:39.005251       1 job_controller_actions.go:204] Current Version is: 2 of job: default/test-job3
I0724 01:27:39.005287       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:39.005315       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:39.005340       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:39.005382       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:39.005392       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:39.005397       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:39.005405       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:39.005411       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:39.005439       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:39.008724       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-5> of Job <default/test-job3>
I0724 01:27:39.011724       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-0> of Job <default/test-job3>
I0724 01:27:39.011731       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-4> of Job <default/test-job3>
I0724 01:27:39.014872       1 job_controller_actions.go:349] Finished Job <default/test-job3> sync up
E0724 01:27:39.016559       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:39.019339       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:39.061866       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:39.162028       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:39.262304       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:39.264299       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:39.265805       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:39.267156       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:39.268923       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:39.270320       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:39.273563       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:39.284362       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:39.287309       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:39.288883       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:39.362586       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:39.462377       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:39.562188       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:39.664435       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:39.762053       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:39.862046       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:39.962108       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:40.061825       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:40.161979       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:40.261924       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:40.362164       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:40.461918       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:40.562077       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
I0724 01:27:40.658250       1 job_controller.go:295] Try to handle request <Job: default/test-job3, Task:default-nginx, Event:OutOfSync, ExitCode:0, Action:, JobVersion: 2>
I0724 01:27:40.658314       1 job_controller.go:312] Execute <SyncJob> on Job <default/test-job3> in <Inqueue> by <*state.inqueueState>.
I0724 01:27:40.658334       1 job_controller_actions.go:200] Starting to sync up Job <default/test-job3>
I0724 01:27:40.658359       1 job_controller_actions.go:204] Current Version is: 2 of job: default/test-job3
I0724 01:27:40.658409       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:40.658426       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:40.658441       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:40.658463       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:40.658476       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:40.658491       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:40.658509       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:40.658523       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:40.658536       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
E0724 01:27:40.665204       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
I0724 01:27:40.669913       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-0> of Job <default/test-job3>
I0724 01:27:40.671038       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-4> of Job <default/test-job3>
I0724 01:27:40.673972       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-5> of Job <default/test-job3>
I0724 01:27:40.683646       1 job_controller_actions.go:349] Finished Job <default/test-job3> sync up
I0724 01:27:40.683698       1 job_controller.go:295] Try to handle request <Job: default/test-job3, Task:default-nginx, Event:OutOfSync, ExitCode:0, Action:, JobVersion: 2>
I0724 01:27:40.683717       1 job_controller.go:312] Execute <SyncJob> on Job <default/test-job3> in <Inqueue> by <*state.inqueueState>.
I0724 01:27:40.683807       1 job_controller_actions.go:200] Starting to sync up Job <default/test-job3>
I0724 01:27:40.683887       1 job_controller_actions.go:204] Current Version is: 2 of job: default/test-job3
I0724 01:27:40.686066       1 job_controller_handler.go:91] Job update event is ignored since no update in 'Spec'.
I0724 01:27:40.687095       1 garbagecollector.go:107] Updating job default/test-job3
I0724 01:27:40.687537       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:40.687614       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:40.687665       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:40.687736       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:40.687777       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:40.687818       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:40.688292       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:40.688309       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:40.688314       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:40.692972       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-0> of Job <default/test-job3>
I0724 01:27:40.695007       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-5> of Job <default/test-job3>
I0724 01:27:40.695022       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-4> of Job <default/test-job3>
I0724 01:27:40.699559       1 job_controller_actions.go:349] Finished Job <default/test-job3> sync up
I0724 01:27:40.700264       1 garbagecollector.go:107] Updating job default/test-job3
I0724 01:27:40.700306       1 job_controller_handler.go:91] Job update event is ignored since no update in 'Spec'.
I0724 01:27:40.755421       1 job_controller.go:295] Try to handle request <Job: default/test-job3, Task:default-nginx, Event:OutOfSync, ExitCode:0, Action:, JobVersion: 2>
I0724 01:27:40.755470       1 job_controller.go:312] Execute <SyncJob> on Job <default/test-job3> in <Inqueue> by <*state.inqueueState>.
I0724 01:27:40.755501       1 job_controller_actions.go:200] Starting to sync up Job <default/test-job3>
I0724 01:27:40.755515       1 job_controller_actions.go:204] Current Version is: 2 of job: default/test-job3
I0724 01:27:40.755556       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:40.755672       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:40.755808       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:40.755842       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:40.755857       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:40.755892       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:40.755917       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:40.755930       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:40.755942       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:40.763335       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-0> of Job <default/test-job3>
I0724 01:27:40.763335       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-4> of Job <default/test-job3>
I0724 01:27:40.763427       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-5> of Job <default/test-job3>
E0724 01:27:40.763718       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
I0724 01:27:40.767425       1 garbagecollector.go:107] Updating job default/test-job3
I0724 01:27:40.768696       1 job_controller_actions.go:349] Finished Job <default/test-job3> sync up
I0724 01:27:40.768846       1 job_controller.go:295] Try to handle request <Job: default/test-job3, Task:, Event:OutOfSync, ExitCode:0, Action:, JobVersion: 0>
I0724 01:27:40.769003       1 job_controller.go:312] Execute <SyncJob> on Job <default/test-job3> in <Running> by <*state.runningState>.
I0724 01:27:40.769032       1 job_controller_actions.go:200] Starting to sync up Job <default/test-job3>
I0724 01:27:40.769053       1 job_controller_actions.go:204] Current Version is: 2 of job: default/test-job3
I0724 01:27:40.769074       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:40.769089       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:40.769104       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:40.769128       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:40.769149       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:40.769164       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:40.769182       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: env on job: <default/test-job3>
I0724 01:27:40.769194       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: ssh on job: <default/test-job3>
I0724 01:27:40.769216       1 job_controller_plugins.go:40] Starting to execute plugin at <pluginOnPodCreate>: svc on job: <default/test-job3>
I0724 01:27:40.775287       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-5> of Job <default/test-job3>
I0724 01:27:40.777404       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-0> of Job <default/test-job3>
I0724 01:27:40.777459       1 job_controller_actions.go:273] Created Task <test-job3-default-nginx-4> of Job <default/test-job3>
I0724 01:27:40.780248       1 job_controller_actions.go:349] Finished Job <default/test-job3> sync up
E0724 01:27:40.861923       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:40.961956       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:41.061717       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:41.161968       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:41.262458       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:41.362189       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:41.462186       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:41.562085       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:41.662156       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:41.762104       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:41.862073       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:41.962265       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:42.061912       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:42.162277       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:42.262132       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:42.362207       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:42.462064       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:42.562144       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:42.662230       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:42.762120       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:42.861985       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:42.962093       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:43.061985       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:43.162004       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:43.262080       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:43.362057       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:43.462070       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:43.562228       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:43.662153       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:43.762370       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:43.862050       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:43.962180       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:44.061882       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:44.162066       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:44.262120       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:44.362175       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:44.462173       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-4>, retry it, err can not find pod <default/test-job3-default-nginx-4> in cache
E0724 01:27:44.562121       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
E0724 01:27:44.662120       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-0>, retry it, err can not find pod <default/test-job3-default-nginx-0> in cache
E0724 01:27:44.762155       1 job_controller_resync.go:62] Failed to sync pod <default/test-job3-default-nginx-5>, retry it, err can not find pod <default/test-job3-default-nginx-5> in cache
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/controllers kind/bug Categorizes issue or PR as related to a bug. priority/high
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix error resync logic hzxuzhonghu/volcano
3 participants
@hzxuzhonghu @k82cn @volcano-sh-bot