Limits for add and recover balance data or zone balance jobs

[panda-sheep]

What type of PR is this?

• bug
• feature
• enhancement

What problem(s) does this PR solve?

Issue(s) number:

Description:

Only affects balance data or zone balance job

1) When adding a job
If a space has failed balance data or zone balance jobs
You cannot add new balance data or zone balance jobs to this space, 
you must first recover the failed balance data or zone balance jobs.
If the job keeps failing, you can stop the job first, and then add a new job of this type

2) When recovering the job
For  balance data or zone balance job.
If you recover a failed job, you can recover.
If you recover a stopped job, compare it by starttime to see if there is an updated failed job and finished job of this type after the stopped job.
If there is, it cannot be recovered, otherwise it can be recovered

For the same type of job, when recovering multiple stopped jobs at a time. 
Only recover the latest stopped job. 
 
The same type of STOPPED job exists in the following form:
job1(stopped) job2(failed)
recover job job1 failed
recover job job2 success

job1(stopped) job2(fininshed) job3(stopped)
recover job job1 failed
recover job job3 success
recover job job1, job3 Only job3 can recover

How do you solve it?

Special notes for your reviewer, ex. impact of this fix, design document, etc:

Checklist:

Tests:

• Unit test(positive and negative cases)
• Function test
• Performance test
• N/A

Affects:

• Documentation affected (Please add the label if documentation needs to be modified.)
• Incompatibility (If it breaks the compatibility, please describe it and add the label.）
• If it's needed to cherry-pick (If cherry-pick to some branches is required, please label the destination version(s).)
• Performance impacted: Consumes more CPU/Memory

Release notes:

Please confirm whether to be reflected in release notes and how to describe:

ex. Fixed the bug .....

[critical27]

Ready?

[critical27]

Generally LGTM, @liwenhui-soul please confirm if this PR satisfy.

[panda-sheep]

Ready?

yeah

[codecov-commenter]

Codecov Report

Merging #4104 (fe9d964) into master (5d96c60) will decrease coverage by 0.01%.
The diff coverage is 67.20%.

@@            Coverage Diff             @@
##           master    #4104      +/-   ##
==========================================
- Coverage   85.10%   85.08%   -0.02%     
==========================================
  Files        1339     1324      -15     
  Lines      132245   131714     -531     
==========================================
- Hits       112544   112066     -478     
+ Misses      19701    19648      -53     

Impacted Files	Coverage Δ
src/clients/meta/MetaClient.cpp	76.41% <0.00%> (-0.03%)	⬇️
src/clients/meta/MetaClient.h	92.30% <ø> (ø)
src/common/graph/Response.h	48.99% <ø> (ø)
src/common/thread/test/GenericThreadPoolTest.cpp	53.42% <0.00%> (-30.14%)	⬇️
src/common/thread/test/GenericWorkerTest.cpp	54.16% <0.00%> (-44.45%)	⬇️
src/graph/executor/ExecutionError.h	0.00% <ø> (ø)
src/graph/executor/Executor.cpp	81.32% <ø> (+0.31%)	⬆️
src/graph/executor/Executor.h	100.00% <ø> (ø)
src/graph/executor/StorageAccessExecutor.cpp	100.00% <ø> (ø)
src/graph/executor/StorageAccessExecutor.h	60.00% <ø> (ø)
... and 272 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update daaed0a...fe9d964. Read the comment docs.

[critical27]

So complicated ... I'm confusing ... Several question, majorly related to RECOVER:

1. If we could recover multiple job, one failed, one recovered, this is weird.
2. Do we really need to support recover STOPPED job? @liuyu85cn PTAL

[panda-sheep]

So complicated ... I'm confusing ... Several question, majorly related to RECOVER:

1. If we could recover multiple job, one failed, one recovered, this is weird.
2. Do we really need to support recover STOPPED job? @liuyu85cn PTAL

1. Multiple jobs can be recovered at one time. For multiple jobs of the same type, it is only necessary to recover the job with the latest starttime as much as possible.
2. The stop job can be recovered. Just need some conditions to limit whether it can be recovered.

[critical27]

So complicated ... I'm confusing ... Several question, majorly related to RECOVER:

1. If we could recover multiple job, one failed, one recovered, this is weird.
2. Do we really need to support recover STOPPED job? @liuyu85cn PTAL

1. Multiple jobs can be recovered at one time. For multiple jobs of the same type, it is only necessary to recover the job with the latest starttime as much as possible.
2. The stop job can be recovered. Just need some conditions to limit whether it can be recovered.

Yeah, some of the recovered jobs actually failed, this does not make sense. I mean we could restrict only recover one job is allowed later if necessary, we could simplify the code, and user won't be confused.

BTW, if we are recovering one job, and recover failed, what message do we get from console? "Recovered job failed"?

[panda-sheep]

So complicated ... I'm confusing ... Several question, majorly related to RECOVER:

1. If we could recover multiple job, one failed, one recovered, this is weird.
2. Do we really need to support recover STOPPED job? @liuyu85cn PTAL

1. Multiple jobs can be recovered at one time. For multiple jobs of the same type, it is only necessary to recover the job with the latest starttime as much as possible.
2. The stop job can be recovered. Just need some conditions to limit whether it can be recovered.

Yeah, some of the recovered jobs actually failed, this does not make sense. I mean we could restrict only recover one job is allowed later if necessary, we could simplify the code, and user won't be confused.

BTW, if we are recovering one job, and recover failed, what message do we get from console? "Recovered job failed"?

1. Yes, we can support it later. Only one jobId can be recovered at a time. This makes the error clearer.
2. At present, recover jobs only return how many jobs can be recovered.
   When recover multiple jobids at a time, a certain jobId cannot be executed and no error is reported.
   BTW, if we are recovering one job, and recover failed, what message do we get from console? "Recovered job failed"?
   So there may be no job to recover and no error to send.
   So here is the advantage of specifying only one jobid at a time when recovering