restore: update and restore GCLifeTime once when parallel

[lance6716]

What problem does this PR solve?

DoChecksum doesn't prepare for parallel case

What is changed and how it works?

There may be multiple DoChecksum running, setting and resetting GC Life Time should not be done when there's unfinished DoChecksum tasks.

Lightning have restoreTables runs simultaneously, so we use this logic

func restoreTables() {
    // init_helper
    for some conurrency {
        go restoreTable()
    }
}

func restoreTable() {
    // call postProcess() -> ... -> DoChecksum()
}

func DoChecksum() {
    // using helper's pointers of lock, running jobs counter, original value
}

Using variable to count running checksum jobs. Before a remote chechsum call starting, lock, increase this counter, check if it just rised from zero then backup original value and call set GC Life Time logic, unlock.
After a remote chechsum finishing, lock, decrease this counter, check if it just drop to zero then call resetting GC Life Time logic, unlock.

The lock, counter, original value are pointed to same per location in one restoreTables.

Check List

Tests

• Unit test
• Integration test

Side effects

Related changes

• Need to cherry-pick to the release branch

[lance6716]

Seems that it's another solution of #218. Should I borrow the code and integration test and revert context-related code? @kennytm @amyangfei

[kennytm]

🤔 Oh I thought you've found another test case which isn't fixed by #218.

An advantage of the Context based solution is that it doesn't rely on global variables.

[lance6716]

🤔 Oh I thought you've found another test case which isn't fixed by #218.

An advantage of the Context based solution is that it doesn't rely on global variables.

I'll learn Context and fix that conflict tomorrow.

[lance6716]

@leoppro found a bug in my atomic integer code.

C1:->atomic check---------------------------------------->set GC->
C2:--------------->atomic check--->remote checksum call---------->

We think there should be a mutex, so I'll add three variable: a mutex for set GC time, a atomic integer as job counter(or a mutex for reset GC time), original GC time.

I still prefer using global variable, since lightning can only deal with one TiDB, and three variables are about "lock for TiDB setting, TiDB running jobs, TiDB setting" accrodingly, global variable won't pollute a lot.

Another way is using context like #218, assign three-variable-struct to context in restoreTables and get values in DoChecksum.

I'd like to learn your suggestions.

[kennytm]

@lance6716 use a struct, related variables should be placed close to each other.

[lance6716]

/run-all-tests

[lance6716]

PTAL @kennytm @amyangfei

[lance6716]

PTAL @leoppro

[lance6716]

/run-all-tests

[lance6716]

/run-all-tests

[kennytm]

Rest LGTM

[lance6716]

/run-all-tests

[lance6716]

/run-all-tests

[kennytm]

Rest LGTM

[lance6716]

/run-all-tests

@leoppro PTAL

[kennytm]

Ping @leoppro

[zier-one]

LGTM