Lightning: Improve Fail-Fast Behavior for max-error.conflict Configuration

[dsdashun]

Enhancement

Currently, there is a configuration setting called max-error.conflict that restricts the maximum number of conflict errors for a Lightning task. However, currently this check is performed only when recording the conflict records into the meta table, after all the conflict data has already been constructed by the pebble DB and written into TiKV. If there are many conflicts, constructing all that conflict data can be very time-consuming. To address this, we propose that, during the construction of KV data, if the number of conflict records hits the max-error.conflict threshold, the entire import process should be halted and all the conflict records should be recorded into the meta table. This way, we can ensure that the max-error.conflict configuration can truly make the Lightning fail fast.

[dsdashun]

/component lightning

[okJiang]

also need to add

if em.remainingError.Conflict.Load() <= 0 {
	return nil
}

before

tidb/br/pkg/lightning/errormanager/errormanager.go

Line 250 in d9149b0

if em.remainingError.Conflict.Sub(int64(len(conflictInfos))) < 0 {

Otherwise, we may write duplicate data into downstream for every task when em.remainingError.Conflict.Load() <= 0. It is unexpected.