kvclient does not make progress on resolved ts for long time.

[3AceShowHand]

KV Client would receive resolved ts event from TiKV CDC Component, which makes progress of the whole sync task. Some of the test scenario shows that the resolved ts may not be forwarded for a long time.

related issues

• TiCDC take long time (may be a day) to recover from TiKV cluster failover #4516 (comment)
• TiCDC does not resolve lock in time #2867

[Tammyxia]

[2022/03/15 01:08:10.052 +00:00] [WARN] [region_worker.go:354] ["region not receiving resolved event from tikv or resolved ts is not pushing for too long time, try to resolve lock"] [changefeed=tikv-scale-sync] [addr=upstream-tikv-1.upstream-tikv-peer.cdc-testbed-tps-660928-1-540.svc:20160] [regionID=637] [span="[6d44444c4a6f6241ff64ff644964784c69ff7374ff0000000000ff000000f700000000ff0000006c00000000fb, 6d44444c4a6f6241ff64ff644964784c69ff7374ff0000000000ff000000f700000000ff0000006d00000000fb)"] [duration=6h0m9.951s] [lastEvent=389.664591ms] [resolvedTs=431825847569874978]
[2022/03/15 01:08:10.053 +00:00] [INFO] [lock_resolver.go:133] ["resolve lock successfully"] [regionID=637] [lockCount=0] [maxVersion=431831509867429888] [changefeed=tikv-scale-sync] [role=processor]
[2022/03/15 01:08:10.664 +00:00] [WARN] [region_worker.go:751] ["The resolvedTs is fallen back in kvclient"] [changefeed=tikv-scale-sync] [EventType=RESOLVED] [resolvedTs=431825847569874978] [lastResolvedTs=431825847635410946] [regionID=272]

[overvenus]

May relate to tikv/tikv#12190

[3AceShowHand]

close by #4516, #2867

[jebter]

/severity major