cdc panic "The CommitTs must be greater than the resolvedTs" after "cdc cli unsafe resolve-lock"

[fubinzh]

What did you do?

1. Met cdc checkpoint stuck issue, due to tikv lock issue
2. follow TiCDC take long time (may be a day) to recover from TiKV cluster failover #4516 (comment), run cdc cli unsafe resolve-lock to unlock

What did you expect to see?

cdc should not panic

What did you see instead?

cdc panic

[root@bogon bigCluster]# kubectl --kubeconfig kubeconfig.yml -n cdc-kafka-big-cluster-tps-1712340-1-428 logs -p tc-ticdc-1
[WARN] TiCDC server data-dir is not set. Please use `cdc server --data-dir` to start the cdc server if possible.
panic: The CommitTs must be greater than the resolvedTs

goroutine 164559 [running]:
[go.uber.org/zap/zapcore.CheckWriteAction.OnWrite](http://go.uber.org/zap/zapcore.CheckWriteAction.OnWrite)(0x4?, 0x4?, {0x0?, 0x0?, 0xc13bc95900?})
        [go.uber.org/zap@v1.24.0/zapcore/entry.go:198](http://go.uber.org/zap@v1.24.0/zapcore/entry.go:198) +0x65
[go.uber.org/zap/zapcore.(*CheckedEntry).Write](http://go.uber.org/zap/zapcore.(*CheckedEntry).Write)(0xc0cc5049c0, {0xc0e8dcb800, 0x4, 0x4})
        [go.uber.org/zap@v1.24.0/zapcore/entry.go:264](http://go.uber.org/zap@v1.24.0/zapcore/entry.go:264) +0x3ec
[go.uber.org/zap.(*Logger).Panic](http://go.uber.org/zap.(*Logger).Panic)(0xc0000e3800?, {0x4012d5b?, 0x0?}, {0xc0e8dcb800, 0x4, 0x4})
        [go.uber.org/zap@v1.24.0/logger.go:258](http://go.uber.org/zap@v1.24.0/logger.go:258) +0x59
[github.com/pingcap/log.Panic](http://github.com/pingcap/log.Panic)({0x4012d5b?, 0xc0537bff80?}, {0xc0e8dcb800?, 0x70?, 0xc0537c8000?})
        [github.com/pingcap/log@v1.1.1-0.20230317032135-a0d097d16e22/global.go:54](http://github.com/pingcap/log@v1.1.1-0.20230317032135-a0d097d16e22/global.go:54) +0x8b
[github.com/pingcap/tiflow/cdc/kv.(*regionWorker).handleEventEntry](http://github.com/pingcap/tiflow/cdc/kv.(*regionWorker).handleEventEntry)(0xc0094523f0, {0x4893ea8, 0xc013713f90}, 0xc0b144b900?, 0xc02385e000)
        [github.com/pingcap/tiflow/cdc/kv/region_worker.go:688](http://github.com/pingcap/tiflow/cdc/kv/region_worker.go:688) +0x10c9
[github.com/pingcap/tiflow/cdc/kv.(*regionWorker).processEvent](http://github.com/pingcap/tiflow/cdc/kv.(*regionWorker).processEvent)(0xc0094523f0, {0x4893ea8, 0xc013713f90}, 0xc04ce498f0)
        [github.com/pingcap/tiflow/cdc/kv/region_worker.go:366](http://github.com/pingcap/tiflow/cdc/kv/region_worker.go:366) +0xf6
[github.com/pingcap/tiflow/cdc/kv.(*regionWorker).eventHandler](http://github.com/pingcap/tiflow/cdc/kv.(*regionWorker).eventHandler)(0xc0094523f0, {0x4893ea8, 0xc013713f90})
        [github.com/pingcap/tiflow/cdc/kv/region_worker.go:512](http://github.com/pingcap/tiflow/cdc/kv/region_worker.go:512) +0x5d7
[github.com/pingcap/tiflow/cdc/kv.(*regionWorker).run.func4()](http://github.com/pingcap/tiflow/cdc/kv.(*regionWorker).run.func4())
        [github.com/pingcap/tiflow/cdc/kv/region_worker.go:603](http://github.com/pingcap/tiflow/cdc/kv/region_worker.go:603) +0x2e
[golang.org/x/sync/errgroup.(*Group).Go.func1()](http://golang.org/x/sync/errgroup.(*Group).Go.func1())
        [golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75](http://golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75) +0x64
created by [golang.org/x/sync/errgroup.(*Group).Go](http://golang.org/x/sync/errgroup.(*Group).Go)
        [golang.org/x/sync@v0.1.0/errgroup/errgroup.go:72](http://golang.org/x/sync@v0.1.0/errgroup/errgroup.go:72) +0xa5

Versions of the cluster

TiCDC version (execute cdc version):

[2023/05/26 10:16:39.798 +08:00] [INFO] [version.go:47] ["Welcome to Change Data Capture (CDC)"] [release-version=v7.1.0] [git-hash=f4b7ef5eef3833b26b81e9d57e76687d4ac93bc7] [git-branch=heads/refs/tags/v7.1.0] [utc-build-time="2023-05-25 06:40:04"] [go-version="go version go1.20.3 linux/amd64"] [failpoint-build=false]                    

[fubinzh]

/severity critical

[nongfushanquan]

/remove-label affects-6.1