[dr-autosync]create or list changefeed failed after force pd as one new cluster

[mayjiang0203]

What did you do?

Down primary dc and for the only pd instance in backup dc as a new cluster, then restart the pd.
Then list or creating one changefeed will be failed.

clinic cluster id: 7235999894101273250

What did you expect to see?

No response

What did you see instead?

[2023/05/22 14:41:49.461 +08:00] [WARN] [client.go:97] ["etcd RPC failed"] [RPC=Get] [error="context deadline exceeded"]
[2023/05/22 14:41:49.461 +08:00] [INFO] [util.go:174] ["get owner failed"] [error="[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded"] [errorVerbose="[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded\[ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:34\ngithub.com/pingcap/tiflow/pkg/etcd.(*CDCEtcdClientImpl).GetOwnerID\n\tgithub.com/pingcap/tiflow/pkg/etcd/etcd.go:579\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).GetOwnerCaptureInfo\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:726\ngithub.com/pingcap/tiflow/cdc/api.ForwardToOwner\n\tgithub.com/pingcap/tiflow/cdc/api/util.go:172\ngithub.com/pingcap/tiflow/cdc/api/middleware.ForwardToOwnerMiddleware.func1\n\tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:87\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/pingcap/tiflow/cdc/api/middleware.ErrorHandleMiddleware.func1\n\tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:64\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/pingcap/tiflow/cdc/api/middleware.LogMiddleware.func1\n\tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:38\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/pingcap/tiflow/cdc/api/middleware.CheckServerReadyMiddleware.func1\n\tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:103\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/gin-gonic/gin.CustomRecoveryWithWriter.func1\n\tgithub.com/gin-gonic/gin@v1.8.1/recovery.go:101\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/gin-gonic/gin.(*Engine).handleHTTPRequest\n\tgithub.com/gin-gonic/gin@v1.8.1/gin.go:616\ngithub.com/gin-gonic/gin.(*Engine).ServeHTTP\n\tgithub.com/gin-gonic/gin@v1.8.1/gin.go:572\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2947\nnet/http.(*conn).serve\n\tnet/http/server.go:1991\nruntime.goexit\n\truntime/asm_amd64.s:1594](http://ngithub.com/pingcap/errors.AddStack/n/tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174/ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs/n/tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/normalize.go:164/ngithub.com/pingcap/tiflow/pkg/errors.WrapError/n/tgithub.com/pingcap/tiflow/pkg/errors/helper.go:34/ngithub.com/pingcap/tiflow/pkg/etcd.(*CDCEtcdClientImpl).GetOwnerID/n/tgithub.com/pingcap/tiflow/pkg/etcd/etcd.go:579/ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).GetOwnerCaptureInfo/n/tgithub.com/pingcap/tiflow/cdc/capture/capture.go:726/ngithub.com/pingcap/tiflow/cdc/api.ForwardToOwner/n/tgithub.com/pingcap/tiflow/cdc/api/util.go:172/ngithub.com/pingcap/tiflow/cdc/api/middleware.ForwardToOwnerMiddleware.func1/n/tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:87/ngithub.com/gin-gonic/gin.(*Context).Next/n/tgithub.com/gin-gonic/gin@v1.8.1/context.go:173/ngithub.com/pingcap/tiflow/cdc/api/middleware.ErrorHandleMiddleware.func1/n/tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:64/ngithub.com/gin-gonic/gin.(*Context).Next/n/tgithub.com/gin-gonic/gin@v1.8.1/context.go:173/ngithub.com/pingcap/tiflow/cdc/api/middleware.LogMiddleware.func1/n/tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:38/ngithub.com/gin-gonic/gin.(*Context).Next/n/tgithub.com/gin-gonic/gin@v1.8.1/context.go:173/ngithub.com/pingcap/tiflow/cdc/api/middleware.CheckServerReadyMiddleware.func1/n/tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:103/ngithub.com/gin-gonic/gin.(*Context).Next/n/tgithub.com/gin-gonic/gin@v1.8.1/context.go:173/ngithub.com/gin-gonic/gin.CustomRecoveryWithWriter.func1/n/tgithub.com/gin-gonic/gin@v1.8.1/recovery.go:101/ngithub.com/gin-gonic/gin.(*Context).Next/n/tgithub.com/gin-gonic/gin@v1.8.1/context.go:173/ngithub.com/gin-gonic/gin.(*Engine).handleHTTPRequest/n/tgithub.com/gin-gonic/gin@v1.8.1/gin.go:616/ngithub.com/gin-gonic/gin.(*Engine).ServeHTTP/n/tgithub.com/gin-gonic/gin@v1.8.1/gin.go:572/nnet/http.serverHandler.ServeHTTP/n/tnet/http/server.go:2947/nnet/http.(*conn).serve/n/tnet/http/server.go:1991/nruntime.goexit/n/truntime/asm_amd64.s:1594)"]
[2023/05/22 14:41:49.461 +08:00] [INFO] [middleware.go:48] [/api/v1/changefeeds] [status=500] [method=GET] [path=/api/v1/changefeeds] [query="state=all"] [ip=[10.233.126.79](http://10.233.126.79/)] [user-agent=Go-http-client/1.1] [client-version=v6.5.1] [error="[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded"] [errorVerbose="[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded\[ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:34\ngithub.com/pingcap/tiflow/pkg/etcd.(*CDCEtcdClientImpl).GetOwnerID\n\tgithub.com/pingcap/tiflow/pkg/etcd/etcd.go:579\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).GetOwnerCaptureInfo\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:726\ngithub.com/pingcap/tiflow/cdc/api.ForwardToOwner\n\tgithub.com/pingcap/tiflow/cdc/api/util.go:172\ngithub.com/pingcap/tiflow/cdc/api/middleware.ForwardToOwnerMiddleware.func1\n\tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:87\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/pingcap/tiflow/cdc/api/middleware.ErrorHandleMiddleware.func1\n\tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:64\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/pingcap/tiflow/cdc/api/middleware.LogMiddleware.func1\n\tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:38\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/pingcap/tiflow/cdc/api/middleware.CheckServerReadyMiddleware.func1\n\tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:103\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/gin-gonic/gin.CustomRecoveryWithWriter.func1\n\tgithub.com/gin-gonic/gin@v1.8.1/recovery.go:101\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/gin-gonic/gin.(*Engine).handleHTTPRequest\n\tgithub.com/gin-gonic/gin@v1.8.1/gin.go:616\ngithub.com/gin-gonic/gin.(*Engine).ServeHTTP\n\tgithub.com/gin-gonic/gin@v1.8.1/gin.go:572\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2947\nnet/http.(*conn).serve\n\tnet/http/server.go:1991\nruntime.goexit\n\truntime/asm_amd64.s:1594](http://ngithub.com/pingcap/errors.AddStack/n/tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174/ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs/n/tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/normalize.go:164/ngithub.com/pingcap/tiflow/pkg/errors.WrapError/n/tgithub.com/pingcap/tiflow/pkg/errors/helper.go:34/ngithub.com/pingcap/tiflow/pkg/etcd.(*CDCEtcdClientImpl).GetOwnerID/n/tgithub.com/pingcap/tiflow/pkg/etcd/etcd.go:579/ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).GetOwnerCaptureInfo/n/tgithub.com/pingcap/tiflow/cdc/capture/capture.go:726/ngithub.com/pingcap/tiflow/cdc/api.ForwardToOwner/n/tgithub.com/pingcap/tiflow/cdc/api/util.go:172/ngithub.com/pingcap/tiflow/cdc/api/middleware.ForwardToOwnerMiddleware.func1/n/tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:87/ngithub.com/gin-gonic/gin.(*Context).Next/n/tgithub.com/gin-gonic/gin@v1.8.1/context.go:173/ngithub.com/pingcap/tiflow/cdc/api/middleware.ErrorHandleMiddleware.func1/n/tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:64/ngithub.com/gin-gonic/gin.(*Context).Next/n/tgithub.com/gin-gonic/gin@v1.8.1/context.go:173/ngithub.com/pingcap/tiflow/cdc/api/middleware.LogMiddleware.func1/n/tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:38/ngithub.com/gin-gonic/gin.(*Context).Next/n/tgithub.com/gin-gonic/gin@v1.8.1/context.go:173/ngithub.com/pingcap/tiflow/cdc/api/middleware.CheckServerReadyMiddleware.func1/n/tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:103/ngithub.com/gin-gonic/gin.(*Context).Next/n/tgithub.com/gin-gonic/gin@v1.8.1/context.go:173/ngithub.com/gin-gonic/gin.CustomRecoveryWithWriter.func1/n/tgithub.com/gin-gonic/gin@v1.8.1/recovery.go:101/ngithub.com/gin-gonic/gin.(*Context).Next/n/tgithub.com/gin-gonic/gin@v1.8.1/context.go:173/ngithub.com/gin-gonic/gin.(*Engine).handleHTTPRequest/n/tgithub.com/gin-gonic/gin@v1.8.1/gin.go:616/ngithub.com/gin-gonic/gin.(*Engine).ServeHTTP/n/tgithub.com/gin-gonic/gin@v1.8.1/gin.go:572/nnet/http.serverHandler.ServeHTTP/n/tnet/http/server.go:2947/nnet/http.(*conn).serve/n/tnet/http/server.go:1991/nruntime.goexit/n/truntime/asm_amd64.s:1594)"] [duration=30.000973357s]

the w/a is change start scripts run_cdc.sh

sh-4.2# cat run_cdc.sh
#!/bin/bash
set -e

# WARNING: This file was auto-generated. Do not edit!
#          All your edit might be overwritten!
DEPLOY_DIR=/tiup/deploy/cdc-8300
cd "${DEPLOY_DIR}" || exit 1
exec bin/cdc server \
    --addr "0.0.0.0:8300" \
    --advertise-addr "ticdc2-peer:8300" \
    --pd "http://pd1-peer:2379,http://pd2-peer:2379,http://pd3-peer:2379" \
    --data-dir="/tiup/data/cdc-8300" \
    --gc-ttl 86400 \
    --config conf/cdc.toml \
    --log-file "/tiup/deploy/cdc-8300/log/cdc.log" 2>> "/tiup/deploy/cdc-8300/log/cdc_stderr.log"

To

sh-4.2# cat run_cdc.sh
#!/bin/bash
set -e

# WARNING: This file was auto-generated. Do not edit!
#          All your edit might be overwritten!
DEPLOY_DIR=/tiup/deploy/cdc-8300
cd "${DEPLOY_DIR}" || exit 1
exec bin/cdc server \
    --addr "0.0.0.0:8300" \
    --advertise-addr "ticdc2-peer:8300" \
    --pd "http://pd3-peer:2379" \
    --data-dir="/tiup/data/cdc-8300" \
    --gc-ttl 86400 \
    --config conf/cdc.toml \
    --log-file "/tiup/deploy/cdc-8300/log/cdc.log" 2>> "/tiup/deploy/cdc-8300/log/cdc_stderr.log"

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):
master

(paste TiCDC version here)

[mayjiang0203]

/severity major
/assign @asddongmen

[asddongmen]

This issue may have been fixed by #8884 #8813 #9106

[mayjiang0203]

Verify it has been fixed.