Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

etcd test failed by data race #10771

Closed
Tracked by #10742
flowbehappy opened this issue Mar 13, 2024 · 2 comments · Fixed by #10772
Closed
Tracked by #10742

etcd test failed by data race #10771

flowbehappy opened this issue Mar 13, 2024 · 2 comments · Fixed by #10772
Assignees
@hongyunyan
Labels
area/ticdc Issues or PRs related to TiCDC. severity/moderate type/bug The issue is confirmed as a bug.

Comments

@flowbehappy
Copy link
Collaborator

What did you do?

Run make cdc & make test

What did you expect to see?

Successfully finish.

What did you see instead?

)$ make cdc & make test
[1] 33885
CGO_ENABLED=1 GO111MODULE=on go build  -trimpath  -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v8.0.0-master" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-03-13 05:09:41" -X "github.com/pingcap/tiflow/pkg/version.GitHash=df8c65ea59612c639b8ccce1d576cd63b1b173b8" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=master" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 darwin/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v8.0.0-master"' -o bin/cdc ./cmd/cdc
cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/failpoint-ctl github.com/pingcap/failpoint/failpoint-ctl
scripts/generate-mock.sh
go generate
cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/protoc-gen-gogofaster github.com/gogo/protobuf/protoc-gen-gogofaster
cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/protoc-gen-go google.golang.org/protobuf/cmd/protoc-gen-go
cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/protoc-gen-go-grpc google.golang.org/grpc/cmd/protoc-gen-go-grpc
cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/protoc-gen-grpc-gateway github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway
cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/protoc-gen-grpc-gateway-v2 github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway
cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/protoc-gen-openapiv2 github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2
generate-protobuf
./scripts/generate-protobuf.sh
generate ./proto/EntryProtocol.proto...
generate ./proto/CanalProtocol.proto...
generate ./proto/CraftBenchmark.proto...
generate ./proto/CDCPeerToPeer.proto...
generate ./dm/proto/dmworker.proto...
generate ./dm/proto/dmmaster.proto...
generate cdc/processor/tablepb/table.proto...
generate cdc/scheduler/schedulepb/table_schedule.proto...
generate enginepb...
mkdir -p "/tmp/tidb_cdc_test"
$(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl enable >/dev/null)
ok  	github.com/pingcap/tiflow/cdc	5.135s	coverage: 95.5% of statements
ok  	github.com/pingcap/tiflow/cdc/api	4.471s	coverage: 8.8% of statements
?   	github.com/pingcap/tiflow/cdc/api/status	[no test files]
ok  	github.com/pingcap/tiflow/cdc/api/middleware	6.071s	coverage: 20.0% of statements
ok  	github.com/pingcap/tiflow/cdc/api/owner	4.400s	coverage: 22.8% of statements
ok  	github.com/pingcap/tiflow/cdc/api/v1	4.855s	coverage: 56.4% of statements
?   	github.com/pingcap/tiflow/cdc/capture/mock	[no test files]
?   	github.com/pingcap/tiflow/cdc/controller/mock	[no test files]
ok  	github.com/pingcap/tiflow/cdc/api/v2	24.277s	coverage: 70.2% of statements
ok  	github.com/pingcap/tiflow/cdc/capture	5.591s	coverage: 20.7% of statements
ok  	github.com/pingcap/tiflow/cdc/controller	4.605s	coverage: 43.7% of statements
?   	github.com/pingcap/tiflow/cdc/owner/mock	[no test files]
ok  	github.com/pingcap/tiflow/cdc/entry	100.544s	coverage: 61.2% of statements
ok  	github.com/pingcap/tiflow/cdc/entry/schema	4.402s	coverage: 58.0% of statements
ok  	github.com/pingcap/tiflow/cdc/kv	82.931s	coverage: 74.3% of statements
ok  	github.com/pingcap/tiflow/cdc/kv/regionlock	3.505s	coverage: 91.6% of statements
ok  	github.com/pingcap/tiflow/cdc/kv/sharedconn	5.674s	coverage: 84.9% of statements
ok  	github.com/pingcap/tiflow/cdc/metrics	1.432s	coverage: 100.0% of statements
ok  	github.com/pingcap/tiflow/cdc/model	3.001s	coverage: 55.4% of statements
ok  	github.com/pingcap/tiflow/cdc/model/codec	2.865s	coverage: 92.1% of statements
ok  	github.com/pingcap/tiflow/cdc/model/codec/v1	2.890s	coverage: 53.2% of statements
?   	github.com/pingcap/tiflow/cdc/processor/mock	[no test files]
?   	github.com/pingcap/tiflow/cdc/processor/sourcemanager	[no test files]
?   	github.com/pingcap/tiflow/cdc/processor/sourcemanager/puller	[no test files]
ok  	github.com/pingcap/tiflow/cdc/owner	48.509s	coverage: 73.6% of statements
ok  	github.com/pingcap/tiflow/cdc/processor	6.017s	coverage: 54.6% of statements
ok  	github.com/pingcap/tiflow/cdc/processor/memquota	2.991s	coverage: 87.0% of statements
ok  	github.com/pingcap/tiflow/cdc/processor/sinkmanager	5.891s	coverage: 73.2% of statements
?   	github.com/pingcap/tiflow/cdc/processor/sourcemanager/sorter/factory	[no test files]
ok  	github.com/pingcap/tiflow/cdc/processor/sourcemanager/sorter	4.234s	coverage: 50.0% of statements
?   	github.com/pingcap/tiflow/cdc/processor/sourcemanager/sorter/mock	[no test files]
ok  	github.com/pingcap/tiflow/cdc/processor/sourcemanager/sorter/memory	4.371s	coverage: 62.7% of statements
ok  	github.com/pingcap/tiflow/cdc/processor/sourcemanager/sorter/pebble	6.532s	coverage: 88.9% of statements
ok  	github.com/pingcap/tiflow/cdc/processor/sourcemanager/sorter/pebble/encoding	2.913s	coverage: 69.1% of statements
ok  	github.com/pingcap/tiflow/cdc/puller	18.244s	coverage: 64.1% of statements
ok  	github.com/pingcap/tiflow/cdc/puller/frontier	56.978s	coverage: 96.1% of statements
?   	github.com/pingcap/tiflow/cdc/redo/writer	[no test files]
?   	github.com/pingcap/tiflow/cdc/redo/writer/blackhole	[no test files]
?   	github.com/pingcap/tiflow/cdc/redo/writer/factory	[no test files]
ok  	github.com/pingcap/tiflow/cdc/puller/memorysorter	60.829s	coverage: 64.2% of statements
ok  	github.com/pingcap/tiflow/cdc/redo	10.743s	coverage: 77.3% of statements
ok  	github.com/pingcap/tiflow/cdc/redo/common	2.826s	coverage: 56.0% of statements
ok  	github.com/pingcap/tiflow/cdc/redo/reader	6.647s	coverage: 71.1% of statements
?   	github.com/pingcap/tiflow/cdc/scheduler	[no test files]
?   	github.com/pingcap/tiflow/cdc/scheduler/internal	[no test files]
ok  	github.com/pingcap/tiflow/cdc/redo/writer/file	3.136s	coverage: 72.9% of statements
ok  	github.com/pingcap/tiflow/cdc/redo/writer/memory	2.875s	coverage: 79.6% of statements
ok  	github.com/pingcap/tiflow/cdc/scheduler/internal/v3	6.420s	coverage: 67.8% of statements
ok  	github.com/pingcap/tiflow/cdc/scheduler/internal/v3/agent	5.094s	coverage: 86.3% of statements
ok  	github.com/pingcap/tiflow/cdc/scheduler/internal/v3/compat	2.973s	coverage: 92.4% of statements
ok  	github.com/pingcap/tiflow/cdc/scheduler/internal/v3/keyspan	8.573s	coverage: 89.3% of statements
ok  	github.com/pingcap/tiflow/cdc/scheduler/internal/v3/member	3.233s	coverage: 90.0% of statements
ok  	github.com/pingcap/tiflow/cdc/scheduler/internal/v3/replication	10.041s	coverage: 72.6% of statements
ok  	github.com/pingcap/tiflow/cdc/scheduler/internal/v3/scheduler	3.095s	coverage: 90.7% of statements
ok  	github.com/pingcap/tiflow/cdc/scheduler/internal/v3/transport	2.895s	coverage: 48.2% of statements
?   	github.com/pingcap/tiflow/cdc/sink/ddlsink	[no test files]
?   	github.com/pingcap/tiflow/cdc/sink/ddlsink/blackhole	[no test files]
?   	github.com/pingcap/tiflow/cdc/sink/ddlsink/factory	[no test files]
ok  	github.com/pingcap/tiflow/cdc/server	21.685s	coverage: 40.6% of statements
ok  	github.com/pingcap/tiflow/cdc/sink/ddlsink/cloudstorage	9.022s	coverage: 60.2% of statements
ok  	github.com/pingcap/tiflow/cdc/sink/ddlsink/mq	4.663s	coverage: 74.0% of statements
ok  	github.com/pingcap/tiflow/cdc/sink/ddlsink/mq/ddlproducer	2.877s	coverage: 21.1% of statements
ok  	github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql	7.955s	coverage: 70.0% of statements
ok  	github.com/pingcap/tiflow/cdc/sink/dmlsink	2.888s	coverage: 92.3% of statements
ok  	github.com/pingcap/tiflow/cdc/sink/dmlsink/blackhole	3.769s	coverage: 75.0% of statements
ok  	github.com/pingcap/tiflow/cdc/sink/dmlsink/cloudstorage	16.588s	coverage: 78.4% of statements
ok  	github.com/pingcap/tiflow/cdc/sink/dmlsink/factory	4.934s	coverage: 12.2% of statements
ok  	github.com/pingcap/tiflow/cdc/sink/dmlsink/mq	23.015s	coverage: 64.1% of statements
ok  	github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/dispatcher	2.925s	coverage: 78.9% of statements
ok  	github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/dispatcher/partition	3.320s	coverage: 89.5% of statements
ok  	github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/dispatcher/topic	1.498s	coverage: 76.5% of statements
?   	github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/transformer	[no test files]
ok  	github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/dmlproducer	3.064s	coverage: 23.4% of statements
ok  	github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/manager	3.591s	coverage: 61.4% of statements
?   	github.com/pingcap/tiflow/cdc/sink/metrics	[no test files]
?   	github.com/pingcap/tiflow/cdc/sink/metrics/cloudstorage	[no test files]
?   	github.com/pingcap/tiflow/cdc/sink/metrics/mq	[no test files]
?   	github.com/pingcap/tiflow/cdc/sink/metrics/tablesink	[no test files]
?   	github.com/pingcap/tiflow/cdc/sink/metrics/txn	[no test files]
ok  	github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/transformer/columnselector	10.980s	coverage: 84.6% of statements
ok  	github.com/pingcap/tiflow/cdc/sink/dmlsink/txn	8.007s	coverage: 75.4% of statements
ok  	github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql	11.377s	coverage: 84.7% of statements
ok  	github.com/pingcap/tiflow/cdc/sink/tablesink	3.608s	coverage: 91.6% of statements
ok  	github.com/pingcap/tiflow/cdc/sink/tablesink/state	1.468s	coverage: 16.7% of statements
ok  	github.com/pingcap/tiflow/cdc/sink/util	3.010s	coverage: 22.6% of statements
?   	github.com/pingcap/tiflow/cdc/syncpointstore	[no test files]
?   	github.com/pingcap/tiflow/cdcv2/capture	[no test files]
?   	github.com/pingcap/tiflow/cdcv2/controller	[no test files]
ok  	github.com/pingcap/tiflow/cdc/sink/validator	4.882s	coverage: 58.8% of statements
?   	github.com/pingcap/tiflow/cdcv2/owner	[no test files]
?   	github.com/pingcap/tiflow/docs/swagger	[no test files]
ok  	github.com/pingcap/tiflow/cdcv2/metadata	4.506s	coverage: 0.0% of statements [no tests to run]
ok  	github.com/pingcap/tiflow/cdcv2/metadata/sql	5.907s	coverage: 32.8% of statements
?   	github.com/pingcap/tiflow/pkg/api/v2	[no test files]
?   	github.com/pingcap/tiflow/pkg/api/v2/mock	[no test files]
ok  	github.com/pingcap/tiflow/pkg/api/internal/rest	6.719s	coverage: 80.5% of statements
?   	github.com/pingcap/tiflow/pkg/causality	[no test files]
ok  	github.com/pingcap/tiflow/pkg/applier	5.272s	coverage: 72.7% of statements
ok  	github.com/pingcap/tiflow/pkg/causality/internal	2.445s	coverage: 94.3% of statements
ok  	github.com/pingcap/tiflow/pkg/causality/tests	3.527s	coverage: 93.9% of statements
?   	github.com/pingcap/tiflow/pkg/cmd	[no test files]
ok  	github.com/pingcap/tiflow/pkg/chann	13.613s	coverage: 88.2% of statements
ok  	github.com/pingcap/tiflow/pkg/chdelay	3.489s	coverage: 97.9% of statements
?   	github.com/pingcap/tiflow/pkg/cmd/context	[no test files]
ok  	github.com/pingcap/tiflow/pkg/cmd/cli	4.531s	coverage: 68.8% of statements
?   	github.com/pingcap/tiflow/pkg/cmd/factory/mock	[no test files]
?   	github.com/pingcap/tiflow/pkg/cmd/version	[no test files]
?   	github.com/pingcap/tiflow/pkg/compression	[no test files]
ok  	github.com/pingcap/tiflow/pkg/cmd/factory	22.711s	coverage: 10.9% of statements
ok  	github.com/pingcap/tiflow/pkg/cmd/redo	4.523s	coverage: 14.8% of statements
ok  	github.com/pingcap/tiflow/pkg/cmd/server	5.603s	coverage: 58.3% of statements
ok  	github.com/pingcap/tiflow/pkg/cmd/util	3.013s	coverage: 73.3% of statements
?   	github.com/pingcap/tiflow/pkg/config/outdated	[no test files]
ok  	github.com/pingcap/tiflow/pkg/config	3.442s	coverage: 69.4% of statements
ok  	github.com/pingcap/tiflow/pkg/container/queue	3.760s	coverage: 98.1% of statements
ok  	github.com/pingcap/tiflow/pkg/container/sortmap	2.552s	coverage: 100.0% of statements
?   	github.com/pingcap/tiflow/pkg/election/mock	[no test files]
ok  	github.com/pingcap/tiflow/pkg/context	5.456s	coverage: 69.2% of statements
ok  	github.com/pingcap/tiflow/pkg/election	11.624s	coverage: 87.2% of statements
ok  	github.com/pingcap/tiflow/pkg/errors	2.446s	coverage: 95.1% of statements
?   	github.com/pingcap/tiflow/pkg/etcd/mock	[no test files]
ok  	github.com/pingcap/tiflow/pkg/errorutil	3.592s	coverage: 88.0% of statements
[2024/03/13 13:38:27.991 +08:00] [WARN] [client.go:286] ["etcd client watchCh blocking too long, reset the watchCh"] [duration=10s] [stack="github.com/pingcap/tiflow/pkg/etcd.(*Client).WatchWithChan\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:288\ngithub.com/pingcap/tiflow/pkg/etcd.TestRevisionNotFallBack.func2\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:302"] [role=]
[2024/03/13 13:38:28.002 +08:00] [WARN] [client.go:286] ["etcd client watchCh blocking too long, reset the watchCh"] [duration=10s] [stack="github.com/pingcap/tiflow/pkg/etcd.(*Client).WatchWithChan\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:288\ngithub.com/pingcap/tiflow/pkg/etcd.TestRevisionNotFallBack.func2\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:302"] [role=]
[2024/03/13 13:38:28.014 +08:00] [WARN] [client.go:286] ["etcd client watchCh blocking too long, reset the watchCh"] [duration=10s] [stack="github.com/pingcap/tiflow/pkg/etcd.(*Client).WatchWithChan\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:288\ngithub.com/pingcap/tiflow/pkg/etcd.TestRevisionNotFallBack.func2\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:302"] [role=]
[2024/03/13 13:38:28.017 +08:00] [INFO] [client.go:248] ["WatchWithChan exited"] [role=]
[2024/03/13 13:38:28.937 +08:00] [WARN] [client.go:110] ["etcd RPC failed"] [RPC=Revoke] [error="etcdserver: requested lease not found"]
[2024/03/13 13:38:29.050 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=10s] [role=]
[2024/03/13 13:38:29.051 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=11s] [role=]
[2024/03/13 13:38:29.052 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=12s] [role=]
[2024/03/13 13:38:29.054 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=13s] [role=]
[2024/03/13 13:38:29.055 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=14s] [role=]
[2024/03/13 13:38:29.056 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=15s] [role=]
[2024/03/13 13:38:29.057 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=16s] [role=]
[2024/03/13 13:38:29.059 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=17s] [role=]
[2024/03/13 13:38:29.061 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=18s] [role=]
[2024/03/13 13:38:29.062 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=19s] [role=]
[2024/03/13 13:38:29.063 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=20s] [role=]
[2024/03/13 13:38:29.064 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=21s] [role=]
[2024/03/13 13:38:29.065 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=22s] [role=]
[2024/03/13 13:38:29.066 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=23s] [role=]
[2024/03/13 13:38:29.068 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=24s] [role=]
[2024/03/13 13:38:29.069 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=25s] [role=]
[2024/03/13 13:38:29.070 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=26s] [role=]
[2024/03/13 13:38:29.071 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=27s] [role=]
[2024/03/13 13:38:29.072 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=28s] [role=]
[2024/03/13 13:38:29.075 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=29s] [role=]
[2024/03/13 13:38:29.077 +08:00] [INFO] [client.go:248] ["WatchWithChan exited"] [role=]
[2024/03/13 13:38:29.078 +08:00] [PANIC] [etcdkey.go:219] [unreachable] [stack="github.com/pingcap/tiflow/pkg/etcd.(*CDCKey).String\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/etcdkey.go:219\ngithub.com/pingcap/tiflow/pkg/etcd.TestEtcdKeyParseError.func1\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/etcdkey_test.go:169\ngithub.com/stretchr/testify/assert.didPanic\n\t/Users/flow/workspace/gopath/pkg/mod/github.com/stretchr/testify@v1.8.4/assert/assertions.go:1158\ngithub.com/stretchr/testify/assert.Panics\n\t/Users/flow/workspace/gopath/pkg/mod/github.com/stretchr/testify@v1.8.4/assert/assertions.go:1172\ngithub.com/stretchr/testify/require.Panics\n\t/Users/flow/workspace/gopath/pkg/mod/github.com/stretchr/testify@v1.8.4/require/require.go:1718\ngithub.com/pingcap/tiflow/pkg/etcd.TestEtcdKeyParseError\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/etcdkey_test.go:168\ntesting.tRunner\n\t/usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595"]
==================
WARNING: DATA RACE
Write at 0x000108c45ca0 by goroutine 152:
  github.com/pingcap/tiflow/pkg/etcd.TestRetry()
      /Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:78 +0x64
  testing.tRunner()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x261
  testing.(*T).Run.func1()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x44

Previous read at 0x000108c45ca0 by goroutine 168:
  github.com/pingcap/tiflow/pkg/etcd.retryRPC()
      /Users/flow/workspace/github/tiflow/pkg/etcd/client.go:118 +0x109
  github.com/pingcap/tiflow/pkg/etcd.(*Client).Put()
      /Users/flow/workspace/github/tiflow/pkg/etcd/client.go:128 +0x2da
  github.com/pingcap/tiflow/pkg/etcd.TestDeleteCaptureInfo()
      /Users/flow/workspace/github/tiflow/pkg/etcd/etcd_test.go:462 +0x85b
  testing.tRunner()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x261
  testing.(*T).Run.func1()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x44

Goroutine 152 (running) created at:
  testing.(*T).Run()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x845
  testing.runTests.func1()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:2054 +0x84
  testing.tRunner()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x261
  testing.runTests()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:2052 +0x8ad
  testing.(*M).Run()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1925 +0xcd7
  go.uber.org/goleak.VerifyTestMain()
      /Users/flow/workspace/gopath/pkg/mod/go.uber.org/goleak@v1.3.0/testmain.go:53 +0x64
  github.com/pingcap/tiflow/pkg/leakutil.SetUpLeakTest()
      /Users/flow/workspace/github/tiflow/pkg/leakutil/leak_helper.go:48 +0x48a
  github.com/pingcap/tiflow/pkg/etcd.TestMain()
      /Users/flow/workspace/github/tiflow/pkg/etcd/main_test.go:23 +0x329
  main.main()
      _testmain.go:121 +0x328

Goroutine 168 (running) created at:
  testing.(*T).Run()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x845
  testing.runTests.func1()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:2054 +0x84
  testing.tRunner()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x261
  testing.runTests()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:2052 +0x8ad
  testing.(*M).Run()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1925 +0xcd7
  go.uber.org/goleak.VerifyTestMain()
      /Users/flow/workspace/gopath/pkg/mod/go.uber.org/goleak@v1.3.0/testmain.go:53 +0x64
  github.com/pingcap/tiflow/pkg/leakutil.SetUpLeakTest()
      /Users/flow/workspace/github/tiflow/pkg/leakutil/leak_helper.go:48 +0x48a
  github.com/pingcap/tiflow/pkg/etcd.TestMain()
      /Users/flow/workspace/github/tiflow/pkg/etcd/main_test.go:23 +0x329
  main.main()
      _testmain.go:121 +0x328
==================
[2024/03/13 13:38:30.001 +08:00] [WARN] [client.go:110] ["etcd RPC failed"] [RPC=Put] [error="mock error"] [errorVerbose="mock error\ngithub.com/pingcap/tiflow/pkg/etcd.(*mockClient).Put\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:43\ngithub.com/pingcap/tiflow/pkg/etcd.(*Client).Put.func1\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:130\ngithub.com/pingcap/tiflow/pkg/etcd.retryRPC.func1\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:108\ngithub.com/pingcap/tiflow/pkg/retry.run\n\t/Users/flow/workspace/github/tiflow/pkg/retry/retry_with_opt.go:57\ngithub.com/pingcap/tiflow/pkg/retry.Do\n\t/Users/flow/workspace/github/tiflow/pkg/retry/retry_with_opt.go:34\ngithub.com/pingcap/tiflow/pkg/etcd.retryRPC\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:107\ngithub.com/pingcap/tiflow/pkg/etcd.(*Client).Put\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:128\ngithub.com/pingcap/tiflow/pkg/etcd.TestRetry\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:88\ntesting.tRunner\n\t/usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595\nruntime.goexit\n\t/usr/local/Cellar/go/1.21.6/libexec/src/runtime/asm_amd64.s:1650"]
[2024/03/13 13:38:30.007 +08:00] [WARN] [client.go:286] ["etcd client watchCh blocking too long, reset the watchCh"] [duration=10s] [stack="github.com/pingcap/tiflow/pkg/etcd.(*Client).WatchWithChan\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:288\ngithub.com/pingcap/tiflow/pkg/etcd.TestWatchChBlocked.func2\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:191"] [role=]
[2024/03/13 13:38:30.018 +08:00] [WARN] [client.go:286] ["etcd client watchCh blocking too long, reset the watchCh"] [duration=10s] [stack="github.com/pingcap/tiflow/pkg/etcd.(*Client).WatchWithChan\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:288\ngithub.com/pingcap/tiflow/pkg/etcd.TestWatchChBlocked.func2\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:191"] [role=]
[2024/03/13 13:38:30.030 +08:00] [INFO] [client.go:248] ["WatchWithChan exited"] [role=]
--- FAIL: TestWatchChBlocked (0.04s)
    testing.go:1465: race detected during execution of test
--- FAIL: TestCheckMultipleCDCClusterExist (2.61s)
    testing.go:61: <nil>
    testing.go:1465: race detected during execution of test
--- FAIL: TestDeleteCaptureInfo (1.61s)
    testing.go:61: <nil>
    testing.go:1465: race detected during execution of test
--- FAIL: TestGetAllChangeFeedInfo (1.28s)
    testing.go:61: <nil>
    testing.go:1465: race detected during execution of test
[2024/03/13 13:38:31.591 +08:00] [WARN] [etcd.go:522] ["unexpected etcd transaction failure, operation: Create"] [namespace=test] [changefeed=create-changefeed]
[2024/03/13 13:38:32.299 +08:00] [WARN] [client.go:110] ["etcd RPC failed"] [RPC=Put] [error="mock error"] [errorVerbose="mock error\ngithub.com/pingcap/tiflow/pkg/etcd.(*mockClient).Put\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:43\ngithub.com/pingcap/tiflow/pkg/etcd.(*Client).Put.func1\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:130\ngithub.com/pingcap/tiflow/pkg/etcd.retryRPC.func1\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:108\ngithub.com/pingcap/tiflow/pkg/retry.run\n\t/Users/flow/workspace/github/tiflow/pkg/retry/retry_with_opt.go:57\ngithub.com/pingcap/tiflow/pkg/retry.Do\n\t/Users/flow/workspace/github/tiflow/pkg/retry/retry_with_opt.go:34\ngithub.com/pingcap/tiflow/pkg/etcd.retryRPC\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:107\ngithub.com/pingcap/tiflow/pkg/etcd.(*Client).Put\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:128\ngithub.com/pingcap/tiflow/pkg/etcd.TestRetry\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:88\ntesting.tRunner\n\t/usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595\nruntime.goexit\n\t/usr/local/Cellar/go/1.21.6/libexec/src/runtime/asm_amd64.s:1650"]
[2024/03/13 13:38:32.300 +08:00] [WARN] [client.go:110] ["etcd RPC failed"] [RPC=Txn] [error="etcdserver: mvcc: database space exceeded"]
--- FAIL: TestGetOwnerRevision (2.35s)
    testing.go:61: <nil>
    testing.go:1465: race detected during execution of test
[2024/03/13 13:38:34.324 +08:00] [WARN] [client.go:110] ["etcd RPC failed"] [RPC=Txn] [error="etcdserver: mvcc: database space exceeded"]
[2024/03/13 13:38:34.324 +08:00] [WARN] [client.go:110] ["etcd RPC failed"] [RPC=Txn] [error="etcdserver: request timed out, possibly due to previous leader failure"]
[2024/03/13 13:38:36.527 +08:00] [WARN] [client.go:110] ["etcd RPC failed"] [RPC=Txn] [error="etcdserver: request timed out, possibly due to previous leader failure"]
[2024/03/13 13:38:36.528 +08:00] [WARN] [client.go:110] ["etcd RPC failed"] [RPC=Txn] [error="context deadline exceeded"]
[2024/03/13 13:38:36.528 +08:00] [WARN] [client.go:110] ["etcd RPC failed"] [RPC=Txn] [error="mock error"] [errorVerbose="mock error\ngithub.com/pingcap/tiflow/pkg/etcd.(*mockTxn).Commit\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:353\ngithub.com/pingcap/tiflow/pkg/etcd.(*Client).Txn.func1\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:172\ngithub.com/pingcap/tiflow/pkg/etcd.retryRPC.func1\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:108\ngithub.com/pingcap/tiflow/pkg/retry.run\n\t/Users/flow/workspace/github/tiflow/pkg/retry/retry_with_opt.go:57\ngithub.com/pingcap/tiflow/pkg/retry.Do\n\t/Users/flow/workspace/github/tiflow/pkg/retry/retry_with_opt.go:34\ngithub.com/pingcap/tiflow/pkg/etcd.retryRPC\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:107\ngithub.com/pingcap/tiflow/pkg/etcd.(*Client).Txn\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:170\ngithub.com/pingcap/tiflow/pkg/etcd.TestRetry\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:114\ntesting.tRunner\n\t/usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595\nruntime.goexit\n\t/usr/local/Cellar/go/1.21.6/libexec/src/runtime/asm_amd64.s:1650"]
--- FAIL: TestRetry (6.53s)
    testing.go:1465: race detected during execution of test
FAIL
coverage: 62.9% of statements
FAIL	github.com/pingcap/tiflow/pkg/etcd	11.173s
?   	github.com/pingcap/tiflow/pkg/integrity	[no test files]
ok  	github.com/pingcap/tiflow/pkg/filter	18.301s	coverage: 89.0% of statements
ok  	github.com/pingcap/tiflow/pkg/flags	1.460s	coverage: 100.0% of statements
ok  	github.com/pingcap/tiflow/pkg/fsutil	2.865s	coverage: 89.0% of statements
ok  	github.com/pingcap/tiflow/pkg/hash	1.403s	coverage: 90.0% of statements
ok  	github.com/pingcap/tiflow/pkg/httputil	2.491s	coverage: 60.0% of statements
ok  	github.com/pingcap/tiflow/pkg/label	1.431s	coverage: 88.2% of statements
ok  	github.com/pingcap/tiflow/pkg/leakutil	1.414s	coverage: 50.0% of statements
ok  	github.com/pingcap/tiflow/pkg/logutil	4.628s	coverage: 56.2% of statements
ok  	github.com/pingcap/tiflow/pkg/migrate	12.099s	coverage: 77.2% of statements
ok  	github.com/pingcap/tiflow/pkg/notify	6.846s	coverage: 100.0% of statements
?   	github.com/pingcap/tiflow/pkg/sink	[no test files]
ok  	github.com/pingcap/tiflow/pkg/orchestrator	186.972s	coverage: 69.0% of statements
ok  	github.com/pingcap/tiflow/pkg/orchestrator/util	1.403s	coverage: 100.0% of statements
ok  	github.com/pingcap/tiflow/pkg/p2p	14.233s	coverage: 82.8% of statements
ok  	github.com/pingcap/tiflow/pkg/p2p/internal	3.055s	coverage: 85.0% of statements
ok  	github.com/pingcap/tiflow/pkg/pdutil	37.864s	coverage: 69.4% of statements
ok  	github.com/pingcap/tiflow/pkg/quotes	1.467s	coverage: 100.0% of statements
ok  	github.com/pingcap/tiflow/pkg/redo	2.644s	coverage: 37.9% of statements
ok  	github.com/pingcap/tiflow/pkg/retry	4.666s	coverage: 94.4% of statements
ok  	github.com/pingcap/tiflow/pkg/security	1.692s	coverage: 12.1% of statements
ok  	github.com/pingcap/tiflow/pkg/sink/cloudstorage	3.132s	coverage: 79.4% of statements
ok  	github.com/pingcap/tiflow/pkg/sink/codec	7.267s	coverage: 40.5% of statements
ok  	github.com/pingcap/tiflow/pkg/sink/codec/avro	6.855s	coverage: 71.5% of statements
ok  	github.com/pingcap/tiflow/pkg/sink/codec/builder	4.208s	coverage: 0.0% of statements
?   	github.com/pingcap/tiflow/pkg/sink/codec/internal	[no test files]
?   	github.com/pingcap/tiflow/pkg/sink/codec/utils	[no test files]
?   	github.com/pingcap/tiflow/pkg/sink/kafka/v2/mock	[no test files]
ok  	github.com/pingcap/tiflow/pkg/sink/codec/canal	112.177s	coverage: 75.0% of statements
ok  	github.com/pingcap/tiflow/pkg/sink/codec/common	2.928s	coverage: 38.4% of statements
ok  	github.com/pingcap/tiflow/pkg/sink/codec/craft	4.629s	coverage: 84.5% of statements
ok  	github.com/pingcap/tiflow/pkg/sink/codec/csv	3.188s	coverage: 80.7% of statements
ok  	github.com/pingcap/tiflow/pkg/sink/codec/debezium	11.911s	coverage: 92.7% of statements
ok  	github.com/pingcap/tiflow/pkg/sink/codec/maxwell	3.315s	coverage: 50.7% of statements
ok  	github.com/pingcap/tiflow/pkg/sink/codec/open	4.385s	coverage: 71.0% of statements
ok  	github.com/pingcap/tiflow/pkg/sink/codec/simple	66.909s	coverage: 79.9% of statements
ok  	github.com/pingcap/tiflow/pkg/sink/kafka	3.717s	coverage: 58.4% of statements
ok  	github.com/pingcap/tiflow/pkg/sink/kafka/claimcheck	3.482s	coverage: 28.0% of statements
ok  	github.com/pingcap/tiflow/pkg/sink/kafka/v2	3.946s	coverage: 76.2% of statements
ok  	github.com/pingcap/tiflow/pkg/sink/mysql	5.547s	coverage: 60.5% of statements
ok  	github.com/pingcap/tiflow/pkg/sink/observer	5.110s	coverage: 77.0% of statements
ok  	github.com/pingcap/tiflow/pkg/sink/pulsar	2.935s	coverage: 39.0% of statements
ok  	github.com/pingcap/tiflow/pkg/spanz	6.366s	coverage: 61.1% of statements
?   	github.com/pingcap/tiflow/pkg/txnutil	[no test files]
ok  	github.com/pingcap/tiflow/pkg/sqlmodel	3.786s	coverage: 78.7% of statements
ok  	github.com/pingcap/tiflow/pkg/tcpserver	13.071s	coverage: 87.0% of statements
ok  	github.com/pingcap/tiflow/pkg/txnutil/gc	5.838s	coverage: 70.8% of statements
ok  	github.com/pingcap/tiflow/pkg/types	1.562s	coverage: 91.7% of statements
ok  	github.com/pingcap/tiflow/pkg/upstream	6.757s	coverage: 60.7% of statements
ok  	github.com/pingcap/tiflow/pkg/util	3.100s	coverage: 48.9% of statements
ok  	github.com/pingcap/tiflow/pkg/uuid	1.404s	coverage: 100.0% of statements
ok  	github.com/pingcap/tiflow/pkg/version	23.248s	coverage: 85.1% of statements
ok  	github.com/pingcap/tiflow/pkg/workerpool	26.955s	coverage: 86.8% of statements
FAIL
make: *** [unit_test] Error 1
[1]+  Done                    make cdc

Versions of the cluster

Master
@flowbehappy flowbehappy added type/bug The issue is confirmed as a bug. area/ticdc Issues or PRs related to TiCDC. labels Mar 13, 2024
@flowbehappy flowbehappy mentioned this issue Mar 13, 2024
Closed
12 tasks
@hongyunyan hongyunyan mentioned this issue Mar 13, 2024
Merged
@fubinzh
Copy link

fubinzh commented Mar 13, 2024

/severity moderate

@fubinzh
Copy link

fubinzh commented Mar 13, 2024

/severity moderate

ti-chi-bot bot pushed a commit that referenced this issue Mar 25, 2024
@hongyunyan
ticdc: Fix unstable test TestRetry (#10772)
3a7286e 
close #10771
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hongyunyan hongyunyan

Labels
area/ticdc Issues or PRs related to TiCDC. severity/moderate type/bug The issue is confirmed as a bug.
Projects
Question and Bug Reports
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ticdc: Fix unstable test TestRetry hongyunyan/tiflow
3 participants
@flowbehappy @fubinzh @hongyunyan