Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some unit tests failed after adding three last commits #17232

Closed
4 tasks
KosovGrigorii opened this issue Jan 12, 2024 · 21 comments · Fixed by #17313
Closed
4 tasks

Some unit tests failed after adding three last commits #17232

KosovGrigorii opened this issue Jan 12, 2024 · 21 comments · Fixed by #17313

Comments

@KosovGrigorii
Copy link

KosovGrigorii commented Jan 12, 2024

Bug report criteria

What happened?

I synced my forked repo with etcd main which added commits b3a5307 db61c96 b3bf59a. After doing this and running unit tests (make test-unit) it fails on go.etcd.io/etcd/server/v3/storage/mvcc (test is called TestHashKVWhenCompacting). However when I run only this specific test everything goes well.

What did you expect to happen?

All unit tests to pass

How can we reproduce it (as minimally and precisely as possible)?

keep repository up to date and run make test-unit

Anything else we need to know?

before I added these last commits all unit tests passed, I run them in main branch

Etcd version (please run commands below)

$ etcd --version
# paste output here

$ etcdctl version
# paste output here

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x166fbc7]

goroutine 734 [running]:
go.etcd.io/bbolt.(*Tx).Bucket(...)
        /Users/kosovgrigorii/go/pkg/mod/go.etcd.io/bbolt@v1.3.8/tx.go:102
go.etcd.io/etcd/server/v3/storage/backend.unsafeForEach(0x0, {0x1df8b58, 0x236e600}, 0xc0002cfae0?)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/storage/backend/batch_tx.go:234 +0x47
go.etcd.io/etcd/server/v3/storage/backend.(*baseReadTx).UnsafeForEach(0xc00042a050, {0x1df8b58, 0x236e600}, 0xc00058c040)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/storage/backend/read_tx.go:70 +0x199
go.etcd.io/etcd/server/v3/storage/mvcc.unsafeHashByRev({0x4bca4288, 0xc00042a050}, 0xc0002cff00?, 0xc0002cff00?, 0xc00001f1e0?)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/storage/mvcc/hash.go:35 +0x1b5
go.etcd.io/etcd/server/v3/storage/mvcc.(*store).hashByRev(0xc0000b2750, 0x2710)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/storage/mvcc/kvstore.go:187 +0x38c
go.etcd.io/etcd/server/v3/storage/mvcc.(*hashStorage).HashByRev(0xc0000ba680, 0x2710)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/storage/mvcc/hash.go:139 +0x1d2
go.etcd.io/etcd/server/v3/storage/mvcc.TestHashKVWhenCompacting.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/storage/mvcc/kvstore_test.go:572 +0x108
created by go.etcd.io/etcd/server/v3/storage/mvcc.TestHashKVWhenCompacting in goroutine 715
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/storage/mvcc/kvstore_test.go:569 +0x31e
FAIL    go.etcd.io/etcd/server/v3/storage/mvcc  28.311s
@ahrtr
Copy link
Member

ahrtr commented Jan 13, 2024

I ran go clean --testcache; make test-unit many times ( > 10 times) on both MacOS and Linux, but couldn't reproduce the issue. Could you provide more details how did you reproduce this issue?

@KosovGrigorii
Copy link
Author

KosovGrigorii commented Jan 17, 2024

Thank you for your answer. I was curious whether my device is the problem, so ran go clean --testcache; make test-unit on a different mac but the issue stays. After i clone etcd-io/etcd repo and open it locally i just run

go mod tidy
go clean --testcache; make test-unit

and every time it gives one of these two errors on go.etcd.io/etcd/server/v3/storage/mvcc test:

panic: schedule: schedule to stopped scheduler

goroutine 715 [running]:
go.etcd.io/etcd/pkg/v3/schedule.(*fifo).Schedule(0xc000146c40, {0x1e4e7a0?, 0xc001205200})
        /Users/grigorijkosov/Desktop/etcd/pkg/schedule/schedule.go:111 +0x279
go.etcd.io/etcd/server/v3/storage/mvcc.(*store).compact(0xc0001c7ba0, 0xc000883f68, 0x26d6, 0x26d5, 0x1)
        /Users/grigorijkosov/Desktop/etcd/server/storage/mvcc/kvstore.go:252 +0x1f0
go.etcd.io/etcd/server/v3/storage/mvcc.(*store).Compact(0xc0001c7ba0, 0xc000883f68, 0x16a2e34?)
        /Users/grigorijkosov/Desktop/etcd/server/storage/mvcc/kvstore.go:278 +0x312
go.etcd.io/etcd/server/v3/storage/mvcc.TestHashKVWhenCompacting.func3()
        /Users/grigorijkosov/Desktop/etcd/server/storage/mvcc/kvstore_test.go:610 +0x1e7
created by go.etcd.io/etcd/server/v3/storage/mvcc.TestHashKVWhenCompacting in goroutine 665
        /Users/grigorijkosov/Desktop/etcd/server/storage/mvcc/kvstore_test.go:607 +0x80b
FAIL    go.etcd.io/etcd/server/v3/storage/mvcc  58.444s
ok      go.etcd.io/etcd/server/v3/storage/schema        40.911s
ok      go.etcd.io/etcd/server/v3/storage/wal   51.019s
ok      go.etcd.io/etcd/server/v3/storage/wal/walpb     1.555s
FAIL
FAIL: (code:1):
  % (cd server && env ETCD_VERIFY=all go test ./... -short -timeout=3m --race)
There was a Failure in module server, aborting...
FAIL: 'unit' FAILED at Wed Jan 17 09:13:50 +05 2024

or this one

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x16a1607]

goroutine 711 [running]:
go.etcd.io/bbolt.(*Tx).Bucket(...)
        /Users/grigorijkosov/go/pkg/mod/go.etcd.io/bbolt@v1.4.0-alpha.0/tx.go:112
go.etcd.io/etcd/server/v3/storage/backend.unsafeForEach(0x0, {0x1e542f8, 0x23dd680}, 0xc000883ac8?)
        /Users/grigorijkosov/Desktop/etcd/server/storage/backend/batch_tx.go:233 +0x47
go.etcd.io/etcd/server/v3/storage/backend.(*baseReadTx).UnsafeForEach(0xc0000da140, {0x1e542f8, 0x23dd680}, 0xc000312180)
        /Users/grigorijkosov/Desktop/etcd/server/storage/backend/read_tx.go:70 +0x210
go.etcd.io/etcd/server/v3/storage/mvcc.unsafeHashByRev({0x793ac18, 0xc0000da140}, 0x105b2df?, 0xc000883d20?, 0xc000883f00?)
        /Users/grigorijkosov/Desktop/etcd/server/storage/mvcc/hash.go:35 +0x1b5
go.etcd.io/etcd/server/v3/storage/mvcc.(*store).hashByRev(0xc000609040, 0x2710)
        /Users/grigorijkosov/Desktop/etcd/server/storage/mvcc/kvstore.go:187 +0x3ab
go.etcd.io/etcd/server/v3/storage/mvcc.(*hashStorage).HashByRev(0xc0000d9040, 0x2710)
        /Users/grigorijkosov/Desktop/etcd/server/storage/mvcc/hash.go:139 +0x1e5
go.etcd.io/etcd/server/v3/storage/mvcc.TestHashKVWhenCompacting.func1()
        /Users/grigorijkosov/Desktop/etcd/server/storage/mvcc/kvstore_test.go:572 +0x126
created by go.etcd.io/etcd/server/v3/storage/mvcc.TestHashKVWhenCompacting in goroutine 683
        /Users/grigorijkosov/Desktop/etcd/server/storage/mvcc/kvstore_test.go:569 +0x370
FAIL    go.etcd.io/etcd/server/v3/storage/mvcc  57.790s
ok      go.etcd.io/etcd/server/v3/storage/schema        39.635s
ok      go.etcd.io/etcd/server/v3/storage/wal   51.177s
ok      go.etcd.io/etcd/server/v3/storage/wal/walpb     0.677s
FAIL
FAIL: (code:1):
  % (cd server && env ETCD_VERIFY=all go test ./... -short -timeout=3m --race)
There was a Failure in module server, aborting...
FAIL: 'unit' FAILED at Wed Jan 17 09:23:22 +05 2024
make: *** [test-unit] Error 255

I don't do any additional work neither do I change anything in the repo

@ahrtr
Copy link
Member

ahrtr commented Jan 17, 2024

Could you execute go env and provide the output?

It seems that it's similar to #17223, somehow the backend has already been closed during the test.

@KosovGrigorii
Copy link
Author

go env output:

GO111MODULE='on'
GOARCH='amd64'
GOBIN=''
GOCACHE='/Users/kosovgrigorii/Library/Caches/go-build'
GOENV='/Users/kosovgrigorii/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/kosovgrigorii/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/kosovgrigorii/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/opt/go/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/opt/go/libexec/pkg/tool/darwin_amd64'
GOVCS=''
GOVERSION='go1.21.3'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/Users/kosovgrigorii/Documents/GitHub/etcd/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/go-build2786560262=/tmp/go-build -gno-record-gcc-switches -fno-common'

@KosovGrigorii
Copy link
Author

KosovGrigorii commented Jan 18, 2024

I also noticed that sometimes during the execution of unit tests go.etcd.io/etcd/server/v3/storage/backend also fails(noticed it a couple times, unfortunately did not copy the output). Yet after I rerun tests everything goes fine. Could this be related to the same issue?

@ahrtr
Copy link
Member

ahrtr commented Jan 20, 2024

@KosovGrigorii
Can you please double check whether you can see similar issue on release-3.5? thx

@KosovGrigorii
Copy link
Author

KosovGrigorii commented Jan 22, 2024

Yes, the issue appears in release-3.5

Logs:

panic
{"level":"info","msg":"compact tree index","revision":1}
{"level":"info","msg":"compact tree index","revision":2}
{"level":"info","msg":"compact tree index","revision":3}
{"level":"info","msg":"compact tree index","revision":4}
{"level":"info","msg":"compact tree index","revision":5}
{"level":"info","msg":"compact tree index","revision":6}
{"level":"info","msg":"compact tree index","revision":7}
{"level":"info","msg":"compact tree index","revision":8}
{"level":"info","msg":"compact tree index","revision":9}
{"level":"info","msg":"compact tree index","revision":10}
{"level":"info","msg":"compact tree index","revision":11}
{"level":"info","msg":"compact tree index","revision":12}
{"level":"info","msg":"compact tree index","revision":13}
{"level":"info","msg":"compact tree index","revision":14}
{"level":"info","msg":"compact tree index","revision":15}
{"level":"info","msg":"compact tree index","revision":16}
{"level":"info","msg":"compact tree index","revision":17}
{"level":"info","msg":"compact tree index","revision":18}
{"level":"info","msg":"compact tree index","revision":19}
{"level":"info","msg":"compact tree index","revision":1}
{"level":"info","msg":"compact tree index","revision":2}
{"level":"info","msg":"compact tree index","revision":3}
{"level":"info","msg":"compact tree index","revision":4}
{"level":"info","msg":"compact tree index","revision":5}
{"level":"info","msg":"compact tree index","revision":6}
{"level":"info","msg":"compact tree index","revision":7}
{"level":"info","msg":"compact tree index","revision":8}
{"level":"info","msg":"compact tree index","revision":9}
{"level":"info","msg":"compact tree index","revision":10}
{"level":"info","msg":"compact tree index","revision":11}
{"level":"info","msg":"compact tree index","revision":12}
{"level":"info","msg":"compact tree index","revision":13}
{"level":"info","msg":"compact tree index","revision":14}
{"level":"info","msg":"compact tree index","revision":15}
{"level":"info","msg":"compact tree index","revision":16}
{"level":"info","msg":"compact tree index","revision":17}
{"level":"info","msg":"compact tree index","revision":18}
{"level":"info","msg":"compact tree index","revision":19}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":4}
{"level":"info","msg":"finished scheduled compaction","compact-revision":4,"took":"174.495µs","hash":1500391675}
{"level":"info","msg":"storing new hash","hash":1500391675,"revision":4,"compact-revision":-1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":4}
{"level":"info","msg":"finished scheduled compaction","compact-revision":4,"took":"134.547µs","hash":1500391675}
{"level":"info","msg":"storing new hash","hash":1500391675,"revision":4,"compact-revision":-1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":1}
{"level":"info","msg":"finished scheduled compaction","compact-revision":1,"took":"64.910207ms","hash":1084519789}
{"level":"info","msg":"storing new hash","hash":1084519789,"revision":1,"compact-revision":-1}
{"level":"info","msg":"compact tree index","revision":2}
{"level":"info","msg":"finished scheduled compaction","compact-revision":2,"took":"76.786142ms","hash":551221740}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":3}
{"level":"info","msg":"finished scheduled compaction","compact-revision":3,"took":"74.095407ms","hash":4023298219}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":0}
{"level":"info","msg":"finished scheduled compaction","compact-revision":0,"took":"76.968832ms","hash":1084519789}
{"level":"info","msg":"storing new hash","hash":1084519789,"revision":0,"compact-revision":-1}
{"level":"info","msg":"compact tree index","revision":1}
{"level":"info","msg":"finished scheduled compaction","compact-revision":1,"took":"77.318506ms","hash":1084519789}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":5}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":4}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":1}
{"level":"info","msg":"finished scheduled compaction","compact-revision":1,"took":"222.451µs","hash":1084519789}
{"level":"info","msg":"storing new hash","hash":1084519789,"revision":1,"compact-revision":-1}
{"level":"info","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":1}
{"level":"info","msg":"kvstore restored","current-rev":3}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":4}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":5}
{"level":"info","msg":"finished scheduled compaction","compact-revision":5,"took":"85.361165ms","hash":873998276}
{"level":"info","msg":"storing new hash","hash":873998276,"revision":5,"compact-revision":-1}
{"level":"info","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":5}
{"level":"info","msg":"kvstore restored","current-rev":5}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"finished scheduled compaction","compact-revision":3,"took":"11.806µs","hash":3457566959}
{"level":"info","msg":"storing new hash","hash":3457566959,"revision":3,"compact-revision":-1}
{"level":"info","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":3}
{"level":"info","msg":"kvstore restored","current-rev":5}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":36}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":4}
{"level":"info","msg":"resume scheduled compaction","meta-bucket-name":"meta","meta-bucket-name-key":"scheduledCompactRev","scheduled-compact-revision":2}
{"level":"info","msg":"compact tree index","revision":2}
{"level":"info","msg":"finished scheduled compaction","compact-revision":2,"took":"63.758µs","hash":3305255506}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":9900}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9900,"took":"1.478744127s","hash":2982592828}
{"level":"info","msg":"storing new hash","hash":2982592828,"revision":9900,"compact-revision":-1}
{"level":"info","msg":"compact tree index","revision":9901}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9901,"took":"76.019267ms","hash":4092902393}
{"level":"info","msg":"storing new hash","hash":4092902393,"revision":9901,"compact-revision":9900}
{"level":"info","msg":"compact tree index","revision":9902}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9902,"took":"84.977714ms","hash":3442946395}
{"level":"info","msg":"storing new hash","hash":3442946395,"revision":9902,"compact-revision":9901}
{"level":"info","msg":"compact tree index","revision":9903}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9903,"took":"60.924531ms","hash":3934476647}
{"level":"info","msg":"storing new hash","hash":3934476647,"revision":9903,"compact-revision":9902}
{"level":"info","msg":"compact tree index","revision":9904}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9904,"took":"95.606132ms","hash":1979317650}
{"level":"info","msg":"storing new hash","hash":1979317650,"revision":9904,"compact-revision":9903}
{"level":"info","msg":"compact tree index","revision":9905}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9905,"took":"85.171604ms","hash":4126965741}
{"level":"info","msg":"storing new hash","hash":4126965741,"revision":9905,"compact-revision":9904}
{"level":"info","msg":"compact tree index","revision":9906}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9906,"took":"61.451301ms","hash":3409966415}
{"level":"info","msg":"storing new hash","hash":3409966415,"revision":9906,"compact-revision":9905}
{"level":"info","msg":"compact tree index","revision":9907}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9907,"took":"75.799342ms","hash":3968572787}
{"level":"info","msg":"storing new hash","hash":3968572787,"revision":9907,"compact-revision":9906}
{"level":"info","msg":"compact tree index","revision":9908}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9908,"took":"101.140248ms","hash":1013037252}
{"level":"info","msg":"storing new hash","hash":1013037252,"revision":9908,"compact-revision":9907}
{"level":"info","msg":"compact tree index","revision":9909}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9909,"took":"59.372802ms","hash":1305141790}
{"level":"info","msg":"storing new hash","hash":1305141790,"revision":9909,"compact-revision":9908}
{"level":"info","msg":"compact tree index","revision":9910}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9910,"took":"109.804017ms","hash":189389043}
{"level":"info","msg":"storing new hash","hash":189389043,"revision":9910,"compact-revision":9909}
{"level":"info","msg":"compact tree index","revision":9911}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9911,"took":"66.589971ms","hash":754823375}
{"level":"info","msg":"storing new hash","hash":754823375,"revision":9911,"compact-revision":9910}
{"level":"info","msg":"compact tree index","revision":9912}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9912,"took":"54.719516ms","hash":4067225043}
{"level":"info","msg":"storing new hash","hash":4067225043,"revision":9912,"compact-revision":9911}
{"level":"info","msg":"compact tree index","revision":9913}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9913,"took":"63.422885ms","hash":804901573}
{"level":"info","msg":"storing new hash","hash":804901573,"revision":9913,"compact-revision":9912}
{"level":"info","msg":"compact tree index","revision":9914}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9914,"took":"80.901952ms","hash":3600863096}
{"level":"info","msg":"storing new hash","hash":3600863096,"revision":9914,"compact-revision":9913}
{"level":"info","msg":"compact tree index","revision":9915}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9915,"took":"75.475986ms","hash":4044662596}
{"level":"info","msg":"storing new hash","hash":4044662596,"revision":9915,"compact-revision":9914}
{"level":"info","msg":"compact tree index","revision":9916}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9916,"took":"60.357299ms","hash":1824345379}
{"level":"info","msg":"storing new hash","hash":1824345379,"revision":9916,"compact-revision":9915}
{"level":"info","msg":"compact tree index","revision":9917}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9917,"took":"60.719195ms","hash":487987193}
{"level":"info","msg":"storing new hash","hash":487987193,"revision":9917,"compact-revision":9916}
{"level":"info","msg":"compact tree index","revision":9918}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9918,"took":"76.836096ms","hash":1855345291}
{"level":"info","msg":"storing new hash","hash":1855345291,"revision":9918,"compact-revision":9917}
{"level":"info","msg":"compact tree index","revision":9919}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9919,"took":"63.523466ms","hash":1226980023}
{"level":"info","msg":"storing new hash","hash":1226980023,"revision":9919,"compact-revision":9918}
{"level":"info","msg":"compact tree index","revision":9920}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9920,"took":"59.038906ms","hash":1439986907}
{"level":"info","msg":"storing new hash","hash":1439986907,"revision":9920,"compact-revision":9919}
{"level":"info","msg":"compact tree index","revision":9921}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9921,"took":"94.143915ms","hash":492703523}
{"level":"info","msg":"storing new hash","hash":492703523,"revision":9921,"compact-revision":9920}
{"level":"info","msg":"compact tree index","revision":9922}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9922,"took":"65.615955ms","hash":597525889}
{"level":"info","msg":"storing new hash","hash":597525889,"revision":9922,"compact-revision":9921}
{"level":"info","msg":"compact tree index","revision":9923}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9923,"took":"60.610697ms","hash":69856701}
{"level":"info","msg":"storing new hash","hash":69856701,"revision":9923,"compact-revision":9922}
{"level":"info","msg":"compact tree index","revision":9924}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9924,"took":"81.818367ms","hash":1578799301}
{"level":"info","msg":"storing new hash","hash":1578799301,"revision":9924,"compact-revision":9923}
{"level":"info","msg":"compact tree index","revision":9925}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9925,"took":"120.143678ms","hash":800180767}
{"level":"info","msg":"storing new hash","hash":800180767,"revision":9925,"compact-revision":9924}
{"level":"info","msg":"compact tree index","revision":9926}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9926,"took":"101.373445ms","hash":292700349}
{"level":"info","msg":"storing new hash","hash":292700349,"revision":9926,"compact-revision":9925}
{"level":"info","msg":"compact tree index","revision":9927}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9927,"took":"63.107421ms","hash":918956161}
{"level":"info","msg":"storing new hash","hash":918956161,"revision":9927,"compact-revision":9926}
{"level":"info","msg":"compact tree index","revision":9928}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9928,"took":"59.07375ms","hash":2769599053}
{"level":"info","msg":"storing new hash","hash":2769599053,"revision":9928,"compact-revision":9927}
{"level":"info","msg":"compact tree index","revision":9929}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9929,"took":"81.223432ms","hash":2021754203}
{"level":"info","msg":"storing new hash","hash":2021754203,"revision":9929,"compact-revision":9928}
{"level":"info","msg":"compact tree index","revision":9930}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9930,"took":"77.898623ms","hash":1040367542}
{"level":"info","msg":"storing new hash","hash":1040367542,"revision":9930,"compact-revision":9929}
{"level":"info","msg":"compact tree index","revision":9931}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9931,"took":"66.902648ms","hash":431413130}
{"level":"info","msg":"storing new hash","hash":431413130,"revision":9931,"compact-revision":9930}
{"level":"info","msg":"compact tree index","revision":9932}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9932,"took":"61.574665ms","hash":2216639981}
{"level":"info","msg":"storing new hash","hash":2216639981,"revision":9932,"compact-revision":9931}
{"level":"info","msg":"compact tree index","revision":9933}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9933,"took":"66.268478ms","hash":4122240823}
{"level":"info","msg":"storing new hash","hash":4122240823,"revision":9933,"compact-revision":9932}
{"level":"info","msg":"compact tree index","revision":9934}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9934,"took":"59.993294ms","hash":216889994}
{"level":"info","msg":"storing new hash","hash":216889994,"revision":9934,"compact-revision":9933}
{"level":"info","msg":"compact tree index","revision":9935}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9935,"took":"21.252298ms","hash":727257782}
{"level":"info","msg":"storing new hash","hash":727257782,"revision":9935,"compact-revision":9934}
{"level":"info","msg":"compact tree index","revision":9936}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9936,"took":"92.412716ms","hash":1941512540}
{"level":"info","msg":"storing new hash","hash":1941512540,"revision":9936,"compact-revision":9935}
{"level":"info","msg":"compact tree index","revision":9937}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9937,"took":"34.984437ms","hash":4089226019}
{"level":"info","msg":"storing new hash","hash":4089226019,"revision":9937,"compact-revision":9936}
{"level":"info","msg":"compact tree index","revision":9938}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9938,"took":"91.999172ms","hash":2151474769}
{"level":"info","msg":"storing new hash","hash":2151474769,"revision":9938,"compact-revision":9937}
{"level":"info","msg":"compact tree index","revision":9939}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9939,"took":"98.715389ms","hash":2810744429}
{"level":"info","msg":"storing new hash","hash":2810744429,"revision":9939,"compact-revision":9938}
{"level":"info","msg":"compact tree index","revision":9940}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9940,"took":"47.039225ms","hash":732857719}
{"level":"info","msg":"storing new hash","hash":732857719,"revision":9940,"compact-revision":9939}
{"level":"info","msg":"compact tree index","revision":9941}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9941,"took":"46.081004ms","hash":1510334381}
{"level":"info","msg":"storing new hash","hash":1510334381,"revision":9941,"compact-revision":9940}
{"level":"info","msg":"compact tree index","revision":9942}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9942,"took":"82.081248ms","hash":1690719503}
{"level":"info","msg":"storing new hash","hash":1690719503,"revision":9942,"compact-revision":9941}
{"level":"info","msg":"compact tree index","revision":9943}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9943,"took":"111.899715ms","hash":1131560243}
{"level":"info","msg":"storing new hash","hash":1131560243,"revision":9943,"compact-revision":9942}
{"level":"info","msg":"compact tree index","revision":9944}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9944,"took":"82.301977ms","hash":1517942576}
{"level":"info","msg":"storing new hash","hash":1517942576,"revision":9944,"compact-revision":9943}
{"level":"info","msg":"compact tree index","revision":9945}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9945,"took":"64.405543ms","hash":2280441894}
{"level":"info","msg":"storing new hash","hash":2280441894,"revision":9945,"compact-revision":9944}
{"level":"info","msg":"compact tree index","revision":9946}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9946,"took":"59.5803ms","hash":3106882180}
{"level":"info","msg":"storing new hash","hash":3106882180,"revision":9946,"compact-revision":9945}
{"level":"info","msg":"compact tree index","revision":9947}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9947,"took":"64.277161ms","hash":2660969144}
{"level":"info","msg":"storing new hash","hash":2660969144,"revision":9947,"compact-revision":9946}
{"level":"info","msg":"compact tree index","revision":9948}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9948,"took":"77.969335ms","hash":1316032271}
{"level":"info","msg":"storing new hash","hash":1316032271,"revision":9948,"compact-revision":9947}
{"level":"info","msg":"compact tree index","revision":9949}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9949,"took":"82.483537ms","hash":1071273429}
{"level":"info","msg":"storing new hash","hash":1071273429,"revision":9949,"compact-revision":9948}
{"level":"info","msg":"compact tree index","revision":9950}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9950,"took":"47.381354ms","hash":2035904312}
{"level":"info","msg":"storing new hash","hash":2035904312,"revision":9950,"compact-revision":9949}
{"level":"info","msg":"compact tree index","revision":9951}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9951,"took":"35.916582ms","hash":1592592132}
{"level":"info","msg":"storing new hash","hash":1592592132,"revision":9951,"compact-revision":9950}
{"level":"info","msg":"compact tree index","revision":9952}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9952,"took":"94.707864ms","hash":251895795}
{"level":"info","msg":"storing new hash","hash":251895795,"revision":9952,"compact-revision":9951}
{"level":"info","msg":"compact tree index","revision":9953}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9953,"took":"94.795392ms","hash":874997598}
{"level":"info","msg":"storing new hash","hash":874997598,"revision":9953,"compact-revision":9952}
{"level":"info","msg":"compact tree index","revision":9954}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9954,"took":"93.921156ms","hash":3447618275}
{"level":"info","msg":"storing new hash","hash":3447618275,"revision":9954,"compact-revision":9953}
{"level":"info","msg":"compact tree index","revision":9955}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9955,"took":"69.639852ms","hash":3939111647}
{"level":"info","msg":"storing new hash","hash":3939111647,"revision":9955,"compact-revision":9954}
{"level":"info","msg":"compact tree index","revision":9956}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9956,"took":"70.268175ms","hash":2003040440}
{"level":"info","msg":"storing new hash","hash":2003040440,"revision":9956,"compact-revision":9955}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x16535e7]

goroutine 684 [running]:
go.etcd.io/bbolt.(*Tx).Bucket(...)
        /Users/kosovgrigorii/go/pkg/mod/go.etcd.io/bbolt@v1.3.8/tx.go:102
go.etcd.io/etcd/server/v3/mvcc/backend.unsafeForEach(0x0, {0x1d28ee8, 0x22417e0}, 0x2000?)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/batch_tx.go:235 +0x47
go.etcd.io/etcd/server/v3/mvcc/backend.(*baseReadTx).UnsafeForEach(0xc000440190, {0x1d28ee8, 0x22417e0}, 0xc000300010)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/read_tx.go:69 +0x199
go.etcd.io/etcd/server/v3/mvcc.unsafeHashByRev({0x1d2a438, 0xc000440190}, 0xc000391f00?, 0xc000391f00?, 0xc00031d040?)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/hash.go:34 +0x1b5
go.etcd.io/etcd/server/v3/mvcc.(*store).hashByRev(0xc00034a5b0, 0x2710)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/kvstore.go:194 +0x385
go.etcd.io/etcd/server/v3/mvcc.(*hashStorage).HashByRev(0xc0006f0180, 0x2710)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/hash.go:133 +0x1d2
go.etcd.io/etcd/server/v3/mvcc.TestHashKVWhenCompacting.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/kvstore_test.go:561 +0x108
created by go.etcd.io/etcd/server/v3/mvcc.TestHashKVWhenCompacting in goroutine 524
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/kvstore_test.go:558 +0x3be
FAIL    go.etcd.io/etcd/server/v3/mvcc  36.796s

When I run just this test all goes well:

``` === RUN TestHashKVWhenCompacting {"level":"info","msg":"kvstore restored","current-rev":1} {"level":"info","msg":"compact tree index","revision":9900} {"level":"info","msg":"finished scheduled compaction","compact-revision":9900,"took":"531.117894ms","hash":2982592828} {"level":"info","msg":"storing new hash","hash":2982592828,"revision":9900,"compact-revision":-1} {"level":"info","msg":"compact tree index","revision":9901} {"level":"info","msg":"finished scheduled compaction","compact-revision":9901,"took":"37.033445ms","hash":4092902393} {"level":"info","msg":"storing new hash","hash":4092902393,"revision":9901,"compact-revision":9900} {"level":"info","msg":"compact tree index","revision":9902} {"level":"info","msg":"finished scheduled compaction","compact-revision":9902,"took":"38.986793ms","hash":3442946395} {"level":"info","msg":"storing new hash","hash":3442946395,"revision":9902,"compact-revision":9901} {"level":"info","msg":"compact tree index","revision":9903} {"level":"info","msg":"finished scheduled compaction","compact-revision":9903,"took":"39.098304ms","hash":3934476647} {"level":"info","msg":"storing new hash","hash":3934476647,"revision":9903,"compact-revision":9902} {"level":"info","msg":"compact tree index","revision":9904} {"level":"info","msg":"finished scheduled compaction","compact-revision":9904,"took":"39.604967ms","hash":1979317650} {"level":"info","msg":"storing new hash","hash":1979317650,"revision":9904,"compact-revision":9903} {"level":"info","msg":"compact tree index","revision":9905} {"level":"info","msg":"finished scheduled compaction","compact-revision":9905,"took":"39.474176ms","hash":4126965741} {"level":"info","msg":"storing new hash","hash":4126965741,"revision":9905,"compact-revision":9904} {"level":"info","msg":"compact tree index","revision":9906} {"level":"info","msg":"finished scheduled compaction","compact-revision":9906,"took":"39.967073ms","hash":3409966415} {"level":"info","msg":"storing new hash","hash":3409966415,"revision":9906,"compact-revision":9905} {"level":"info","msg":"compact tree index","revision":9907} {"level":"info","msg":"finished scheduled compaction","compact-revision":9907,"took":"38.439449ms","hash":3968572787} {"level":"info","msg":"storing new hash","hash":3968572787,"revision":9907,"compact-revision":9906} {"level":"info","msg":"compact tree index","revision":9908} {"level":"info","msg":"finished scheduled compaction","compact-revision":9908,"took":"57.393176ms","hash":1013037252} {"level":"info","msg":"storing new hash","hash":1013037252,"revision":9908,"compact-revision":9907} {"level":"info","msg":"compact tree index","revision":9909} {"level":"info","msg":"finished scheduled compaction","compact-revision":9909,"took":"39.863965ms","hash":1305141790} {"level":"info","msg":"storing new hash","hash":1305141790,"revision":9909,"compact-revision":9908} {"level":"info","msg":"compact tree index","revision":9910} {"level":"info","msg":"finished scheduled compaction","compact-revision":9910,"took":"38.088003ms","hash":189389043} {"level":"info","msg":"storing new hash","hash":189389043,"revision":9910,"compact-revision":9909} {"level":"info","msg":"compact tree index","revision":9911} {"level":"info","msg":"finished scheduled compaction","compact-revision":9911,"took":"38.941662ms","hash":754823375} {"level":"info","msg":"storing new hash","hash":754823375,"revision":9911,"compact-revision":9910} {"level":"info","msg":"compact tree index","revision":9912} {"level":"info","msg":"finished scheduled compaction","compact-revision":9912,"took":"39.274764ms","hash":4067225043} {"level":"info","msg":"storing new hash","hash":4067225043,"revision":9912,"compact-revision":9911} {"level":"info","msg":"compact tree index","revision":9913} {"level":"info","msg":"finished scheduled compaction","compact-revision":9913,"took":"39.844201ms","hash":804901573} {"level":"info","msg":"storing new hash","hash":804901573,"revision":9913,"compact-revision":9912} {"level":"info","msg":"compact tree index","revision":9914} {"level":"info","msg":"finished scheduled compaction","compact-revision":9914,"took":"41.744758ms","hash":3600863096} {"level":"info","msg":"storing new hash","hash":3600863096,"revision":9914,"compact-revision":9913} {"level":"info","msg":"compact tree index","revision":9915} {"level":"info","msg":"finished scheduled compaction","compact-revision":9915,"took":"38.780256ms","hash":4044662596} {"level":"info","msg":"storing new hash","hash":4044662596,"revision":9915,"compact-revision":9914} {"level":"info","msg":"compact tree index","revision":9916} {"level":"info","msg":"finished scheduled compaction","compact-revision":9916,"took":"38.495205ms","hash":1824345379} {"level":"info","msg":"storing new hash","hash":1824345379,"revision":9916,"compact-revision":9915} {"level":"info","msg":"compact tree index","revision":9917} {"level":"info","msg":"finished scheduled compaction","compact-revision":9917,"took":"37.077781ms","hash":487987193} {"level":"info","msg":"storing new hash","hash":487987193,"revision":9917,"compact-revision":9916} {"level":"info","msg":"compact tree index","revision":9918} {"level":"info","msg":"finished scheduled compaction","compact-revision":9918,"took":"37.465793ms","hash":1855345291} {"level":"info","msg":"storing new hash","hash":1855345291,"revision":9918,"compact-revision":9917} {"level":"info","msg":"compact tree index","revision":9919} {"level":"info","msg":"finished scheduled compaction","compact-revision":9919,"took":"39.528318ms","hash":1226980023} {"level":"info","msg":"storing new hash","hash":1226980023,"revision":9919,"compact-revision":9918} {"level":"info","msg":"compact tree index","revision":9920} {"level":"info","msg":"finished scheduled compaction","compact-revision":9920,"took":"38.043078ms","hash":1439986907} {"level":"info","msg":"storing new hash","hash":1439986907,"revision":9920,"compact-revision":9919} {"level":"info","msg":"compact tree index","revision":9921} {"level":"info","msg":"finished scheduled compaction","compact-revision":9921,"took":"37.108918ms","hash":492703523} {"level":"info","msg":"storing new hash","hash":492703523,"revision":9921,"compact-revision":9920} {"level":"info","msg":"compact tree index","revision":9922} {"level":"info","msg":"finished scheduled compaction","compact-revision":9922,"took":"38.038564ms","hash":597525889} {"level":"info","msg":"storing new hash","hash":597525889,"revision":9922,"compact-revision":9921} {"level":"info","msg":"compact tree index","revision":9923} {"level":"info","msg":"finished scheduled compaction","compact-revision":9923,"took":"37.480654ms","hash":69856701} {"level":"info","msg":"storing new hash","hash":69856701,"revision":9923,"compact-revision":9922} {"level":"info","msg":"compact tree index","revision":9924} {"level":"info","msg":"finished scheduled compaction","compact-revision":9924,"took":"37.161385ms","hash":1578799301} {"level":"info","msg":"storing new hash","hash":1578799301,"revision":9924,"compact-revision":9923} {"level":"info","msg":"compact tree index","revision":9925} {"level":"info","msg":"finished scheduled compaction","compact-revision":9925,"took":"37.993913ms","hash":800180767} {"level":"info","msg":"storing new hash","hash":800180767,"revision":9925,"compact-revision":9924} {"level":"info","msg":"compact tree index","revision":9926} {"level":"info","msg":"finished scheduled compaction","compact-revision":9926,"took":"37.489503ms","hash":292700349} {"level":"info","msg":"storing new hash","hash":292700349,"revision":9926,"compact-revision":9925} {"level":"info","msg":"compact tree index","revision":9927} {"level":"info","msg":"finished scheduled compaction","compact-revision":9927,"took":"37.596499ms","hash":918956161} {"level":"info","msg":"storing new hash","hash":918956161,"revision":9927,"compact-revision":9926} {"level":"info","msg":"compact tree index","revision":9928} {"level":"info","msg":"finished scheduled compaction","compact-revision":9928,"took":"37.824566ms","hash":2769599053} {"level":"info","msg":"storing new hash","hash":2769599053,"revision":9928,"compact-revision":9927} {"level":"info","msg":"compact tree index","revision":9929} {"level":"info","msg":"finished scheduled compaction","compact-revision":9929,"took":"39.38965ms","hash":2021754203} {"level":"info","msg":"storing new hash","hash":2021754203,"revision":9929,"compact-revision":9928} {"level":"info","msg":"compact tree index","revision":9930} {"level":"info","msg":"finished scheduled compaction","compact-revision":9930,"took":"40.007752ms","hash":1040367542} {"level":"info","msg":"storing new hash","hash":1040367542,"revision":9930,"compact-revision":9929} {"level":"info","msg":"compact tree index","revision":9931} {"level":"info","msg":"finished scheduled compaction","compact-revision":9931,"took":"39.483494ms","hash":431413130} {"level":"info","msg":"storing new hash","hash":431413130,"revision":9931,"compact-revision":9930} {"level":"info","msg":"compact tree index","revision":9932} {"level":"info","msg":"finished scheduled compaction","compact-revision":9932,"took":"38.606602ms","hash":2216639981} {"level":"info","msg":"storing new hash","hash":2216639981,"revision":9932,"compact-revision":9931} {"level":"info","msg":"compact tree index","revision":9933} {"level":"info","msg":"finished scheduled compaction","compact-revision":9933,"took":"38.808989ms","hash":4122240823} {"level":"info","msg":"storing new hash","hash":4122240823,"revision":9933,"compact-revision":9932} {"level":"info","msg":"compact tree index","revision":9934} {"level":"info","msg":"finished scheduled compaction","compact-revision":9934,"took":"38.559068ms","hash":216889994} {"level":"info","msg":"storing new hash","hash":216889994,"revision":9934,"compact-revision":9933} {"level":"info","msg":"compact tree index","revision":9935} {"level":"info","msg":"finished scheduled compaction","compact-revision":9935,"took":"39.931832ms","hash":727257782} {"level":"info","msg":"storing new hash","hash":727257782,"revision":9935,"compact-revision":9934} {"level":"info","msg":"compact tree index","revision":9936} {"level":"info","msg":"finished scheduled compaction","compact-revision":9936,"took":"38.992033ms","hash":1941512540} {"level":"info","msg":"storing new hash","hash":1941512540,"revision":9936,"compact-revision":9935} {"level":"info","msg":"compact tree index","revision":9937} {"level":"info","msg":"finished scheduled compaction","compact-revision":9937,"took":"40.052351ms","hash":4089226019} {"level":"info","msg":"storing new hash","hash":4089226019,"revision":9937,"compact-revision":9936} {"level":"info","msg":"compact tree index","revision":9938} {"level":"info","msg":"finished scheduled compaction","compact-revision":9938,"took":"39.876929ms","hash":2151474769} {"level":"info","msg":"storing new hash","hash":2151474769,"revision":9938,"compact-revision":9937} {"level":"info","msg":"compact tree index","revision":9939} {"level":"info","msg":"finished scheduled compaction","compact-revision":9939,"took":"38.000922ms","hash":2810744429} {"level":"info","msg":"storing new hash","hash":2810744429,"revision":9939,"compact-revision":9938} {"level":"info","msg":"compact tree index","revision":9940} {"level":"info","msg":"finished scheduled compaction","compact-revision":9940,"took":"40.154967ms","hash":732857719} {"level":"info","msg":"storing new hash","hash":732857719,"revision":9940,"compact-revision":9939} {"level":"info","msg":"compact tree index","revision":9941} {"level":"info","msg":"finished scheduled compaction","compact-revision":9941,"took":"39.449725ms","hash":1510334381} {"level":"info","msg":"storing new hash","hash":1510334381,"revision":9941,"compact-revision":9940} {"level":"info","msg":"compact tree index","revision":9942} {"level":"info","msg":"finished scheduled compaction","compact-revision":9942,"took":"39.209631ms","hash":1690719503} {"level":"info","msg":"storing new hash","hash":1690719503,"revision":9942,"compact-revision":9941} {"level":"info","msg":"compact tree index","revision":9943} {"level":"info","msg":"finished scheduled compaction","compact-revision":9943,"took":"24.591389ms","hash":1131560243} {"level":"info","msg":"storing new hash","hash":1131560243,"revision":9943,"compact-revision":9942} {"level":"info","msg":"compact tree index","revision":9944} {"level":"info","msg":"finished scheduled compaction","compact-revision":9944,"took":"55.477173ms","hash":1517942576} {"level":"info","msg":"storing new hash","hash":1517942576,"revision":9944,"compact-revision":9943} {"level":"info","msg":"compact tree index","revision":9945} {"level":"info","msg":"finished scheduled compaction","compact-revision":9945,"took":"38.211667ms","hash":2280441894} {"level":"info","msg":"storing new hash","hash":2280441894,"revision":9945,"compact-revision":9944} {"level":"info","msg":"compact tree index","revision":9946} {"level":"info","msg":"finished scheduled compaction","compact-revision":9946,"took":"39.594767ms","hash":3106882180} {"level":"info","msg":"storing new hash","hash":3106882180,"revision":9946,"compact-revision":9945} {"level":"info","msg":"compact tree index","revision":9947} {"level":"info","msg":"finished scheduled compaction","compact-revision":9947,"took":"37.51918ms","hash":2660969144} {"level":"info","msg":"storing new hash","hash":2660969144,"revision":9947,"compact-revision":9946} {"level":"info","msg":"compact tree index","revision":9948} {"level":"info","msg":"finished scheduled compaction","compact-revision":9948,"took":"23.447729ms","hash":1316032271} {"level":"info","msg":"storing new hash","hash":1316032271,"revision":9948,"compact-revision":9947} {"level":"info","msg":"compact tree index","revision":9949} {"level":"info","msg":"finished scheduled compaction","compact-revision":9949,"took":"37.430554ms","hash":1071273429} {"level":"info","msg":"storing new hash","hash":1071273429,"revision":9949,"compact-revision":9948} {"level":"info","msg":"compact tree index","revision":9950} {"level":"info","msg":"finished scheduled compaction","compact-revision":9950,"took":"40.04844ms","hash":2035904312} {"level":"info","msg":"storing new hash","hash":2035904312,"revision":9950,"compact-revision":9949} {"level":"info","msg":"compact tree index","revision":9951} {"level":"info","msg":"finished scheduled compaction","compact-revision":9951,"took":"38.970112ms","hash":1592592132} {"level":"info","msg":"storing new hash","hash":1592592132,"revision":9951,"compact-revision":9950} {"level":"info","msg":"compact tree index","revision":9952} {"level":"info","msg":"finished scheduled compaction","compact-revision":9952,"took":"40.220537ms","hash":251895795} {"level":"info","msg":"storing new hash","hash":251895795,"revision":9952,"compact-revision":9951} {"level":"info","msg":"compact tree index","revision":9953} {"level":"info","msg":"finished scheduled compaction","compact-revision":9953,"took":"39.441269ms","hash":874997598} {"level":"info","msg":"storing new hash","hash":874997598,"revision":9953,"compact-revision":9952} {"level":"info","msg":"compact tree index","revision":9954} {"level":"info","msg":"finished scheduled compaction","compact-revision":9954,"took":"37.44552ms","hash":3447618275} {"level":"info","msg":"storing new hash","hash":3447618275,"revision":9954,"compact-revision":9953} {"level":"info","msg":"compact tree index","revision":9955} {"level":"info","msg":"finished scheduled compaction","compact-revision":9955,"took":"38.924624ms","hash":3939111647} {"level":"info","msg":"storing new hash","hash":3939111647,"revision":9955,"compact-revision":9954} {"level":"info","msg":"compact tree index","revision":9956} {"level":"info","msg":"finished scheduled compaction","compact-revision":9956,"took":"39.186731ms","hash":2003040440} {"level":"info","msg":"storing new hash","hash":2003040440,"revision":9956,"compact-revision":9955} {"level":"info","msg":"compact tree index","revision":9957} {"level":"info","msg":"finished scheduled compaction","compact-revision":9957,"took":"37.946538ms","hash":113812066} {"level":"info","msg":"storing new hash","hash":113812066,"revision":9957,"compact-revision":9956} {"level":"info","msg":"compact tree index","revision":9958} {"level":"info","msg":"finished scheduled compaction","compact-revision":9958,"took":"38.518906ms","hash":1967712016} {"level":"info","msg":"storing new hash","hash":1967712016,"revision":9958,"compact-revision":9957} {"level":"info","msg":"compact tree index","revision":9959} {"level":"info","msg":"finished scheduled compaction","compact-revision":9959,"took":"39.858969ms","hash":1392283436} {"level":"info","msg":"storing new hash","hash":1392283436,"revision":9959,"compact-revision":9958} {"level":"info","msg":"compact tree index","revision":9960} {"level":"info","msg":"finished scheduled compaction","compact-revision":9960,"took":"39.178273ms","hash":3546829457} {"level":"info","msg":"storing new hash","hash":3546829457,"revision":9960,"compact-revision":9959} {"level":"info","msg":"compact tree index","revision":9961} {"level":"info","msg":"finished scheduled compaction","compact-revision":9961,"took":"37.6502ms","hash":251489671} {"level":"info","msg":"storing new hash","hash":251489671,"revision":9961,"compact-revision":9960} {"level":"info","msg":"compact tree index","revision":9962} {"level":"info","msg":"finished scheduled compaction","compact-revision":9962,"took":"39.551539ms","hash":809428773} {"level":"info","msg":"storing new hash","hash":809428773,"revision":9962,"compact-revision":9961} {"level":"info","msg":"compact tree index","revision":9963} {"level":"info","msg":"finished scheduled compaction","compact-revision":9963,"took":"37.710255ms","hash":394956569} {"level":"info","msg":"storing new hash","hash":394956569,"revision":9963,"compact-revision":9962} {"level":"info","msg":"compact tree index","revision":9964} {"level":"info","msg":"finished scheduled compaction","compact-revision":9964,"took":"39.803146ms","hash":1304031841} {"level":"info","msg":"storing new hash","hash":1304031841,"revision":9964,"compact-revision":9963} {"level":"info","msg":"compact tree index","revision":9965} {"level":"info","msg":"finished scheduled compaction","compact-revision":9965,"took":"38.816169ms","hash":1007855803} {"level":"info","msg":"storing new hash","hash":1007855803,"revision":9965,"compact-revision":9964} {"level":"info","msg":"compact tree index","revision":9966} {"level":"info","msg":"finished scheduled compaction","compact-revision":9966,"took":"39.686713ms","hash":47260185} {"level":"info","msg":"storing new hash","hash":47260185,"revision":9966,"compact-revision":9965} {"level":"info","msg":"compact tree index","revision":9967} {"level":"info","msg":"finished scheduled compaction","compact-revision":9967,"took":"38.509201ms","hash":627395109} {"level":"info","msg":"storing new hash","hash":627395109,"revision":9967,"compact-revision":9966} {"level":"info","msg":"compact tree index","revision":9968} {"level":"info","msg":"finished scheduled compaction","compact-revision":9968,"took":"39.902588ms","hash":818329119} {"level":"info","msg":"storing new hash","hash":818329119,"revision":9968,"compact-revision":9967} {"level":"info","msg":"compact tree index","revision":9969} {"level":"info","msg":"finished scheduled compaction","compact-revision":9969,"took":"37.931787ms","hash":2965582944} {"level":"info","msg":"storing new hash","hash":2965582944,"revision":9969,"compact-revision":9968} {"level":"info","msg":"compact tree index","revision":9970} {"level":"info","msg":"finished scheduled compaction","compact-revision":9970,"took":"39.124404ms","hash":4131393165} {"level":"info","msg":"storing new hash","hash":4131393165,"revision":9970,"compact-revision":9969} {"level":"info","msg":"compact tree index","revision":9971} {"level":"info","msg":"finished scheduled compaction","compact-revision":9971,"took":"39.378283ms","hash":3522455217} {"level":"info","msg":"storing new hash","hash":3522455217,"revision":9971,"compact-revision":9970} {"level":"info","msg":"compact tree index","revision":9972} {"level":"info","msg":"finished scheduled compaction","compact-revision":9972,"took":"38.292064ms","hash":1281199318} {"level":"info","msg":"storing new hash","hash":1281199318,"revision":9972,"compact-revision":9971} {"level":"info","msg":"compact tree index","revision":9973} {"level":"info","msg":"finished scheduled compaction","compact-revision":9973,"took":"24.604337ms","hash":1039586828} {"level":"info","msg":"storing new hash","hash":1039586828,"revision":9973,"compact-revision":9972} {"level":"info","msg":"compact tree index","revision":9974} {"level":"info","msg":"finished scheduled compaction","compact-revision":9974,"took":"38.473345ms","hash":3299855281} {"level":"info","msg":"storing new hash","hash":3299855281,"revision":9974,"compact-revision":9973} {"level":"info","msg":"compact tree index","revision":9975} {"level":"info","msg":"finished scheduled compaction","compact-revision":9975,"took":"38.241949ms","hash":3810239373} {"level":"info","msg":"storing new hash","hash":3810239373,"revision":9975,"compact-revision":9974} {"level":"info","msg":"compact tree index","revision":9976} {"level":"info","msg":"finished scheduled compaction","compact-revision":9976,"took":"39.40521ms","hash":1032519313} {"level":"info","msg":"storing new hash","hash":1032519313,"revision":9976,"compact-revision":9975} {"level":"info","msg":"compact tree index","revision":9977} {"level":"info","msg":"finished scheduled compaction","compact-revision":9977,"took":"39.980974ms","hash":3760177543} {"level":"info","msg":"storing new hash","hash":3760177543,"revision":9977,"compact-revision":9976} {"level":"info","msg":"compact tree index","revision":9978} {"level":"info","msg":"finished scheduled compaction","compact-revision":9978,"took":"38.796998ms","hash":2476705013} {"level":"info","msg":"storing new hash","hash":2476705013,"revision":9978,"compact-revision":9977} {"level":"info","msg":"compact tree index","revision":9979} {"level":"info","msg":"finished scheduled compaction","compact-revision":9979,"took":"56.449979ms","hash":3022777545} {"level":"info","msg":"storing new hash","hash":3022777545,"revision":9979,"compact-revision":9978} {"level":"info","msg":"compact tree index","revision":9980} {"level":"info","msg":"finished scheduled compaction","compact-revision":9980,"took":"43.590793ms","hash":2769888363} {"level":"info","msg":"storing new hash","hash":2769888363,"revision":9980,"compact-revision":9979} {"level":"info","msg":"compact tree index","revision":9981} {"level":"info","msg":"finished scheduled compaction","compact-revision":9981,"took":"40.39439ms","hash":3568467633} {"level":"info","msg":"storing new hash","hash":3568467633,"revision":9981,"compact-revision":9980} {"level":"info","msg":"compact tree index","revision":9982} {"level":"info","msg":"finished scheduled compaction","compact-revision":9982,"took":"39.958299ms","hash":3933336595} {"level":"info","msg":"storing new hash","hash":3933336595,"revision":9982,"compact-revision":9981} {"level":"info","msg":"compact tree index","revision":9983} {"level":"info","msg":"finished scheduled compaction","compact-revision":9983,"took":"24.134317ms","hash":3452279855} {"level":"info","msg":"storing new hash","hash":3452279855,"revision":9983,"compact-revision":9982} {"level":"info","msg":"compact tree index","revision":9984} {"level":"info","msg":"finished scheduled compaction","compact-revision":9984,"took":"38.06843ms","hash":4232457539} {"level":"info","msg":"storing new hash","hash":4232457539,"revision":9984,"compact-revision":9983} {"level":"info","msg":"compact tree index","revision":9985} {"level":"info","msg":"finished scheduled compaction","compact-revision":9985,"took":"39.131664ms","hash":3973754021} {"level":"info","msg":"storing new hash","hash":3973754021,"revision":9985,"compact-revision":9984} {"level":"info","msg":"compact tree index","revision":9986} {"level":"info","msg":"finished scheduled compaction","compact-revision":9986,"took":"40.618892ms","hash":3524863495} {"level":"info","msg":"storing new hash","hash":3524863495,"revision":9986,"compact-revision":9985} {"level":"info","msg":"compact tree index","revision":9987} {"level":"info","msg":"finished scheduled compaction","compact-revision":9987,"took":"39.11281ms","hash":4121775675} {"level":"info","msg":"storing new hash","hash":4121775675,"revision":9987,"compact-revision":9986} {"level":"info","msg":"compact tree index","revision":9988} {"level":"info","msg":"finished scheduled compaction","compact-revision":9988,"took":"39.766703ms","hash":625437580} {"level":"info","msg":"storing new hash","hash":625437580,"revision":9988,"compact-revision":9987} {"level":"info","msg":"compact tree index","revision":9989} {"level":"info","msg":"finished scheduled compaction","compact-revision":9989,"took":"38.581062ms","hash":1424765270} {"level":"info","msg":"storing new hash","hash":1424765270,"revision":9989,"compact-revision":9988} {"level":"info","msg":"compact tree index","revision":9990} {"level":"info","msg":"finished scheduled compaction","compact-revision":9990,"took":"39.832066ms","hash":309267387} {"level":"info","msg":"storing new hash","hash":309267387,"revision":9990,"compact-revision":9989} {"level":"info","msg":"compact tree index","revision":9991} {"level":"info","msg":"finished scheduled compaction","compact-revision":9991,"took":"39.830583ms","hash":903570311} {"level":"info","msg":"storing new hash","hash":903570311,"revision":9991,"compact-revision":9990} {"level":"info","msg":"compact tree index","revision":9992} {"level":"info","msg":"finished scheduled compaction","compact-revision":9992,"took":"38.373346ms","hash":3947511451} {"level":"info","msg":"storing new hash","hash":3947511451,"revision":9992,"compact-revision":9991} {"level":"info","msg":"compact tree index","revision":9993} {"level":"info","msg":"finished scheduled compaction","compact-revision":9993,"took":"56.062534ms","hash":920584589} {"level":"info","msg":"storing new hash","hash":920584589,"revision":9993,"compact-revision":9992} {"level":"info","msg":"compact tree index","revision":9994} {"level":"info","msg":"finished scheduled compaction","compact-revision":9994,"took":"37.849921ms","hash":3481672752} {"level":"info","msg":"storing new hash","hash":3481672752,"revision":9994,"compact-revision":9993} {"level":"info","msg":"compact tree index","revision":9995} {"level":"info","msg":"finished scheduled compaction","compact-revision":9995,"took":"38.681113ms","hash":3895620620} {"level":"info","msg":"storing new hash","hash":3895620620,"revision":9995,"compact-revision":9994} {"level":"info","msg":"compact tree index","revision":9996} {"level":"info","msg":"finished scheduled compaction","compact-revision":9996,"took":"38.95543ms","hash":1973132907} {"level":"info","msg":"storing new hash","hash":1973132907,"revision":9996,"compact-revision":9995} {"level":"info","msg":"compact tree index","revision":9997} {"level":"info","msg":"finished scheduled compaction","compact-revision":9997,"took":"39.036076ms","hash":70304945} {"level":"info","msg":"storing new hash","hash":70304945,"revision":9997,"compact-revision":9996} {"level":"info","msg":"compact tree index","revision":9998} {"level":"info","msg":"finished scheduled compaction","compact-revision":9998,"took":"39.463179ms","hash":2008056259} {"level":"info","msg":"storing new hash","hash":2008056259,"revision":9998,"compact-revision":9997} {"level":"info","msg":"compact tree index","revision":9999} {"level":"info","msg":"finished scheduled compaction","compact-revision":9999,"took":"38.159897ms","hash":1342499327} {"level":"info","msg":"storing new hash","hash":1342499327,"revision":9999,"compact-revision":9998} {"level":"info","msg":"compact tree index","revision":10000} {"level":"info","msg":"finished scheduled compaction","compact-revision":10000,"took":"38.984869ms","hash":4044197094} {"level":"info","msg":"storing new hash","hash":4044197094,"revision":10000,"compact-revision":9999} --- PASS: TestHashKVWhenCompacting (9.56s) PASS ```

@ahrtr
Copy link
Member

ahrtr commented Jan 22, 2024

Thanks for the response. Could you also verify whether you can see this issue on v3.5.9 and v3.5.7?

$ git checkout v3.5.9
$ git checkout v3.5.7

@KosovGrigorii
Copy link
Author

on 3.5.9 mvcc passes

ok      go.etcd.io/etcd/server/v3/mvcc  51.829s

yet other test fails(dk if this is relevant, gonna leave logs here anyway)

{"level":"warn","ts":"2024-01-22T14:34:49.368538+0500","caller":"embed/config.go:673","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2024-01-22T14:34:49.368806+0500","caller":"embed/etcd.go:127","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":"2024-01-22T14:34:49.371165+0500","caller":"embed/etcd.go:376","msg":"closing etcd server","name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test431350300","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]}
{"level":"info","ts":"2024-01-22T14:34:49.371226+0500","caller":"embed/etcd.go:378","msg":"closed etcd server","name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test431350300","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]}
{"level":"info","ts":"2024-01-22T14:34:49.371284+0500","caller":"verify/verify.go:57","msg":"verification of persisted state","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test431350300"}
{"level":"panic","ts":"2024-01-22T14:34:49.371406+0500","caller":"backend/backend.go:189","msg":"failed to open database","path":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test431350300/member/snap/db","error":"open /var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test431350300/member/snap/db: no such file or directory","stacktrace":"go.etcd.io/etcd/server/v3/mvcc/backend.newBackend\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189\ngo.etcd.io/etcd/server/v3/mvcc/backend.New\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163\ngo.etcd.io/etcd/server/v3/verify.Verify\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76\ngo.etcd.io/etcd/server/v3/verify.VerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94\ngo.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:379\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:439\ngo.etcd.io/etcd/server/v3/embed.StartEtcd.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:116\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:132\ngo.etcd.io/etcd/server/v3/embed.TestEnableAuth\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34\ntesting.tRunner\n\t/usr/local/opt/go/libexec/src/testing/testing.go:1595"}
{"level":"error","ts":"2024-01-22T14:34:49.371619+0500","caller":"verify/verify.go:64","msg":"verification of persisted state failed","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test431350300","stacktrace":"go.etcd.io/etcd/server/v3/verify.Verify.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:64\nruntime.gopanic\n\t/usr/local/opt/go/libexec/src/runtime/panic.go:920\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write\n\t/Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/zapcore/entry.go:234\ngo.uber.org/zap.(*Logger).Panic\n\t/Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/logger.go:227\ngo.etcd.io/etcd/server/v3/mvcc/backend.newBackend\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189\ngo.etcd.io/etcd/server/v3/mvcc/backend.New\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163\ngo.etcd.io/etcd/server/v3/verify.Verify\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76\ngo.etcd.io/etcd/server/v3/verify.VerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94\ngo.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:379\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:439\ngo.etcd.io/etcd/server/v3/embed.StartEtcd.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:116\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:132\ngo.etcd.io/etcd/server/v3/embed.TestEnableAuth\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34\ntesting.tRunner\n\t/usr/local/opt/go/libexec/src/testing/testing.go:1595"}
--- FAIL: TestEnableAuth (0.02s)
panic: failed to open database [recovered]
        panic: failed to open database [recovered]
        panic: failed to open database

goroutine 151 [running]:
testing.tRunner.func1.2({0x219d1c0, 0xc000650c50})
        /usr/local/opt/go/libexec/src/testing/testing.go:1545 +0x366
testing.tRunner.func1()
        /usr/local/opt/go/libexec/src/testing/testing.go:1548 +0x630
panic({0x219d1c0?, 0xc000650c50?})
        /usr/local/opt/go/libexec/src/runtime/panic.go:920 +0x270
go.etcd.io/etcd/server/v3/verify.Verify.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:66 +0x505
panic({0x219d1c0?, 0xc000650c50?})
        /usr/local/opt/go/libexec/src/runtime/panic.go:920 +0x270
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc000238900, {0xc0002e6100, 0x2, 0x2})
        /Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/zapcore/entry.go:234 +0x61b
go.uber.org/zap.(*Logger).Panic(0xc000198600?, {0x237d6cf, 0x17}, {0xc0002e6100, 0x2, 0x2})
        /Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/logger.go:227 +0x68
go.etcd.io/etcd/server/v3/mvcc/backend.newBackend({{0xc000198600, 0x52}, 0x5f5e100, 0x2710, {0x0, 0x0}, 0x280000000, 0xc0005aca00, 0x0, 0x0, ...})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189 +0x6b3
go.etcd.io/etcd/server/v3/mvcc/backend.New(...)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163
go.etcd.io/etcd/server/v3/verify.Verify({{0xc00004a140?, 0xb?}, 0xa0?, 0xc0005aca00?})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76 +0x478
go.etcd.io/etcd/server/v3/verify.VerifyIfEnabled({{0xc00004a140?, 0xc0005bb140?}, 0x48?, 0xc0005aca00?})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94 +0x76
go.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled({{0xc00004a140?, 0x23762ff?}, 0x12?, 0xc0005aca00?})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103 +0x66
go.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:379 +0x9e
go.etcd.io/etcd/server/v3/embed.(*Etcd).Close(0xc0004d8a00)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:439 +0xcf4
go.etcd.io/etcd/server/v3/embed.StartEtcd.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:116 +0x109
go.etcd.io/etcd/server/v3/embed.StartEtcd(0xc0005d6600)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:132 +0xa2c
go.etcd.io/etcd/server/v3/embed.TestEnableAuth(0xc000590ea0)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34 +0x1d0
testing.tRunner(0xc000590ea0, 0x23d5500)
        /usr/local/opt/go/libexec/src/testing/testing.go:1595 +0x239
created by testing.(*T).Run in goroutine 1
        /usr/local/opt/go/libexec/src/testing/testing.go:1648 +0x82b
FAIL    go.etcd.io/etcd/server/v3/embed 2.468s

The result on 3.5.7 is exactly the same

@ahrtr
Copy link
Member

ahrtr commented Jan 22, 2024

It seems that configurePeerListeners failed, but unfortunately the error log wasn't printed out. Could you run the test on branch https://github.com/etcd-io/etcd/tree/test_3.5.9?

@KosovGrigorii
Copy link
Author

KosovGrigorii commented Jan 22, 2024

do you mean mvcc test or embed?
In test_3.5.9 branch all unit tests run successfully

If you meant TestEnableAuth then here its logs:

``` === RUN TestEnableAuth {"level":"warn","ts":"2024-01-22T17:20:43.771558+0500","caller":"embed/config.go:673","msg":"Running http and grpc server on single port. This is not recommended for production."} {"level":"info","ts":"2024-01-22T17:20:43.771704+0500","caller":"embed/etcd.go:127","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]} {"level":"info","ts":"2024-01-22T17:20:43.773683+0500","caller":"embed/etcd.go:135","msg":"configuring client listeners","listen-client-urls":["http://localhost:2379"]} {"level":"info","ts":"2024-01-22T17:20:43.774113+0500","caller":"embed/etcd.go:309","msg":"starting an etcd server","etcd-version":"3.5.9","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.21.3","go-os":"darwin","go-arch":"amd64","max-cpu-set":12,"max-cpu-available":12,"member-initialized":false,"name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1987866872","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1987866872/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"],"listen-client-urls":["http://localhost:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"default=http://localhost:2380","initial-cluster-state":"new","initial-cluster-token":"etcd-cluster","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"} {"level":"info","ts":"2024-01-22T17:20:43.796911+0500","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1987866872/member/snap/db","took":"22.27332ms"} {"level":"info","ts":"2024-01-22T17:20:43.86723+0500","caller":"etcdserver/raft.go:495","msg":"starting local member","local-member-id":"8e9e05c52164694d","cluster-id":"cdf818194e3a8c32"} {"level":"info","ts":"2024-01-22T17:20:43.867431+0500","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d switched to configuration voters=()"} {"level":"info","ts":"2024-01-22T17:20:43.867518+0500","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became follower at term 0"} {"level":"info","ts":"2024-01-22T17:20:43.867549+0500","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 8e9e05c52164694d [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]"} {"level":"info","ts":"2024-01-22T17:20:43.867574+0500","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became follower at term 1"} {"level":"info","ts":"2024-01-22T17:20:43.86768+0500","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d switched to configuration voters=(10276657743932975437)"} {"level":"warn","ts":"2024-01-22T17:20:43.934394+0500","caller":"auth/store.go:1238","msg":"simple token is not cryptographically signed"} {"level":"info","ts":"2024-01-22T17:20:43.974676+0500","caller":"mvcc/kvstore.go:393","msg":"kvstore restored","current-rev":1} {"level":"info","ts":"2024-01-22T17:20:44.013173+0500","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"} {"level":"info","ts":"2024-01-22T17:20:44.05117+0500","caller":"etcdserver/server.go:854","msg":"starting etcd server","local-member-id":"8e9e05c52164694d","local-server-version":"3.5.9","cluster-version":"to_be_decided"} {"level":"warn","ts":"2024-01-22T17:20:44.051296+0500","caller":"etcdserver/metrics.go:224","msg":"failed to get file descriptor usage","error":"cannot get FDUsage on darwin"} {"level":"info","ts":"2024-01-22T17:20:44.051298+0500","caller":"etcdserver/server.go:738","msg":"started as single-node; fast-forwarding election ticks","local-member-id":"8e9e05c52164694d","forward-ticks":9,"forward-duration":"900ms","election-ticks":10,"election-timeout":"1s"} {"level":"info","ts":"2024-01-22T17:20:44.051389+0500","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1987866872/member/snap","suffix":"snap.db","max":5,"interval":"30s"} {"level":"info","ts":"2024-01-22T17:20:44.051451+0500","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1987866872/member/snap","suffix":"snap","max":5,"interval":"30s"} {"level":"info","ts":"2024-01-22T17:20:44.051469+0500","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1987866872/member/wal","suffix":"wal","max":5,"interval":"30s"} {"level":"info","ts":"2024-01-22T17:20:44.053673+0500","caller":"embed/etcd.go:278","msg":"now serving peer/client/metrics","local-member-id":"8e9e05c52164694d","initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"],"listen-client-urls":["http://localhost:2379"],"listen-metrics-urls":[]} {"level":"info","ts":"2024-01-22T17:20:44.05372+0500","caller":"embed/etcd.go:598","msg":"serving peer traffic","address":"127.0.0.1:2380"} {"level":"info","ts":"2024-01-22T17:20:44.053745+0500","caller":"embed/etcd.go:570","msg":"cmux::serve","address":"127.0.0.1:2380"} {"level":"info","ts":"2024-01-22T17:20:44.070517+0500","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d switched to configuration voters=(10276657743932975437)"} {"level":"info","ts":"2024-01-22T17:20:44.070675+0500","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","added-peer-id":"8e9e05c52164694d","added-peer-peer-urls":["http://localhost:2380"]} {"level":"info","ts":"2024-01-22T17:20:44.669392+0500","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d is starting a new election at term 1"} {"level":"info","ts":"2024-01-22T17:20:44.669483+0500","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became pre-candidate at term 1"} {"level":"info","ts":"2024-01-22T17:20:44.669531+0500","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d received MsgPreVoteResp from 8e9e05c52164694d at term 1"} {"level":"info","ts":"2024-01-22T17:20:44.669611+0500","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became candidate at term 2"} {"level":"info","ts":"2024-01-22T17:20:44.66963+0500","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 2"} {"level":"info","ts":"2024-01-22T17:20:44.66965+0500","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became leader at term 2"} {"level":"info","ts":"2024-01-22T17:20:44.669675+0500","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 2"} {"level":"info","ts":"2024-01-22T17:20:44.690901+0500","caller":"etcdserver/server.go:2571","msg":"setting up initial cluster version using v2 API","cluster-version":"3.5"} {"level":"info","ts":"2024-01-22T17:20:44.690998+0500","caller":"etcdserver/server.go:2062","msg":"published local member to cluster through raft","local-member-id":"8e9e05c52164694d","local-member-attributes":"{Name:default ClientURLs:[http://localhost:2379]}","request-path":"/0/members/8e9e05c52164694d/attributes","cluster-id":"cdf818194e3a8c32","publish-timeout":"7s"} {"level":"info","ts":"2024-01-22T17:20:44.691015+0500","caller":"embed/serve.go:103","msg":"ready to serve client requests"} {"level":"info","ts":"2024-01-22T17:20:44.691772+0500","caller":"embed/serve.go:187","msg":"serving client traffic insecurely; this is strongly discouraged!","traffic":"grpc+http","address":"127.0.0.1:2379"} {"level":"info","ts":"2024-01-22T17:20:44.709374+0500","caller":"auth/store.go:769","msg":"created a role","role-name":"root"} {"level":"info","ts":"2024-01-22T17:20:44.727184+0500","caller":"membership/cluster.go:584","msg":"set initial cluster version","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","cluster-version":"3.5"} {"level":"info","ts":"2024-01-22T17:20:44.727268+0500","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"} {"level":"info","ts":"2024-01-22T17:20:44.727297+0500","caller":"etcdserver/server.go:2595","msg":"cluster version is updated","cluster-version":"3.5"} {"level":"info","ts":"2024-01-22T17:20:44.790475+0500","caller":"auth/store.go:435","msg":"added a user","user-name":"root"} {"level":"info","ts":"2024-01-22T17:20:44.850474+0500","caller":"auth/store.go:548","msg":"granted a role to a user","user-name":"root","user-roles":["root"],"added-role-name":"root"} {"level":"info","ts":"2024-01-22T17:20:44.870692+0500","caller":"auth/store.go:257","msg":"enabled authentication"} {"level":"info","ts":"2024-01-22T17:20:44.909353+0500","caller":"embed/etcd.go:376","msg":"closing etcd server","name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1987866872","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]} {"level":"warn","ts":"2024-01-22T17:20:44.909442+0500","caller":"embed/serve.go:160","msg":"stopping insecure grpc server due to error","error":"accept tcp 127.0.0.1:2379: use of closed network connection"} {"level":"warn","ts":"2024-01-22T17:20:44.909515+0500","caller":"embed/serve.go:162","msg":"stopped insecure grpc server due to error","error":"accept tcp 127.0.0.1:2379: use of closed network connection"} {"level":"info","ts":"2024-01-22T17:20:44.909543+0500","caller":"etcdserver/server.go:1465","msg":"skipped leadership transfer for single voting member cluster","local-member-id":"8e9e05c52164694d","current-leader-member-id":"8e9e05c52164694d"} {"level":"info","ts":"2024-01-22T17:20:44.979785+0500","caller":"embed/etcd.go:580","msg":"stopping serving peer traffic","address":"127.0.0.1:2380"} {"level":"info","ts":"2024-01-22T17:20:44.979874+0500","caller":"embed/etcd.go:585","msg":"stopped serving peer traffic","address":"127.0.0.1:2380"} {"level":"info","ts":"2024-01-22T17:20:44.979895+0500","caller":"embed/etcd.go:378","msg":"closed etcd server","name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1987866872","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]} --- PASS: TestEnableAuth (1.21s) PASS ```

If you meant TestHashKVWhenCompacting:

``` === RUN TestHashKVWhenCompacting {"level":"info","msg":"kvstore restored","current-rev":1} {"level":"info","msg":"compact tree index","revision":9899} {"level":"info","msg":"finished scheduled compaction","compact-revision":9899,"took":"767.404567ms","hash":2463323419} {"level":"info","msg":"storing new hash","hash":2463323419,"revision":9899,"compact-revision":-1} {"level":"info","msg":"compact tree index","revision":9900} {"level":"info","msg":"finished scheduled compaction","compact-revision":9900,"took":"552.393µs","hash":2187321635} {"level":"info","msg":"storing new hash","hash":2187321635,"revision":9900,"compact-revision":9899} {"level":"info","msg":"compact tree index","revision":9901} {"level":"info","msg":"finished scheduled compaction","compact-revision":9901,"took":"39.626187ms","hash":4092902393} {"level":"info","msg":"storing new hash","hash":4092902393,"revision":9901,"compact-revision":9900} {"level":"info","msg":"compact tree index","revision":9902} {"level":"info","msg":"finished scheduled compaction","compact-revision":9902,"took":"22.322µs","hash":3442946395} {"level":"info","msg":"storing new hash","hash":3442946395,"revision":9902,"compact-revision":9901} {"level":"info","msg":"compact tree index","revision":9903} {"level":"info","msg":"finished scheduled compaction","compact-revision":9903,"took":"39.049402ms","hash":3934476647} {"level":"info","msg":"storing new hash","hash":3934476647,"revision":9903,"compact-revision":9902} {"level":"info","msg":"compact tree index","revision":9904} {"level":"info","msg":"finished scheduled compaction","compact-revision":9904,"took":"323.328µs","hash":1979317650} {"level":"info","msg":"storing new hash","hash":1979317650,"revision":9904,"compact-revision":9903} {"level":"info","msg":"compact tree index","revision":9905} {"level":"info","msg":"finished scheduled compaction","compact-revision":9905,"took":"252.654µs","hash":4126965741} {"level":"info","msg":"storing new hash","hash":4126965741,"revision":9905,"compact-revision":9904} {"level":"info","msg":"compact tree index","revision":9906} {"level":"info","msg":"finished scheduled compaction","compact-revision":9906,"took":"208.412µs","hash":3409966415} {"level":"info","msg":"storing new hash","hash":3409966415,"revision":9906,"compact-revision":9905} {"level":"info","msg":"compact tree index","revision":9907} {"level":"info","msg":"finished scheduled compaction","compact-revision":9907,"took":"195.204µs","hash":3968572787} {"level":"info","msg":"storing new hash","hash":3968572787,"revision":9907,"compact-revision":9906} {"level":"info","msg":"compact tree index","revision":9908} {"level":"info","msg":"finished scheduled compaction","compact-revision":9908,"took":"179.669µs","hash":1013037252} {"level":"info","msg":"storing new hash","hash":1013037252,"revision":9908,"compact-revision":9907} {"level":"info","msg":"compact tree index","revision":9909} {"level":"info","msg":"finished scheduled compaction","compact-revision":9909,"took":"177.709µs","hash":1305141790} {"level":"info","msg":"storing new hash","hash":1305141790,"revision":9909,"compact-revision":9908} {"level":"info","msg":"compact tree index","revision":9910} {"level":"info","msg":"finished scheduled compaction","compact-revision":9910,"took":"36.682µs","hash":189389043} {"level":"info","msg":"storing new hash","hash":189389043,"revision":9910,"compact-revision":9909} {"level":"info","msg":"compact tree index","revision":9911} {"level":"info","msg":"finished scheduled compaction","compact-revision":9911,"took":"24.595µs","hash":754823375} {"level":"info","msg":"storing new hash","hash":754823375,"revision":9911,"compact-revision":9910} {"level":"info","msg":"compact tree index","revision":9912} {"level":"info","msg":"finished scheduled compaction","compact-revision":9912,"took":"25.864µs","hash":4067225043} {"level":"info","msg":"storing new hash","hash":4067225043,"revision":9912,"compact-revision":9911} {"level":"info","msg":"compact tree index","revision":9913} {"level":"info","msg":"finished scheduled compaction","compact-revision":9913,"took":"24.663µs","hash":804901573} {"level":"info","msg":"storing new hash","hash":804901573,"revision":9913,"compact-revision":9912} {"level":"info","msg":"compact tree index","revision":9914} {"level":"info","msg":"finished scheduled compaction","compact-revision":9914,"took":"29.164µs","hash":3600863096} {"level":"info","msg":"storing new hash","hash":3600863096,"revision":9914,"compact-revision":9913} {"level":"info","msg":"compact tree index","revision":9915} {"level":"info","msg":"finished scheduled compaction","compact-revision":9915,"took":"42.504µs","hash":4044662596} {"level":"info","msg":"storing new hash","hash":4044662596,"revision":9915,"compact-revision":9914} {"level":"info","msg":"compact tree index","revision":9916} {"level":"info","msg":"finished scheduled compaction","compact-revision":9916,"took":"69.147µs","hash":1824345379} {"level":"info","msg":"storing new hash","hash":1824345379,"revision":9916,"compact-revision":9915} {"level":"info","msg":"compact tree index","revision":9917} {"level":"info","msg":"finished scheduled compaction","compact-revision":9917,"took":"28.639µs","hash":487987193} {"level":"info","msg":"storing new hash","hash":487987193,"revision":9917,"compact-revision":9916} {"level":"info","msg":"compact tree index","revision":9918} {"level":"info","msg":"finished scheduled compaction","compact-revision":9918,"took":"26.397µs","hash":1855345291} {"level":"info","msg":"storing new hash","hash":1855345291,"revision":9918,"compact-revision":9917} {"level":"info","msg":"compact tree index","revision":9919} {"level":"info","msg":"finished scheduled compaction","compact-revision":9919,"took":"146.887µs","hash":1226980023} {"level":"info","msg":"storing new hash","hash":1226980023,"revision":9919,"compact-revision":9918} {"level":"info","msg":"compact tree index","revision":9920} {"level":"info","msg":"finished scheduled compaction","compact-revision":9920,"took":"35.512µs","hash":1439986907} {"level":"info","msg":"storing new hash","hash":1439986907,"revision":9920,"compact-revision":9919} {"level":"info","msg":"compact tree index","revision":9921} {"level":"info","msg":"finished scheduled compaction","compact-revision":9921,"took":"42.014µs","hash":492703523} {"level":"info","msg":"storing new hash","hash":492703523,"revision":9921,"compact-revision":9920} {"level":"info","msg":"compact tree index","revision":9922} {"level":"info","msg":"finished scheduled compaction","compact-revision":9922,"took":"46.973µs","hash":597525889} {"level":"info","msg":"storing new hash","hash":597525889,"revision":9922,"compact-revision":9921} {"level":"info","msg":"compact tree index","revision":9923} {"level":"info","msg":"finished scheduled compaction","compact-revision":9923,"took":"22.493µs","hash":69856701} {"level":"info","msg":"storing new hash","hash":69856701,"revision":9923,"compact-revision":9922} {"level":"info","msg":"compact tree index","revision":9924} {"level":"info","msg":"finished scheduled compaction","compact-revision":9924,"took":"25.666µs","hash":1578799301} {"level":"info","msg":"storing new hash","hash":1578799301,"revision":9924,"compact-revision":9923} {"level":"info","msg":"compact tree index","revision":9925} {"level":"info","msg":"finished scheduled compaction","compact-revision":9925,"took":"22.856µs","hash":800180767} {"level":"info","msg":"storing new hash","hash":800180767,"revision":9925,"compact-revision":9924} {"level":"info","msg":"compact tree index","revision":9926} {"level":"info","msg":"finished scheduled compaction","compact-revision":9926,"took":"65.212µs","hash":292700349} {"level":"info","msg":"storing new hash","hash":292700349,"revision":9926,"compact-revision":9925} {"level":"info","msg":"compact tree index","revision":9927} {"level":"info","msg":"finished scheduled compaction","compact-revision":9927,"took":"76.464µs","hash":918956161} {"level":"info","msg":"storing new hash","hash":918956161,"revision":9927,"compact-revision":9926} {"level":"info","msg":"compact tree index","revision":9928} {"level":"info","msg":"finished scheduled compaction","compact-revision":9928,"took":"72.386µs","hash":2769599053} {"level":"info","msg":"storing new hash","hash":2769599053,"revision":9928,"compact-revision":9927} {"level":"info","msg":"compact tree index","revision":9929} {"level":"info","msg":"finished scheduled compaction","compact-revision":9929,"took":"23.596µs","hash":2021754203} {"level":"info","msg":"storing new hash","hash":2021754203,"revision":9929,"compact-revision":9928} {"level":"info","msg":"compact tree index","revision":9930} {"level":"info","msg":"finished scheduled compaction","compact-revision":9930,"took":"75.085µs","hash":1040367542} {"level":"info","msg":"storing new hash","hash":1040367542,"revision":9930,"compact-revision":9929} {"level":"info","msg":"compact tree index","revision":9931} {"level":"info","msg":"finished scheduled compaction","compact-revision":9931,"took":"26.373µs","hash":431413130} {"level":"info","msg":"storing new hash","hash":431413130,"revision":9931,"compact-revision":9930} {"level":"info","msg":"compact tree index","revision":9932} {"level":"info","msg":"finished scheduled compaction","compact-revision":9932,"took":"30.062µs","hash":2216639981} {"level":"info","msg":"storing new hash","hash":2216639981,"revision":9932,"compact-revision":9931} {"level":"info","msg":"compact tree index","revision":9933} {"level":"info","msg":"finished scheduled compaction","compact-revision":9933,"took":"65.148µs","hash":4122240823} {"level":"info","msg":"storing new hash","hash":4122240823,"revision":9933,"compact-revision":9932} {"level":"info","msg":"compact tree index","revision":9934} {"level":"info","msg":"finished scheduled compaction","compact-revision":9934,"took":"37.8µs","hash":216889994} {"level":"info","msg":"storing new hash","hash":216889994,"revision":9934,"compact-revision":9933} {"level":"info","msg":"compact tree index","revision":9935} {"level":"info","msg":"finished scheduled compaction","compact-revision":9935,"took":"29.307µs","hash":727257782} {"level":"info","msg":"storing new hash","hash":727257782,"revision":9935,"compact-revision":9934} {"level":"info","msg":"compact tree index","revision":9936} {"level":"info","msg":"finished scheduled compaction","compact-revision":9936,"took":"26.764µs","hash":1941512540} {"level":"info","msg":"storing new hash","hash":1941512540,"revision":9936,"compact-revision":9935} {"level":"info","msg":"compact tree index","revision":9937} {"level":"info","msg":"finished scheduled compaction","compact-revision":9937,"took":"27.903µs","hash":4089226019} {"level":"info","msg":"storing new hash","hash":4089226019,"revision":9937,"compact-revision":9936} {"level":"info","msg":"compact tree index","revision":9938} {"level":"info","msg":"finished scheduled compaction","compact-revision":9938,"took":"32.557µs","hash":2151474769} {"level":"info","msg":"storing new hash","hash":2151474769,"revision":9938,"compact-revision":9937} {"level":"info","msg":"compact tree index","revision":9939} {"level":"info","msg":"finished scheduled compaction","compact-revision":9939,"took":"48.246µs","hash":2810744429} {"level":"info","msg":"storing new hash","hash":2810744429,"revision":9939,"compact-revision":9938} {"level":"info","msg":"compact tree index","revision":9940} {"level":"info","msg":"finished scheduled compaction","compact-revision":9940,"took":"26.572µs","hash":732857719} {"level":"info","msg":"storing new hash","hash":732857719,"revision":9940,"compact-revision":9939} {"level":"info","msg":"compact tree index","revision":9941} {"level":"info","msg":"finished scheduled compaction","compact-revision":9941,"took":"19.579µs","hash":1510334381} {"level":"info","msg":"storing new hash","hash":1510334381,"revision":9941,"compact-revision":9940} {"level":"info","msg":"compact tree index","revision":9942} {"level":"info","msg":"finished scheduled compaction","compact-revision":9942,"took":"19.391µs","hash":1690719503} {"level":"info","msg":"storing new hash","hash":1690719503,"revision":9942,"compact-revision":9941} {"level":"info","msg":"compact tree index","revision":9943} {"level":"info","msg":"finished scheduled compaction","compact-revision":9943,"took":"27.395µs","hash":1131560243} {"level":"info","msg":"storing new hash","hash":1131560243,"revision":9943,"compact-revision":9942} {"level":"info","msg":"compact tree index","revision":9944} {"level":"info","msg":"finished scheduled compaction","compact-revision":9944,"took":"47.095µs","hash":1517942576} {"level":"info","msg":"storing new hash","hash":1517942576,"revision":9944,"compact-revision":9943} {"level":"info","msg":"compact tree index","revision":9945} {"level":"info","msg":"finished scheduled compaction","compact-revision":9945,"took":"40.192µs","hash":2280441894} {"level":"info","msg":"storing new hash","hash":2280441894,"revision":9945,"compact-revision":9944} {"level":"info","msg":"compact tree index","revision":9946} {"level":"info","msg":"finished scheduled compaction","compact-revision":9946,"took":"41.015µs","hash":3106882180} {"level":"info","msg":"storing new hash","hash":3106882180,"revision":9946,"compact-revision":9945} {"level":"info","msg":"compact tree index","revision":9947} {"level":"info","msg":"finished scheduled compaction","compact-revision":9947,"took":"26.637µs","hash":2660969144} {"level":"info","msg":"storing new hash","hash":2660969144,"revision":9947,"compact-revision":9946} {"level":"info","msg":"compact tree index","revision":9948} {"level":"info","msg":"finished scheduled compaction","compact-revision":9948,"took":"31.553µs","hash":1316032271} {"level":"info","msg":"storing new hash","hash":1316032271,"revision":9948,"compact-revision":9947} {"level":"info","msg":"compact tree index","revision":9949} {"level":"info","msg":"finished scheduled compaction","compact-revision":9949,"took":"19.857µs","hash":1071273429} {"level":"info","msg":"storing new hash","hash":1071273429,"revision":9949,"compact-revision":9948} {"level":"info","msg":"compact tree index","revision":9950} {"level":"info","msg":"finished scheduled compaction","compact-revision":9950,"took":"24.496µs","hash":2035904312} {"level":"info","msg":"storing new hash","hash":2035904312,"revision":9950,"compact-revision":9949} {"level":"info","msg":"compact tree index","revision":9951} {"level":"info","msg":"finished scheduled compaction","compact-revision":9951,"took":"28.578µs","hash":1592592132} {"level":"info","msg":"storing new hash","hash":1592592132,"revision":9951,"compact-revision":9950} {"level":"info","msg":"compact tree index","revision":9952} {"level":"info","msg":"finished scheduled compaction","compact-revision":9952,"took":"30.78µs","hash":251895795} {"level":"info","msg":"storing new hash","hash":251895795,"revision":9952,"compact-revision":9951} {"level":"info","msg":"compact tree index","revision":9953} {"level":"info","msg":"finished scheduled compaction","compact-revision":9953,"took":"235.466µs","hash":874997598} {"level":"info","msg":"storing new hash","hash":874997598,"revision":9953,"compact-revision":9952} {"level":"info","msg":"compact tree index","revision":9954} {"level":"info","msg":"finished scheduled compaction","compact-revision":9954,"took":"18.607µs","hash":3447618275} {"level":"info","msg":"storing new hash","hash":3447618275,"revision":9954,"compact-revision":9953} {"level":"info","msg":"compact tree index","revision":9955} {"level":"info","msg":"finished scheduled compaction","compact-revision":9955,"took":"19.46µs","hash":3939111647} {"level":"info","msg":"storing new hash","hash":3939111647,"revision":9955,"compact-revision":9954} {"level":"info","msg":"compact tree index","revision":9956} {"level":"info","msg":"finished scheduled compaction","compact-revision":9956,"took":"20.202µs","hash":2003040440} {"level":"info","msg":"storing new hash","hash":2003040440,"revision":9956,"compact-revision":9955} {"level":"info","msg":"compact tree index","revision":9957} {"level":"info","msg":"finished scheduled compaction","compact-revision":9957,"took":"19.222µs","hash":113812066} {"level":"info","msg":"storing new hash","hash":113812066,"revision":9957,"compact-revision":9956} {"level":"info","msg":"compact tree index","revision":9958} {"level":"info","msg":"finished scheduled compaction","compact-revision":9958,"took":"40.753µs","hash":1967712016} {"level":"info","msg":"storing new hash","hash":1967712016,"revision":9958,"compact-revision":9957} {"level":"info","msg":"compact tree index","revision":9959} {"level":"info","msg":"finished scheduled compaction","compact-revision":9959,"took":"72.169µs","hash":1392283436} {"level":"info","msg":"storing new hash","hash":1392283436,"revision":9959,"compact-revision":9958} {"level":"info","msg":"compact tree index","revision":9960} {"level":"info","msg":"finished scheduled compaction","compact-revision":9960,"took":"29.04µs","hash":3546829457} {"level":"info","msg":"storing new hash","hash":3546829457,"revision":9960,"compact-revision":9959} {"level":"info","msg":"compact tree index","revision":9961} {"level":"info","msg":"finished scheduled compaction","compact-revision":9961,"took":"32.254µs","hash":251489671} {"level":"info","msg":"storing new hash","hash":251489671,"revision":9961,"compact-revision":9960} {"level":"info","msg":"compact tree index","revision":9962} {"level":"info","msg":"finished scheduled compaction","compact-revision":9962,"took":"18.655µs","hash":809428773} {"level":"info","msg":"storing new hash","hash":809428773,"revision":9962,"compact-revision":9961} {"level":"info","msg":"compact tree index","revision":9963} {"level":"info","msg":"finished scheduled compaction","compact-revision":9963,"took":"20.374µs","hash":394956569} {"level":"info","msg":"storing new hash","hash":394956569,"revision":9963,"compact-revision":9962} {"level":"info","msg":"compact tree index","revision":9964} {"level":"info","msg":"finished scheduled compaction","compact-revision":9964,"took":"41.621µs","hash":1304031841} {"level":"info","msg":"storing new hash","hash":1304031841,"revision":9964,"compact-revision":9963} {"level":"info","msg":"compact tree index","revision":9965} {"level":"info","msg":"finished scheduled compaction","compact-revision":9965,"took":"25.378µs","hash":1007855803} {"level":"info","msg":"storing new hash","hash":1007855803,"revision":9965,"compact-revision":9964} {"level":"info","msg":"compact tree index","revision":9966} {"level":"info","msg":"finished scheduled compaction","compact-revision":9966,"took":"44.72µs","hash":47260185} {"level":"info","msg":"storing new hash","hash":47260185,"revision":9966,"compact-revision":9965} {"level":"info","msg":"compact tree index","revision":9967} {"level":"info","msg":"finished scheduled compaction","compact-revision":9967,"took":"76.281µs","hash":627395109} {"level":"info","msg":"storing new hash","hash":627395109,"revision":9967,"compact-revision":9966} {"level":"info","msg":"compact tree index","revision":9968} {"level":"info","msg":"finished scheduled compaction","compact-revision":9968,"took":"28.019µs","hash":818329119} {"level":"info","msg":"storing new hash","hash":818329119,"revision":9968,"compact-revision":9967} {"level":"info","msg":"compact tree index","revision":9969} {"level":"info","msg":"finished scheduled compaction","compact-revision":9969,"took":"46.605µs","hash":2965582944} {"level":"info","msg":"storing new hash","hash":2965582944,"revision":9969,"compact-revision":9968} {"level":"info","msg":"compact tree index","revision":9970} {"level":"info","msg":"finished scheduled compaction","compact-revision":9970,"took":"30.333µs","hash":4131393165} {"level":"info","msg":"storing new hash","hash":4131393165,"revision":9970,"compact-revision":9969} {"level":"info","msg":"compact tree index","revision":9971} {"level":"info","msg":"finished scheduled compaction","compact-revision":9971,"took":"28.535µs","hash":3522455217} {"level":"info","msg":"storing new hash","hash":3522455217,"revision":9971,"compact-revision":9970} {"level":"info","msg":"compact tree index","revision":9972} {"level":"info","msg":"finished scheduled compaction","compact-revision":9972,"took":"28.588µs","hash":1281199318} {"level":"info","msg":"storing new hash","hash":1281199318,"revision":9972,"compact-revision":9971} {"level":"info","msg":"compact tree index","revision":9973} {"level":"info","msg":"finished scheduled compaction","compact-revision":9973,"took":"23.901µs","hash":1039586828} {"level":"info","msg":"storing new hash","hash":1039586828,"revision":9973,"compact-revision":9972} {"level":"info","msg":"compact tree index","revision":9974} {"level":"info","msg":"finished scheduled compaction","compact-revision":9974,"took":"20.51µs","hash":3299855281} {"level":"info","msg":"storing new hash","hash":3299855281,"revision":9974,"compact-revision":9973} {"level":"info","msg":"compact tree index","revision":9975} {"level":"info","msg":"finished scheduled compaction","compact-revision":9975,"took":"70.638µs","hash":3810239373} {"level":"info","msg":"storing new hash","hash":3810239373,"revision":9975,"compact-revision":9974} {"level":"info","msg":"compact tree index","revision":9976} {"level":"info","msg":"finished scheduled compaction","compact-revision":9976,"took":"56.244µs","hash":1032519313} {"level":"info","msg":"storing new hash","hash":1032519313,"revision":9976,"compact-revision":9975} {"level":"info","msg":"compact tree index","revision":9977} {"level":"info","msg":"finished scheduled compaction","compact-revision":9977,"took":"41.898µs","hash":3760177543} {"level":"info","msg":"storing new hash","hash":3760177543,"revision":9977,"compact-revision":9976} {"level":"info","msg":"compact tree index","revision":9978} {"level":"info","msg":"finished scheduled compaction","compact-revision":9978,"took":"25.664µs","hash":2476705013} {"level":"info","msg":"storing new hash","hash":2476705013,"revision":9978,"compact-revision":9977} {"level":"info","msg":"compact tree index","revision":9979} {"level":"info","msg":"finished scheduled compaction","compact-revision":9979,"took":"41.915µs","hash":3022777545} {"level":"info","msg":"storing new hash","hash":3022777545,"revision":9979,"compact-revision":9978} {"level":"info","msg":"compact tree index","revision":9980} {"level":"info","msg":"finished scheduled compaction","compact-revision":9980,"took":"36.512µs","hash":2769888363} {"level":"info","msg":"storing new hash","hash":2769888363,"revision":9980,"compact-revision":9979} {"level":"info","msg":"compact tree index","revision":9981} {"level":"info","msg":"finished scheduled compaction","compact-revision":9981,"took":"22.605µs","hash":3568467633} {"level":"info","msg":"storing new hash","hash":3568467633,"revision":9981,"compact-revision":9980} {"level":"info","msg":"compact tree index","revision":9982} {"level":"info","msg":"finished scheduled compaction","compact-revision":9982,"took":"25.066µs","hash":3933336595} {"level":"info","msg":"storing new hash","hash":3933336595,"revision":9982,"compact-revision":9981} {"level":"info","msg":"compact tree index","revision":9983} {"level":"info","msg":"finished scheduled compaction","compact-revision":9983,"took":"55.061µs","hash":3452279855} {"level":"info","msg":"storing new hash","hash":3452279855,"revision":9983,"compact-revision":9982} {"level":"info","msg":"compact tree index","revision":9984} {"level":"info","msg":"finished scheduled compaction","compact-revision":9984,"took":"49.356µs","hash":4232457539} {"level":"info","msg":"storing new hash","hash":4232457539,"revision":9984,"compact-revision":9983} {"level":"info","msg":"compact tree index","revision":9985} {"level":"info","msg":"finished scheduled compaction","compact-revision":9985,"took":"22.695µs","hash":3973754021} {"level":"info","msg":"storing new hash","hash":3973754021,"revision":9985,"compact-revision":9984} {"level":"info","msg":"compact tree index","revision":9986} {"level":"info","msg":"finished scheduled compaction","compact-revision":9986,"took":"46.631µs","hash":3524863495} {"level":"info","msg":"storing new hash","hash":3524863495,"revision":9986,"compact-revision":9985} {"level":"info","msg":"compact tree index","revision":9987} {"level":"info","msg":"finished scheduled compaction","compact-revision":9987,"took":"29.964µs","hash":4121775675} {"level":"info","msg":"storing new hash","hash":4121775675,"revision":9987,"compact-revision":9986} {"level":"info","msg":"compact tree index","revision":9988} {"level":"info","msg":"finished scheduled compaction","compact-revision":9988,"took":"34.301µs","hash":625437580} {"level":"info","msg":"storing new hash","hash":625437580,"revision":9988,"compact-revision":9987} {"level":"info","msg":"compact tree index","revision":9989} {"level":"info","msg":"finished scheduled compaction","compact-revision":9989,"took":"21.713µs","hash":1424765270} {"level":"info","msg":"storing new hash","hash":1424765270,"revision":9989,"compact-revision":9988} {"level":"info","msg":"compact tree index","revision":9990} {"level":"info","msg":"finished scheduled compaction","compact-revision":9990,"took":"22.821µs","hash":309267387} {"level":"info","msg":"storing new hash","hash":309267387,"revision":9990,"compact-revision":9989} {"level":"info","msg":"compact tree index","revision":9991} {"level":"info","msg":"finished scheduled compaction","compact-revision":9991,"took":"32.931µs","hash":903570311} {"level":"info","msg":"storing new hash","hash":903570311,"revision":9991,"compact-revision":9990} {"level":"info","msg":"compact tree index","revision":9992} {"level":"info","msg":"finished scheduled compaction","compact-revision":9992,"took":"17.13µs","hash":3947511451} {"level":"info","msg":"storing new hash","hash":3947511451,"revision":9992,"compact-revision":9991} {"level":"info","msg":"compact tree index","revision":9993} {"level":"info","msg":"finished scheduled compaction","compact-revision":9993,"took":"23.487µs","hash":920584589} {"level":"info","msg":"storing new hash","hash":920584589,"revision":9993,"compact-revision":9992} {"level":"info","msg":"compact tree index","revision":9994} {"level":"info","msg":"finished scheduled compaction","compact-revision":9994,"took":"27.991µs","hash":3481672752} {"level":"info","msg":"storing new hash","hash":3481672752,"revision":9994,"compact-revision":9993} {"level":"info","msg":"compact tree index","revision":9995} {"level":"info","msg":"finished scheduled compaction","compact-revision":9995,"took":"20.165µs","hash":3895620620} {"level":"info","msg":"storing new hash","hash":3895620620,"revision":9995,"compact-revision":9994} {"level":"info","msg":"compact tree index","revision":9996} {"level":"info","msg":"finished scheduled compaction","compact-revision":9996,"took":"38.049µs","hash":1973132907} {"level":"info","msg":"storing new hash","hash":1973132907,"revision":9996,"compact-revision":9995} {"level":"info","msg":"compact tree index","revision":9997} {"level":"info","msg":"finished scheduled compaction","compact-revision":9997,"took":"20.264µs","hash":70304945} {"level":"info","msg":"storing new hash","hash":70304945,"revision":9997,"compact-revision":9996} {"level":"info","msg":"compact tree index","revision":9998} {"level":"info","msg":"finished scheduled compaction","compact-revision":9998,"took":"22.684µs","hash":2008056259} {"level":"info","msg":"storing new hash","hash":2008056259,"revision":9998,"compact-revision":9997} {"level":"info","msg":"compact tree index","revision":9999} {"level":"info","msg":"finished scheduled compaction","compact-revision":9999,"took":"30.598µs","hash":1342499327} {"level":"info","msg":"storing new hash","hash":1342499327,"revision":9999,"compact-revision":9998} --- PASS: TestHashKVWhenCompacting (5.99s) PASS ```

@ahrtr
Copy link
Member

ahrtr commented Jan 22, 2024

do you mean mvcc test or embed?

I mean go clean --testcache; make test-unit

@KosovGrigorii
Copy link
Author

KosovGrigorii commented Jan 22, 2024

All tests passed on test_3.5.9 branch

``` TEST_OPTS: PASSES='unit' log-file: test-MTcwNTkyNjIyNgo.log PASSES='unit' ./test.sh 2>&1 | tee test-MTcwNTkyNjIyNgo.log Running with --race Starting at: Mon Jan 22 17:23:47 +05 2024

'unit' started at Mon Jan 22 17:23:47 +05 2024
% (cd api && 'env' 'go' 'test' '-short' '-timeout=3m' '--race' './...')
? go.etcd.io/etcd/api/v3/authpb [no test files]
? go.etcd.io/etcd/api/v3/etcdserverpb/gw [no test files]
? go.etcd.io/etcd/api/v3/membershippb [no test files]
? go.etcd.io/etcd/api/v3/mvccpb [no test files]
? go.etcd.io/etcd/api/v3/version [no test files]
ok go.etcd.io/etcd/api/v3/etcdserverpb 1.760s
ok go.etcd.io/etcd/api/v3/v3rpc/rpctypes 1.380s
% (cd pkg && 'env' 'go' 'test' '-short' '-timeout=3m' '--race' './...')
? go.etcd.io/etcd/pkg/v3/cobrautl [no test files]
? go.etcd.io/etcd/pkg/v3/contention [no test files]
? go.etcd.io/etcd/pkg/v3/cpuutil [no test files]
? go.etcd.io/etcd/pkg/v3/debugutil [no test files]
? go.etcd.io/etcd/pkg/v3/grpc_testing [no test files]
ok go.etcd.io/etcd/pkg/v3/adt 1.543s
? go.etcd.io/etcd/pkg/v3/runtime [no test files]
ok go.etcd.io/etcd/pkg/v3/crc 1.603s
ok go.etcd.io/etcd/pkg/v3/expect 2.270s
ok go.etcd.io/etcd/pkg/v3/flags 1.900s
ok go.etcd.io/etcd/pkg/v3/httputil 2.107s
ok go.etcd.io/etcd/pkg/v3/idutil 2.289s
ok go.etcd.io/etcd/pkg/v3/ioutil 2.476s
ok go.etcd.io/etcd/pkg/v3/netutil 3.694s
ok go.etcd.io/etcd/pkg/v3/osutil 3.099s
ok go.etcd.io/etcd/pkg/v3/pbutil 2.867s
ok go.etcd.io/etcd/pkg/v3/proxy 7.555s
ok go.etcd.io/etcd/pkg/v3/report 3.089s
ok go.etcd.io/etcd/pkg/v3/schedule 2.404s
ok go.etcd.io/etcd/pkg/v3/stringutil 2.514s
ok go.etcd.io/etcd/pkg/v3/traceutil 2.112s
ok go.etcd.io/etcd/pkg/v3/wait 2.300s
% (cd raft && 'env' 'go' 'test' '-short' '-timeout=3m' '--race' './...')
ok go.etcd.io/etcd/raft/v3 3.375s
ok go.etcd.io/etcd/raft/v3/confchange 2.597s
ok go.etcd.io/etcd/raft/v3/quorum 2.702s
ok go.etcd.io/etcd/raft/v3/raftpb 1.314s
ok go.etcd.io/etcd/raft/v3/rafttest 2.186s
ok go.etcd.io/etcd/raft/v3/tracker 1.868s
% (cd client/pkg && 'env' 'go' 'test' '-short' '-timeout=3m' '--race' './...')
? go.etcd.io/etcd/client/pkg/v3/systemd [no test files]
ok go.etcd.io/etcd/client/pkg/v3/fileutil 2.411s
ok go.etcd.io/etcd/client/pkg/v3/logutil 1.290s
ok go.etcd.io/etcd/client/pkg/v3/pathutil 1.458s
ok go.etcd.io/etcd/client/pkg/v3/srv 2.066s
ok go.etcd.io/etcd/client/pkg/v3/testutil 1.885s
ok go.etcd.io/etcd/client/pkg/v3/tlsutil 1.646s
ok go.etcd.io/etcd/client/pkg/v3/transport 3.692s
ok go.etcd.io/etcd/client/pkg/v3/types 2.346s
% (cd client/v2 && 'env' 'go' 'test' '-short' '-timeout=3m' '--race' './...')
ok go.etcd.io/etcd/client/v2 1.392s
% (cd client/v3 && 'env' 'go' 'test' '-short' '-timeout=3m' '--race' './...')
? go.etcd.io/etcd/client/v3/credentials [no test files]
? go.etcd.io/etcd/client/v3/experimental/recipes [no test files]
? go.etcd.io/etcd/client/v3/internal/resolver [no test files]
? go.etcd.io/etcd/client/v3/leasing [no test files]
? go.etcd.io/etcd/client/v3/mirror [no test files]
? go.etcd.io/etcd/client/v3/mock/mockserver [no test files]
? go.etcd.io/etcd/client/v3/naming [no test files]
? go.etcd.io/etcd/client/v3/naming/endpoints [no test files]
? go.etcd.io/etcd/client/v3/naming/endpoints/internal [no test files]
? go.etcd.io/etcd/client/v3/naming/resolver [no test files]
? go.etcd.io/etcd/client/v3/snapshot [no test files]
ok go.etcd.io/etcd/client/v3 5.758s
ok go.etcd.io/etcd/client/v3/clientv3util 2.042s [no tests to run]
ok go.etcd.io/etcd/client/v3/concurrency 1.631s
ok go.etcd.io/etcd/client/v3/internal/endpoint 2.746s
ok go.etcd.io/etcd/client/v3/namespace 2.559s
ok go.etcd.io/etcd/client/v3/ordering 2.957s
ok go.etcd.io/etcd/client/v3/yaml 3.355s
% (cd server && 'env' 'go' 'test' '-short' '-timeout=3m' '--race' './...')
? go.etcd.io/etcd/server/v3/etcdserver/api [no test files]
? go.etcd.io/etcd/server/v3/etcdserver/api/snap/snappb [no test files]
ok go.etcd.io/etcd/server/v3 1.930s
? go.etcd.io/etcd/server/v3/etcdserver/api/v2stats [no test files]
? go.etcd.io/etcd/server/v3/etcdserver/api/v2v3 [no test files]
? go.etcd.io/etcd/server/v3/etcdserver/api/v3alarm [no test files]
? go.etcd.io/etcd/server/v3/etcdserver/api/v3client [no test files]
? go.etcd.io/etcd/server/v3/etcdserver/api/v3election [no test files]
? go.etcd.io/etcd/server/v3/etcdserver/api/v3election/v3electionpb [no test files]
? go.etcd.io/etcd/server/v3/etcdserver/api/v3election/v3electionpb/gw [no test files]
? go.etcd.io/etcd/server/v3/etcdserver/api/v3lock [no test files]
? go.etcd.io/etcd/server/v3/etcdserver/api/v3lock/v3lockpb [no test files]
? go.etcd.io/etcd/server/v3/etcdserver/api/v3lock/v3lockpb/gw [no test files]
? go.etcd.io/etcd/server/v3/lease/leasepb [no test files]
? go.etcd.io/etcd/server/v3/mock/mockstore [no test files]
? go.etcd.io/etcd/server/v3/mock/mockstorage [no test files]
? go.etcd.io/etcd/server/v3/mock/mockwait [no test files]
? go.etcd.io/etcd/server/v3/mvcc/backend/testing [no test files]
? go.etcd.io/etcd/server/v3/mvcc/buckets [no test files]
? go.etcd.io/etcd/server/v3/proxy/grpcproxy [no test files]
? go.etcd.io/etcd/server/v3/proxy/grpcproxy/adapter [no test files]
? go.etcd.io/etcd/server/v3/proxy/grpcproxy/cache [no test files]
? go.etcd.io/etcd/server/v3/storage/mvcc/testutil [no test files]
? go.etcd.io/etcd/server/v3/verify [no test files]
ok go.etcd.io/etcd/server/v3/auth 28.648s
ok go.etcd.io/etcd/server/v3/config 2.230s
ok go.etcd.io/etcd/server/v3/datadir 2.403s
ok go.etcd.io/etcd/server/v3/embed 8.290s
ok go.etcd.io/etcd/server/v3/etcdmain 6.638s
ok go.etcd.io/etcd/server/v3/etcdserver 33.517s
ok go.etcd.io/etcd/server/v3/etcdserver/api/etcdhttp 2.913s
ok go.etcd.io/etcd/server/v3/etcdserver/api/membership 16.756s
ok go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp 3.439s
ok go.etcd.io/etcd/server/v3/etcdserver/api/snap 3.712s
ok go.etcd.io/etcd/server/v3/etcdserver/api/v2auth 7.060s
ok go.etcd.io/etcd/server/v3/etcdserver/api/v2discovery 5.179s
ok go.etcd.io/etcd/server/v3/etcdserver/api/v2error 5.165s
ok go.etcd.io/etcd/server/v3/etcdserver/api/v2http 4.994s
ok go.etcd.io/etcd/server/v3/etcdserver/api/v2http/httptypes 4.905s
ok go.etcd.io/etcd/server/v3/etcdserver/api/v2store 4.673s
ok go.etcd.io/etcd/server/v3/etcdserver/api/v3compactor 6.120s
ok go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc 1.795s
ok go.etcd.io/etcd/server/v3/etcdserver/cindex 3.847s
ok go.etcd.io/etcd/server/v3/lease 17.456s
ok go.etcd.io/etcd/server/v3/lease/leasehttp 5.724s
ok go.etcd.io/etcd/server/v3/mvcc 52.127s
ok go.etcd.io/etcd/server/v3/mvcc/backend 18.289s
ok go.etcd.io/etcd/server/v3/proxy/httpproxy 2.226s
ok go.etcd.io/etcd/server/v3/proxy/tcpproxy 2.471s
ok go.etcd.io/etcd/server/v3/wal 45.579s
ok go.etcd.io/etcd/server/v3/wal/walpb 1.730s
% (cd etcdutl && 'env' 'go' 'test' '-short' '-timeout=3m' '--race' './...')
? go.etcd.io/etcd/etcdutl/v3/etcdutl [no test files]
? go.etcd.io/etcd/etcdutl/v3/snapshot [no test files]
ok go.etcd.io/etcd/etcdutl/v3 1.514s
% (cd etcdctl && 'env' 'go' 'test' '-short' '-timeout=3m' '--race' './...')
? go.etcd.io/etcd/etcdctl/v3/ctlv2 [no test files]
? go.etcd.io/etcd/etcdctl/v3/ctlv3 [no test files]
ok go.etcd.io/etcd/etcdctl/v3 1.602s
ok go.etcd.io/etcd/etcdctl/v3/ctlv2/command 2.031s
ok go.etcd.io/etcd/etcdctl/v3/ctlv3/command 2.499s
% (cd tests && 'env' 'go' 'test' '-short' '-timeout=3m' '--race' './...')
? go.etcd.io/etcd/tests/v3/functional/cmd/etcd-agent [no test files]
? go.etcd.io/etcd/tests/v3/functional/cmd/etcd-proxy [no test files]
? go.etcd.io/etcd/tests/v3/functional/cmd/etcd-runner [no test files]
? go.etcd.io/etcd/tests/v3/functional/cmd/etcd-tester [no test files]
? go.etcd.io/etcd/tests/v3/functional/runner [no test files]
ok go.etcd.io/etcd/tests/v3/e2e 3.288s
ok go.etcd.io/etcd/tests/v3/functional/agent 2.006s
ok go.etcd.io/etcd/tests/v3/functional/rpcpb 3.813s
ok go.etcd.io/etcd/tests/v3/functional/tester 2.662s
ok go.etcd.io/etcd/tests/v3/integration 4.128s
ok go.etcd.io/etcd/tests/v3/integration/client 4.820s
ok go.etcd.io/etcd/tests/v3/integration/client/examples 6.113s
ok go.etcd.io/etcd/tests/v3/integration/clientv3 5.590s
ok go.etcd.io/etcd/tests/v3/integration/clientv3/concurrency 6.257s
ok go.etcd.io/etcd/tests/v3/integration/clientv3/connectivity 6.882s
ok go.etcd.io/etcd/tests/v3/integration/clientv3/examples 7.403s
ok go.etcd.io/etcd/tests/v3/integration/clientv3/experimental/recipes 7.916s
ok go.etcd.io/etcd/tests/v3/integration/clientv3/lease 7.293s
ok go.etcd.io/etcd/tests/v3/integration/clientv3/naming 7.191s
ok go.etcd.io/etcd/tests/v3/integration/clientv3/snapshot 7.173s
ok go.etcd.io/etcd/tests/v3/integration/embed 7.339s
ok go.etcd.io/etcd/tests/v3/integration/proxy/grpcproxy 7.109s
ok go.etcd.io/etcd/tests/v3/integration/snapshot 7.039s
ok go.etcd.io/etcd/tests/v3/integration/v2store 7.193s
% 'env' 'go' 'test' '-short' '-timeout=3m' '--race' './...'
? go.etcd.io/etcd/v3 [no test files]
? go.etcd.io/etcd/v3/contrib/lock/client [no test files]
? go.etcd.io/etcd/v3/contrib/lock/storage [no test files]
? go.etcd.io/etcd/v3/tools/benchmark [no test files]
? go.etcd.io/etcd/v3/tools/benchmark/cmd [no test files]
? go.etcd.io/etcd/v3/tools/etcd-dump-db [no test files]
? go.etcd.io/etcd/v3/tools/etcd-dump-metrics [no test files]
? go.etcd.io/etcd/v3/tools/local-tester/bridge [no test files]
ok go.etcd.io/etcd/v3/contrib/raftexample 16.887s
ok go.etcd.io/etcd/v3/tools/etcd-dump-logs 1.823s
'unit' completed at Mon Jan 22 17:25:59 +05 2024
SUCCESS
! egrep "(--- FAIL:|DATA RACE|panic: test timed out|appears to have leaked)" -B50 -A10 test-MTcwNTkyNjIyNgo.log


</details>

@ahrtr
Copy link
Member

ahrtr commented Jan 22, 2024

It's a little strange. I did not change anything, I just added some more debug log in test_3.5.9 branch on top of v3.5.9.

How was the error in #17232 (comment) generated?

@KosovGrigorii
Copy link
Author

I used go clean --testcache; make test(in 3.5.9 there is no test-unit command in makefile). But I just ran this command again and all unit tests passed on both 3.5.9 and 3.5.7. I have no idea what happened the first time...

The problem with mvcc on the main branch stays though. Also tried to rerun unit test and it failed

@ahrtr
Copy link
Member

ahrtr commented Jan 24, 2024

I used go clean --testcache; make test(in 3.5.9 there is no test-unit command in makefile)

Right, test-unit was added since 3.5.10. Sorry for the confusion.

In order to avoid misunderstanding, could you run command below on 3.5.7, 3.5.8 and 3.5.9 ?

go clean --testcache; PASSES='unit' RACE='true' CPU='4' ./test.sh

and run command below on v3.5.10, v3.5.11, release-3.5 and main?

go clean --testcache; RACE='true' CPU='4' make test-unit

@KosovGrigorii
Copy link
Author

KosovGrigorii commented Jan 24, 2024

results after running on 3.5.7:

verification failed
{"level":"info","ts":"2024-01-24T16:30:43.134+0500","caller":"embed/etcd.go:124","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":"2024-01-24T16:30:43.136+0500","caller":"embed/etcd.go:373","msg":"closing etcd server","name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test3202522293","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]}
{"level":"info","ts":"2024-01-24T16:30:43.136+0500","caller":"embed/etcd.go:375","msg":"closed etcd server","name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test3202522293","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]}
{"level":"info","ts":"2024-01-24T16:30:43.136+0500","caller":"verify/verify.go:57","msg":"verification of persisted state","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test3202522293"}
{"level":"panic","ts":"2024-01-24T16:30:43.137+0500","caller":"backend/backend.go:189","msg":"failed to open database","path":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test3202522293/member/snap/db","error":"open /var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test3202522293/member/snap/db: no such file or directory","stacktrace":"go.etcd.io/etcd/server/v3/mvcc/backend.newBackend\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189\ngo.etcd.io/etcd/server/v3/mvcc/backend.New\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163\ngo.etcd.io/etcd/server/v3/verify.Verify\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76\ngo.etcd.io/etcd/server/v3/verify.VerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94\ngo.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:376\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:436\ngo.etcd.io/etcd/server/v3/embed.StartEtcd.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:113\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:129\ngo.etcd.io/etcd/server/v3/embed.TestEnableAuth\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34\ntesting.tRunner\n\t/usr/local/opt/go/libexec/src/testing/testing.go:1595"}
{"level":"error","ts":"2024-01-24T16:30:43.137+0500","caller":"verify/verify.go:64","msg":"verification of persisted state failed","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test3202522293","stacktrace":"go.etcd.io/etcd/server/v3/verify.Verify.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:64\nruntime.gopanic\n\t/usr/local/opt/go/libexec/src/runtime/panic.go:920\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write\n\t/Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/zapcore/entry.go:234\ngo.uber.org/zap.(*Logger).Panic\n\t/Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/logger.go:227\ngo.etcd.io/etcd/server/v3/mvcc/backend.newBackend\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189\ngo.etcd.io/etcd/server/v3/mvcc/backend.New\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163\ngo.etcd.io/etcd/server/v3/verify.Verify\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76\ngo.etcd.io/etcd/server/v3/verify.VerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94\ngo.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:376\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:436\ngo.etcd.io/etcd/server/v3/embed.StartEtcd.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:113\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:129\ngo.etcd.io/etcd/server/v3/embed.TestEnableAuth\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34\ntesting.tRunner\n\t/usr/local/opt/go/libexec/src/testing/testing.go:1595"}
--- FAIL: TestEnableAuth (0.04s)
panic: failed to open database [recovered]
        panic: failed to open database [recovered]
        panic: failed to open database

goroutine 148 [running]:
testing.tRunner.func1.2({0x2119b00, 0xc000490510})
        /usr/local/opt/go/libexec/src/testing/testing.go:1545 +0x366
testing.tRunner.func1()
        /usr/local/opt/go/libexec/src/testing/testing.go:1548 +0x630
panic({0x2119b00?, 0xc000490510?})
        /usr/local/opt/go/libexec/src/runtime/panic.go:920 +0x270
go.etcd.io/etcd/server/v3/verify.Verify.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:66 +0x505
panic({0x2119b00?, 0xc000490510?})
        /usr/local/opt/go/libexec/src/runtime/panic.go:920 +0x270
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0002b20c0, {0xc000170180, 0x2, 0x2})
        /Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/zapcore/entry.go:234 +0x61b
go.uber.org/zap.(*Logger).Panic(0xc00028e240?, {0x22f399b, 0x17}, {0xc000170180, 0x2, 0x2})
        /Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/logger.go:227 +0x68
go.etcd.io/etcd/server/v3/mvcc/backend.newBackend({{0xc00028e240, 0x53}, 0x5f5e100, 0x2710, {0x0, 0x0}, 0x280000000, 0xc000478730, 0x0, 0x0, ...})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189 +0x6b3
go.etcd.io/etcd/server/v3/mvcc/backend.New(...)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163
go.etcd.io/etcd/server/v3/verify.Verify({{0xc000294000?, 0xb?}, 0xe0?, 0xc000478730?})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76 +0x478
go.etcd.io/etcd/server/v3/verify.VerifyIfEnabled({{0xc000294000?, 0xc000174580?}, 0xc8?, 0xc000478730?})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94 +0x76
go.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled({{0xc000294000?, 0x22ec609?}, 0x12?, 0xc000478730?})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103 +0x66
go.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:376 +0x9e
go.etcd.io/etcd/server/v3/embed.(*Etcd).Close(0xc0002b6000)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:436 +0xcf4
go.etcd.io/etcd/server/v3/embed.StartEtcd.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:113 +0x109
go.etcd.io/etcd/server/v3/embed.StartEtcd(0xc00029e000)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:129 +0xa0c
go.etcd.io/etcd/server/v3/embed.TestEnableAuth(0xc000558340)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34 +0x1d0
testing.tRunner(0xc000558340, 0x234a670)
        /usr/local/opt/go/libexec/src/testing/testing.go:1595 +0x239
created by testing.(*T).Run in goroutine 1
        /usr/local/opt/go/libexec/src/testing/testing.go:1648 +0x82b
FAIL    go.etcd.io/etcd/server/v3/embed 3.835s
ok      go.etcd.io/etcd/server/v3/etcdmain      4.507s
ok      go.etcd.io/etcd/server/v3/etcdserver    30.015s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/etcdhttp       4.825s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/membership     19.906s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp       6.726s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/snap   7.506s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2auth 5.099s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2discovery    3.905s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2error        4.141s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2http 3.357s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2http/httptypes       3.625s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2store        3.028s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v3compactor    3.457s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc  2.155s
ok      go.etcd.io/etcd/server/v3/etcdserver/cindex     3.770s
ok      go.etcd.io/etcd/server/v3/lease 17.001s
ok      go.etcd.io/etcd/server/v3/lease/leasehttp       7.431s
ok      go.etcd.io/etcd/server/v3/mvcc  48.989s
ok      go.etcd.io/etcd/server/v3/mvcc/backend  15.543s
ok      go.etcd.io/etcd/server/v3/proxy/httpproxy       0.574s
ok      go.etcd.io/etcd/server/v3/proxy/tcpproxy        1.989s
ok      go.etcd.io/etcd/server/v3/wal   41.430s
ok      go.etcd.io/etcd/server/v3/wal/walpb     1.546s
FAIL
FAIL: (code:1):
  % (cd server && 'env' 'go' 'test' '-short' '-timeout=3m' '--race=true' '--cpu=4' './...')
FAIL: 'unit' failed at Wed Jan 24 16:31:38 +05 2024

3.5.8:

verification failed
{"level":"warn","ts":"2024-01-24T16:33:30.792095+0500","caller":"embed/config.go:673","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2024-01-24T16:33:30.792311+0500","caller":"embed/etcd.go:127","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":"2024-01-24T16:33:30.794777+0500","caller":"embed/etcd.go:376","msg":"closing etcd server","name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1449737618","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]}
{"level":"info","ts":"2024-01-24T16:33:30.794841+0500","caller":"embed/etcd.go:378","msg":"closed etcd server","name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1449737618","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]}
{"level":"info","ts":"2024-01-24T16:33:30.794897+0500","caller":"verify/verify.go:57","msg":"verification of persisted state","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1449737618"}
{"level":"panic","ts":"2024-01-24T16:33:30.795031+0500","caller":"backend/backend.go:189","msg":"failed to open database","path":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1449737618/member/snap/db","error":"open /var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1449737618/member/snap/db: no such file or directory","stacktrace":"go.etcd.io/etcd/server/v3/mvcc/backend.newBackend\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189\ngo.etcd.io/etcd/server/v3/mvcc/backend.New\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163\ngo.etcd.io/etcd/server/v3/verify.Verify\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76\ngo.etcd.io/etcd/server/v3/verify.VerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94\ngo.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:379\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:439\ngo.etcd.io/etcd/server/v3/embed.StartEtcd.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:116\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:132\ngo.etcd.io/etcd/server/v3/embed.TestEnableAuth\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34\ntesting.tRunner\n\t/usr/local/opt/go/libexec/src/testing/testing.go:1595"}
{"level":"error","ts":"2024-01-24T16:33:30.795233+0500","caller":"verify/verify.go:64","msg":"verification of persisted state failed","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1449737618","stacktrace":"go.etcd.io/etcd/server/v3/verify.Verify.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:64\nruntime.gopanic\n\t/usr/local/opt/go/libexec/src/runtime/panic.go:920\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write\n\t/Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/zapcore/entry.go:234\ngo.uber.org/zap.(*Logger).Panic\n\t/Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/logger.go:227\ngo.etcd.io/etcd/server/v3/mvcc/backend.newBackend\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189\ngo.etcd.io/etcd/server/v3/mvcc/backend.New\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163\ngo.etcd.io/etcd/server/v3/verify.Verify\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76\ngo.etcd.io/etcd/server/v3/verify.VerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94\ngo.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:379\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:439\ngo.etcd.io/etcd/server/v3/embed.StartEtcd.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:116\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:132\ngo.etcd.io/etcd/server/v3/embed.TestEnableAuth\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34\ntesting.tRunner\n\t/usr/local/opt/go/libexec/src/testing/testing.go:1595"}
--- FAIL: TestEnableAuth (0.02s)
panic: failed to open database [recovered]
        panic: failed to open database [recovered]
        panic: failed to open database

goroutine 148 [running]:
testing.tRunner.func1.2({0x219cc40, 0xc0004fead0})
        /usr/local/opt/go/libexec/src/testing/testing.go:1545 +0x366
testing.tRunner.func1()
        /usr/local/opt/go/libexec/src/testing/testing.go:1548 +0x630
panic({0x219cc40?, 0xc0004fead0?})
        /usr/local/opt/go/libexec/src/runtime/panic.go:920 +0x270
go.etcd.io/etcd/server/v3/verify.Verify.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:66 +0x505
panic({0x219cc40?, 0xc0004fead0?})
        /usr/local/opt/go/libexec/src/runtime/panic.go:920 +0x270
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0002509c0, {0xc000226200, 0x2, 0x2})
        /Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/zapcore/entry.go:234 +0x61b
go.uber.org/zap.(*Logger).Panic(0xc000022300?, {0x237d0ef, 0x17}, {0xc000226200, 0x2, 0x2})
        /Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/logger.go:227 +0x68
go.etcd.io/etcd/server/v3/mvcc/backend.newBackend({{0xc000022300, 0x53}, 0x5f5e100, 0x2710, {0x0, 0x0}, 0x280000000, 0xc0005ae910, 0x0, 0x0, ...})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189 +0x6b3
go.etcd.io/etcd/server/v3/mvcc/backend.New(...)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163
go.etcd.io/etcd/server/v3/verify.Verify({{0xc000238050?, 0xb?}, 0x20?, 0xc0005ae910?})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76 +0x478
go.etcd.io/etcd/server/v3/verify.VerifyIfEnabled({{0xc000238050?, 0xc000518dc0?}, 0x48?, 0xc0005ae910?})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94 +0x76
go.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled({{0xc000238050?, 0x2375d1f?}, 0x12?, 0xc0005ae910?})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103 +0x66
go.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:379 +0x9e
go.etcd.io/etcd/server/v3/embed.(*Etcd).Close(0xc000589500)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:439 +0xcf4
go.etcd.io/etcd/server/v3/embed.StartEtcd.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:116 +0x109
go.etcd.io/etcd/server/v3/embed.StartEtcd(0xc0005d8600)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:132 +0xa2c
go.etcd.io/etcd/server/v3/embed.TestEnableAuth(0xc000182d00)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34 +0x1d0
testing.tRunner(0xc000182d00, 0x23d4f20)
        /usr/local/opt/go/libexec/src/testing/testing.go:1595 +0x239
created by testing.(*T).Run in goroutine 1
        /usr/local/opt/go/libexec/src/testing/testing.go:1648 +0x82b
FAIL    go.etcd.io/etcd/server/v3/embed 0.843s
ok      go.etcd.io/etcd/server/v3/etcdmain      2.080s
?       go.etcd.io/etcd/server/v3/lease/leasepb [no test files]
?       go.etcd.io/etcd/server/v3/mock/mockstorage      [no test files]
?       go.etcd.io/etcd/server/v3/mock/mockstore        [no test files]
?       go.etcd.io/etcd/server/v3/mock/mockwait [no test files]
?       go.etcd.io/etcd/server/v3/mvcc/backend/testing  [no test files]
?       go.etcd.io/etcd/server/v3/mvcc/buckets  [no test files]
?       go.etcd.io/etcd/server/v3/proxy/grpcproxy       [no test files]
?       go.etcd.io/etcd/server/v3/proxy/grpcproxy/adapter       [no test files]
?       go.etcd.io/etcd/server/v3/proxy/grpcproxy/cache [no test files]
?       go.etcd.io/etcd/server/v3/storage/mvcc/testutil [no test files]
?       go.etcd.io/etcd/server/v3/verify        [no test files]
ok      go.etcd.io/etcd/server/v3/etcdserver    26.256s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/etcdhttp       1.667s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/membership     16.407s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp       3.200s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/snap   2.919s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2auth 3.401s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2discovery    2.927s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2error        3.213s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2http 3.328s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2http/httptypes       3.472s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2store        3.353s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v3compactor    4.839s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc  1.654s
ok      go.etcd.io/etcd/server/v3/etcdserver/cindex     3.752s
ok      go.etcd.io/etcd/server/v3/lease 17.008s
ok      go.etcd.io/etcd/server/v3/lease/leasehttp       7.023s
ok      go.etcd.io/etcd/server/v3/mvcc  49.708s
ok      go.etcd.io/etcd/server/v3/mvcc/backend  16.847s
ok      go.etcd.io/etcd/server/v3/proxy/httpproxy       3.203s
ok      go.etcd.io/etcd/server/v3/proxy/tcpproxy        3.559s
ok      go.etcd.io/etcd/server/v3/wal   41.179s
ok      go.etcd.io/etcd/server/v3/wal/walpb     3.477s
FAIL
FAIL: (code:1):
  % (cd server && 'env' 'go' 'test' '-short' '-timeout=3m' '--race=true' '--cpu=4' './...')
FAIL: 'unit' failed at Wed Jan 24 16:34:30 +05 2024

3.5.9:

verification failed ``` {"level":"warn","ts":"2024-01-24T16:37:25.292845+0500","caller":"embed/config.go:673","msg":"Running http and grpc server on single port. This is not recommended for production."} {"level":"info","ts":"2024-01-24T16:37:25.293058+0500","caller":"embed/etcd.go:127","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]} {"level":"info","ts":"2024-01-24T16:37:25.294901+0500","caller":"embed/etcd.go:376","msg":"closing etcd server","name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1495128825","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]} {"level":"info","ts":"2024-01-24T16:37:25.294961+0500","caller":"embed/etcd.go:378","msg":"closed etcd server","name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1495128825","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]} {"level":"info","ts":"2024-01-24T16:37:25.29501+0500","caller":"verify/verify.go:57","msg":"verification of persisted state","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1495128825"} {"level":"panic","ts":"2024-01-24T16:37:25.295111+0500","caller":"backend/backend.go:189","msg":"failed to open database","path":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1495128825/member/snap/db","error":"open /var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1495128825/member/snap/db: no such file or directory","stacktrace":"go.etcd.io/etcd/server/v3/mvcc/backend.newBackend\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189\ngo.etcd.io/etcd/server/v3/mvcc/backend.New\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163\ngo.etcd.io/etcd/server/v3/verify.Verify\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76\ngo.etcd.io/etcd/server/v3/verify.VerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94\ngo.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:379\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:439\ngo.etcd.io/etcd/server/v3/embed.StartEtcd.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:116\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:132\ngo.etcd.io/etcd/server/v3/embed.TestEnableAuth\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34\ntesting.tRunner\n\t/usr/local/opt/go/libexec/src/testing/testing.go:1595"} {"level":"error","ts":"2024-01-24T16:37:25.295279+0500","caller":"verify/verify.go:64","msg":"verification of persisted state failed","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1495128825","stacktrace":"go.etcd.io/etcd/server/v3/verify.Verify.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:64\nruntime.gopanic\n\t/usr/local/opt/go/libexec/src/runtime/panic.go:920\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write\n\t/Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/zapcore/entry.go:234\ngo.uber.org/zap.(*Logger).Panic\n\t/Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/logger.go:227\ngo.etcd.io/etcd/server/v3/mvcc/backend.newBackend\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189\ngo.etcd.io/etcd/server/v3/mvcc/backend.New\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163\ngo.etcd.io/etcd/server/v3/verify.Verify\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76\ngo.etcd.io/etcd/server/v3/verify.VerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94\ngo.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:379\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:439\ngo.etcd.io/etcd/server/v3/embed.StartEtcd.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:116\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:132\ngo.etcd.io/etcd/server/v3/embed.TestEnableAuth\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34\ntesting.tRunner\n\t/usr/local/opt/go/libexec/src/testing/testing.go:1595"} --- FAIL: TestEnableAuth (0.00s) panic: failed to open database [recovered] panic: failed to open database [recovered] panic: failed to open database

goroutine 146 [running]:
testing.tRunner.func1.2({0x219d1c0, 0xc000598cc0})
/usr/local/opt/go/libexec/src/testing/testing.go:1545 +0x366
testing.tRunner.func1()
/usr/local/opt/go/libexec/src/testing/testing.go:1548 +0x630
panic({0x219d1c0?, 0xc000598cc0?})
/usr/local/opt/go/libexec/src/runtime/panic.go:920 +0x270
go.etcd.io/etcd/server/v3/verify.Verify.func1()
/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:66 +0x505
panic({0x219d1c0?, 0xc000598cc0?})
/usr/local/opt/go/libexec/src/runtime/panic.go:920 +0x270
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0002fe240, {0xc00020a180, 0x2, 0x2})
/Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/zapcore/entry.go:234 +0x61b
go.uber.org/zap.(*Logger).Panic(0xc0002dc300?, {0x237d6cf, 0x17}, {0xc00020a180, 0x2, 0x2})
/Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/logger.go:227 +0x68
go.etcd.io/etcd/server/v3/mvcc/backend.newBackend({{0xc0002dc300, 0x53}, 0x5f5e100, 0x2710, {0x0, 0x0}, 0x280000000, 0xc00059c730, 0x0, 0x0, ...})
/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189 +0x6b3
go.etcd.io/etcd/server/v3/mvcc/backend.New(...)
/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163
go.etcd.io/etcd/server/v3/verify.Verify({{0xc000146190?, 0xb?}, 0xa0?, 0xc00059c730?})
/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76 +0x478
go.etcd.io/etcd/server/v3/verify.VerifyIfEnabled({{0xc000146190?, 0xc0000a1740?}, 0x88?, 0xc00059c730?})
/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94 +0x76
go.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled({{0xc000146190?, 0x23762ff?}, 0x12?, 0xc00059c730?})
/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103 +0x66
go.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1()
/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:379 +0x9e
go.etcd.io/etcd/server/v3/embed.(*Etcd).Close(0xc0001b2e00)
/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:439 +0xcf4
go.etcd.io/etcd/server/v3/embed.StartEtcd.func1()
/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:116 +0x109
go.etcd.io/etcd/server/v3/embed.StartEtcd(0xc0002ec000)
/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:132 +0xa2c
go.etcd.io/etcd/server/v3/embed.TestEnableAuth(0xc000102ea0)
/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34 +0x1d0
testing.tRunner(0xc000102ea0, 0x23d5500)
/usr/local/opt/go/libexec/src/testing/testing.go:1595 +0x239
created by testing.(*T).Run in goroutine 1
/usr/local/opt/go/libexec/src/testing/testing.go:1648 +0x82b
FAIL go.etcd.io/etcd/server/v3/embed 2.091s
ok go.etcd.io/etcd/server/v3/etcdmain 3.628s
ok go.etcd.io/etcd/server/v3/etcdserver 27.083s
ok go.etcd.io/etcd/server/v3/etcdserver/api/etcdhttp 4.787s
ok go.etcd.io/etcd/server/v3/etcdserver/api/membership 20.563s
ok go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp 5.097s
ok go.etcd.io/etcd/server/v3/etcdserver/api/snap 7.436s
ok go.etcd.io/etcd/server/v3/etcdserver/api/v2auth 7.253s
ok go.etcd.io/etcd/server/v3/etcdserver/api/v2discovery 5.361s
ok go.etcd.io/etcd/server/v3/etcdserver/api/v2error 5.414s
ok go.etcd.io/etcd/server/v3/etcdserver/api/v2http 4.091s
ok go.etcd.io/etcd/server/v3/etcdserver/api/v2http/httptypes 4.336s
ok go.etcd.io/etcd/server/v3/etcdserver/api/v2store 3.977s
ok go.etcd.io/etcd/server/v3/etcdserver/api/v3compactor 5.411s
ok go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc 3.078s
ok go.etcd.io/etcd/server/v3/etcdserver/cindex 4.382s
ok go.etcd.io/etcd/server/v3/lease 16.506s
ok go.etcd.io/etcd/server/v3/lease/leasehttp 5.877s
ok go.etcd.io/etcd/server/v3/mvcc 50.254s
ok go.etcd.io/etcd/server/v3/mvcc/backend 16.915s
ok go.etcd.io/etcd/server/v3/proxy/httpproxy 2.329s
ok go.etcd.io/etcd/server/v3/proxy/tcpproxy 2.290s
ok go.etcd.io/etcd/server/v3/wal 43.059s
ok go.etcd.io/etcd/server/v3/wal/walpb 1.612s
FAIL
FAIL: (code:1):
% (cd server && 'env' 'go' 'test' '-short' '-timeout=3m' '--race=true' '--cpu=4' './...')
FAIL: 'unit' failed at Wed Jan 24 16:38:23 +05 2024

3.5.10:

Verification failed and bbolt panicking
{"level":"warn","ts":"2024-01-24T16:40:07.325117+0500","caller":"embed/config.go:676","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2024-01-24T16:40:07.325342+0500","caller":"embed/etcd.go:127","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":"2024-01-24T16:40:07.327251+0500","caller":"embed/etcd.go:376","msg":"closing etcd server","name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test4277713686","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]}
{"level":"info","ts":"2024-01-24T16:40:07.327312+0500","caller":"embed/etcd.go:378","msg":"closed etcd server","name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test4277713686","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]}
{"level":"info","ts":"2024-01-24T16:40:07.327372+0500","caller":"verify/verify.go:57","msg":"verification of persisted state","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test4277713686"}
{"level":"panic","ts":"2024-01-24T16:40:07.327473+0500","caller":"backend/backend.go:189","msg":"failed to open database","path":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test4277713686/member/snap/db","error":"open /var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test4277713686/member/snap/db: no such file or directory","stacktrace":"go.etcd.io/etcd/server/v3/mvcc/backend.newBackend\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189\ngo.etcd.io/etcd/server/v3/mvcc/backend.New\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163\ngo.etcd.io/etcd/server/v3/verify.Verify\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76\ngo.etcd.io/etcd/server/v3/verify.VerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94\ngo.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:379\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:439\ngo.etcd.io/etcd/server/v3/embed.StartEtcd.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:116\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:132\ngo.etcd.io/etcd/server/v3/embed.TestEnableAuth\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34\ntesting.tRunner\n\t/usr/local/opt/go/libexec/src/testing/testing.go:1595"}
{"level":"error","ts":"2024-01-24T16:40:07.327645+0500","caller":"verify/verify.go:64","msg":"verification of persisted state failed","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test4277713686","stacktrace":"go.etcd.io/etcd/server/v3/verify.Verify.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:64\nruntime.gopanic\n\t/usr/local/opt/go/libexec/src/runtime/panic.go:920\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write\n\t/Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/zapcore/entry.go:234\ngo.uber.org/zap.(*Logger).Panic\n\t/Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/logger.go:227\ngo.etcd.io/etcd/server/v3/mvcc/backend.newBackend\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189\ngo.etcd.io/etcd/server/v3/mvcc/backend.New\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163\ngo.etcd.io/etcd/server/v3/verify.Verify\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76\ngo.etcd.io/etcd/server/v3/verify.VerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94\ngo.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:379\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:439\ngo.etcd.io/etcd/server/v3/embed.StartEtcd.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:116\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:132\ngo.etcd.io/etcd/server/v3/embed.TestEnableAuth\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34\ntesting.tRunner\n\t/usr/local/opt/go/libexec/src/testing/testing.go:1595"}
--- FAIL: TestEnableAuth (0.02s)
panic: failed to open database [recovered]
        panic: failed to open database [recovered]
        panic: failed to open database

goroutine 124 [running]:
testing.tRunner.func1.2({0x21ce480, 0xc000117030})
        /usr/local/opt/go/libexec/src/testing/testing.go:1545 +0x366
testing.tRunner.func1()
        /usr/local/opt/go/libexec/src/testing/testing.go:1548 +0x630
panic({0x21ce480?, 0xc000117030?})
        /usr/local/opt/go/libexec/src/runtime/panic.go:920 +0x270
go.etcd.io/etcd/server/v3/verify.Verify.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:66 +0x505
panic({0x21ce480?, 0xc000117030?})
        /usr/local/opt/go/libexec/src/runtime/panic.go:920 +0x270
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0000320c0, {0xc00051c200, 0x2, 0x2})
        /Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/zapcore/entry.go:234 +0x61b
go.uber.org/zap.(*Logger).Panic(0xc00002c300?, {0x23b9220, 0x17}, {0xc00051c200, 0x2, 0x2})
        /Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/logger.go:227 +0x68
go.etcd.io/etcd/server/v3/mvcc/backend.newBackend({{0xc00002c300, 0x53}, 0x5f5e100, 0x2710, {0x0, 0x0}, 0x280000000, 0xc00010ecd0, 0x0, 0x0, ...})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189 +0x6b3
go.etcd.io/etcd/server/v3/mvcc/backend.New(...)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163
go.etcd.io/etcd/server/v3/verify.Verify({{0xc00004a320?, 0xb?}, 0xa0?, 0xc00010ecd0?})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76 +0x478
go.etcd.io/etcd/server/v3/verify.VerifyIfEnabled({{0xc00004a320?, 0xc000469300?}, 0xc8?, 0xc00010ecd0?})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94 +0x76
go.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled({{0xc00004a320?, 0x23b1d2a?}, 0x12?, 0xc00010ecd0?})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103 +0x66
go.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:379 +0x9e
go.etcd.io/etcd/server/v3/embed.(*Etcd).Close(0xc00052d500)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:439 +0xcf4
go.etcd.io/etcd/server/v3/embed.StartEtcd.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:116 +0x109
go.etcd.io/etcd/server/v3/embed.StartEtcd(0xc0000dd200)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:132 +0xa2c
go.etcd.io/etcd/server/v3/embed.TestEnableAuth(0xc000582ea0)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34 +0x1d0
testing.tRunner(0xc000582ea0, 0x2411c38)
        /usr/local/opt/go/libexec/src/testing/testing.go:1595 +0x239
created by testing.(*T).Run in goroutine 1
        /usr/local/opt/go/libexec/src/testing/testing.go:1648 +0x82b
FAIL    go.etcd.io/etcd/server/v3/embed 0.814s
ok      go.etcd.io/etcd/server/v3/etcdmain      2.849s
?       go.etcd.io/etcd/server/v3/lease/leasepb [no test files]
?       go.etcd.io/etcd/server/v3/mock/mockwait [no test files]
?       go.etcd.io/etcd/server/v3/mock/mockstorage      [no test files]
?       go.etcd.io/etcd/server/v3/mock/mockstore        [no test files]
?       go.etcd.io/etcd/server/v3/mvcc/backend/testing  [no test files]
?       go.etcd.io/etcd/server/v3/mvcc/buckets  [no test files]
?       go.etcd.io/etcd/server/v3/proxy/grpcproxy       [no test files]
?       go.etcd.io/etcd/server/v3/proxy/grpcproxy/adapter       [no test files]
?       go.etcd.io/etcd/server/v3/proxy/grpcproxy/cache [no test files]
?       go.etcd.io/etcd/server/v3/storage/mvcc/testutil [no test files]
?       go.etcd.io/etcd/server/v3/verify        [no test files]
ok      go.etcd.io/etcd/server/v3/etcdserver    22.507s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/etcdhttp       2.664s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/membership     14.668s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp       3.508s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/snap   3.893s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2auth 3.406s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2discovery    2.969s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2error        3.192s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2http 1.747s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2http/httptypes       1.931s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2store        2.289s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v3compactor    3.362s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc  1.602s
ok      go.etcd.io/etcd/server/v3/etcdserver/cindex     3.783s
ok      go.etcd.io/etcd/server/v3/lease 15.029s
ok      go.etcd.io/etcd/server/v3/lease/leasehttp       6.340s
{"level":"info","msg":"compact tree index","revision":1}
{"level":"info","msg":"compact tree index","revision":2}
{"level":"info","msg":"compact tree index","revision":3}
{"level":"info","msg":"compact tree index","revision":4}
{"level":"info","msg":"compact tree index","revision":5}
{"level":"info","msg":"compact tree index","revision":6}
{"level":"info","msg":"compact tree index","revision":7}
{"level":"info","msg":"compact tree index","revision":8}
{"level":"info","msg":"compact tree index","revision":9}
{"level":"info","msg":"compact tree index","revision":10}
{"level":"info","msg":"compact tree index","revision":11}
{"level":"info","msg":"compact tree index","revision":12}
{"level":"info","msg":"compact tree index","revision":13}
{"level":"info","msg":"compact tree index","revision":14}
{"level":"info","msg":"compact tree index","revision":15}
{"level":"info","msg":"compact tree index","revision":16}
{"level":"info","msg":"compact tree index","revision":17}
{"level":"info","msg":"compact tree index","revision":18}
{"level":"info","msg":"compact tree index","revision":19}
{"level":"info","msg":"compact tree index","revision":1}
{"level":"info","msg":"compact tree index","revision":2}
{"level":"info","msg":"compact tree index","revision":3}
{"level":"info","msg":"compact tree index","revision":4}
{"level":"info","msg":"compact tree index","revision":5}
{"level":"info","msg":"compact tree index","revision":6}
{"level":"info","msg":"compact tree index","revision":7}
{"level":"info","msg":"compact tree index","revision":8}
{"level":"info","msg":"compact tree index","revision":9}
{"level":"info","msg":"compact tree index","revision":10}
{"level":"info","msg":"compact tree index","revision":11}
{"level":"info","msg":"compact tree index","revision":12}
{"level":"info","msg":"compact tree index","revision":13}
{"level":"info","msg":"compact tree index","revision":14}
{"level":"info","msg":"compact tree index","revision":15}
{"level":"info","msg":"compact tree index","revision":16}
{"level":"info","msg":"compact tree index","revision":17}
{"level":"info","msg":"compact tree index","revision":18}
{"level":"info","msg":"compact tree index","revision":19}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":4}
{"level":"info","msg":"finished scheduled compaction","compact-revision":4,"took":"247.972µs","hash":1500391675}
{"level":"info","msg":"storing new hash","hash":1500391675,"revision":4,"compact-revision":-1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":4}
{"level":"info","msg":"finished scheduled compaction","compact-revision":4,"took":"231.024µs","hash":1500391675}
{"level":"info","msg":"storing new hash","hash":1500391675,"revision":4,"compact-revision":-1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":1}
{"level":"info","msg":"finished scheduled compaction","compact-revision":1,"took":"227.454µs","hash":1084519789}
{"level":"info","msg":"storing new hash","hash":1084519789,"revision":1,"compact-revision":-1}
{"level":"info","msg":"compact tree index","revision":2}
{"level":"info","msg":"finished scheduled compaction","compact-revision":2,"took":"206.336µs","hash":551221740}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":3}
{"level":"info","msg":"finished scheduled compaction","compact-revision":3,"took":"29.208311ms","hash":4023298219}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":0}
{"level":"info","msg":"finished scheduled compaction","compact-revision":0,"took":"69.921777ms","hash":1084519789}
{"level":"info","msg":"storing new hash","hash":1084519789,"revision":0,"compact-revision":-1}
{"level":"info","msg":"compact tree index","revision":1}
{"level":"info","msg":"finished scheduled compaction","compact-revision":1,"took":"123.160211ms","hash":1084519789}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":4}
{"level":"info","msg":"finished scheduled compaction","compact-revision":4,"took":"135.344µs","hash":1012267086}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":5}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":4}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":1}
{"level":"info","msg":"finished scheduled compaction","compact-revision":1,"took":"206.118µs","hash":1084519789}
{"level":"info","msg":"storing new hash","hash":1084519789,"revision":1,"compact-revision":-1}
{"level":"info","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":1}
{"level":"info","msg":"kvstore restored","current-rev":3}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":4}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":5}
{"level":"info","msg":"finished scheduled compaction","compact-revision":5,"took":"120.04µs","hash":873998276}
{"level":"info","msg":"storing new hash","hash":873998276,"revision":5,"compact-revision":-1}
{"level":"info","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":5}
{"level":"info","msg":"kvstore restored","current-rev":5}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"finished scheduled compaction","compact-revision":3,"took":"12.567µs","hash":3457566959}
{"level":"info","msg":"storing new hash","hash":3457566959,"revision":3,"compact-revision":-1}
{"level":"info","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":3}
{"level":"info","msg":"kvstore restored","current-rev":5}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":35}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":4}
{"level":"info","msg":"resume scheduled compaction","meta-bucket-name":"meta","meta-bucket-name-key":"scheduledCompactRev","scheduled-compact-revision":2}
{"level":"info","msg":"compact tree index","revision":2}
{"level":"info","msg":"finished scheduled compaction","compact-revision":2,"took":"84.28µs","hash":3305255506}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":9900}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9900,"took":"1.086920458s","hash":2982592828}
{"level":"info","msg":"storing new hash","hash":2982592828,"revision":9900,"compact-revision":-1}
{"level":"info","msg":"compact tree index","revision":9901}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9901,"took":"1.100417ms","hash":4092902393}
{"level":"info","msg":"storing new hash","hash":4092902393,"revision":9901,"compact-revision":9900}
{"level":"info","msg":"compact tree index","revision":9902}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9902,"took":"1.916979ms","hash":3442946395}
{"level":"info","msg":"storing new hash","hash":3442946395,"revision":9902,"compact-revision":9901}
{"level":"info","msg":"compact tree index","revision":9903}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9903,"took":"2.188067ms","hash":3934476647}
{"level":"info","msg":"storing new hash","hash":3934476647,"revision":9903,"compact-revision":9902}
{"level":"info","msg":"compact tree index","revision":9904}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9904,"took":"2.004739ms","hash":1979317650}
{"level":"info","msg":"storing new hash","hash":1979317650,"revision":9904,"compact-revision":9903}
{"level":"info","msg":"compact tree index","revision":9905}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9905,"took":"1.043581ms","hash":4126965741}
{"level":"info","msg":"storing new hash","hash":4126965741,"revision":9905,"compact-revision":9904}
{"level":"info","msg":"compact tree index","revision":9906}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9906,"took":"880.743µs","hash":3409966415}
{"level":"info","msg":"storing new hash","hash":3409966415,"revision":9906,"compact-revision":9905}
{"level":"info","msg":"compact tree index","revision":9907}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9907,"took":"912.209µs","hash":3968572787}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9908}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9908,"took":"987.308µs","hash":1013037252}
{"level":"info","msg":"storing new hash","hash":1013037252,"revision":9908,"compact-revision":9907}
{"level":"info","msg":"compact tree index","revision":9909}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9909,"took":"1.021446ms","hash":1305141790}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9910}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9910,"took":"1.394089ms","hash":189389043}
{"level":"info","msg":"storing new hash","hash":189389043,"revision":9910,"compact-revision":9909}
{"level":"info","msg":"compact tree index","revision":9911}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9911,"took":"1.088105ms","hash":754823375}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9912}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9912,"took":"935.272µs","hash":4067225043}
{"level":"info","msg":"storing new hash","hash":4067225043,"revision":9912,"compact-revision":9911}
{"level":"info","msg":"compact tree index","revision":9913}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9913,"took":"1.747955ms","hash":804901573}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9914}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9914,"took":"1.698905ms","hash":3600863096}
{"level":"info","msg":"storing new hash","hash":3600863096,"revision":9914,"compact-revision":9913}
{"level":"info","msg":"compact tree index","revision":9915}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9915,"took":"2.07367ms","hash":4044662596}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9916}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9916,"took":"2.119105ms","hash":1824345379}
{"level":"info","msg":"storing new hash","hash":1824345379,"revision":9916,"compact-revision":9915}
{"level":"info","msg":"compact tree index","revision":9917}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9917,"took":"1.749198ms","hash":487987193}
{"level":"info","msg":"storing new hash","hash":487987193,"revision":9917,"compact-revision":9916}
{"level":"info","msg":"compact tree index","revision":9918}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9918,"took":"1.689766ms","hash":1855345291}
{"level":"info","msg":"storing new hash","hash":1855345291,"revision":9918,"compact-revision":9917}
{"level":"info","msg":"compact tree index","revision":9919}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9919,"took":"1.729692ms","hash":1226980023}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9920}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9920,"took":"1.659456ms","hash":1439986907}
{"level":"info","msg":"storing new hash","hash":1439986907,"revision":9920,"compact-revision":9919}
{"level":"info","msg":"compact tree index","revision":9921}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9921,"took":"1.017867ms","hash":492703523}
{"level":"info","msg":"storing new hash","hash":492703523,"revision":9921,"compact-revision":9920}
{"level":"info","msg":"compact tree index","revision":9922}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9922,"took":"907.154µs","hash":597525889}
{"level":"info","msg":"storing new hash","hash":597525889,"revision":9922,"compact-revision":9921}
{"level":"info","msg":"compact tree index","revision":9923}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9923,"took":"1.005046ms","hash":69856701}
{"level":"info","msg":"storing new hash","hash":69856701,"revision":9923,"compact-revision":9922}
{"level":"info","msg":"compact tree index","revision":9924}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9924,"took":"1.942964ms","hash":1578799301}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9925}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9925,"took":"1.74714ms","hash":800180767}
{"level":"info","msg":"storing new hash","hash":800180767,"revision":9925,"compact-revision":9924}
{"level":"info","msg":"compact tree index","revision":9926}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9926,"took":"1.752667ms","hash":292700349}
{"level":"info","msg":"storing new hash","hash":292700349,"revision":9926,"compact-revision":9925}
{"level":"info","msg":"compact tree index","revision":9927}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9927,"took":"1.648149ms","hash":918956161}
{"level":"info","msg":"storing new hash","hash":918956161,"revision":9927,"compact-revision":9926}
{"level":"info","msg":"compact tree index","revision":9928}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9928,"took":"1.616577ms","hash":2769599053}
{"level":"info","msg":"storing new hash","hash":2769599053,"revision":9928,"compact-revision":9927}
{"level":"info","msg":"compact tree index","revision":9929}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9929,"took":"1.307066ms","hash":2021754203}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9930}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9930,"took":"1.59093ms","hash":1040367542}
{"level":"info","msg":"storing new hash","hash":1040367542,"revision":9930,"compact-revision":9929}
{"level":"info","msg":"compact tree index","revision":9931}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9931,"took":"1.580533ms","hash":431413130}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9932}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9932,"took":"1.807895ms","hash":2216639981}
{"level":"info","msg":"storing new hash","hash":2216639981,"revision":9932,"compact-revision":9931}
{"level":"info","msg":"compact tree index","revision":9933}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9933,"took":"1.550344ms","hash":4122240823}
{"level":"info","msg":"storing new hash","hash":4122240823,"revision":9933,"compact-revision":9932}
{"level":"info","msg":"compact tree index","revision":9934}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9934,"took":"1.00121ms","hash":216889994}
{"level":"info","msg":"storing new hash","hash":216889994,"revision":9934,"compact-revision":9933}
{"level":"info","msg":"compact tree index","revision":9935}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9935,"took":"1.483269ms","hash":727257782}
{"level":"info","msg":"storing new hash","hash":727257782,"revision":9935,"compact-revision":9934}
{"level":"info","msg":"compact tree index","revision":9936}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9936,"took":"1.490634ms","hash":1941512540}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9937}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9937,"took":"1.596178ms","hash":4089226019}
{"level":"info","msg":"storing new hash","hash":4089226019,"revision":9937,"compact-revision":9936}
{"level":"info","msg":"compact tree index","revision":9938}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9938,"took":"1.53674ms","hash":2151474769}
{"level":"info","msg":"storing new hash","hash":2151474769,"revision":9938,"compact-revision":9937}
{"level":"info","msg":"compact tree index","revision":9939}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9939,"took":"1.413697ms","hash":2810744429}
{"level":"info","msg":"storing new hash","hash":2810744429,"revision":9939,"compact-revision":9938}
{"level":"info","msg":"compact tree index","revision":9940}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9940,"took":"1.616767ms","hash":732857719}
{"level":"info","msg":"storing new hash","hash":732857719,"revision":9940,"compact-revision":9939}
{"level":"info","msg":"compact tree index","revision":9941}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9941,"took":"1.485771ms","hash":1510334381}
{"level":"info","msg":"storing new hash","hash":1510334381,"revision":9941,"compact-revision":9940}
{"level":"info","msg":"compact tree index","revision":9942}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9942,"took":"1.39763ms","hash":1690719503}
{"level":"info","msg":"storing new hash","hash":1690719503,"revision":9942,"compact-revision":9941}
{"level":"info","msg":"compact tree index","revision":9943}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9943,"took":"1.429398ms","hash":1131560243}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9944}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9944,"took":"1.393057ms","hash":1517942576}
{"level":"info","msg":"storing new hash","hash":1517942576,"revision":9944,"compact-revision":9943}
{"level":"info","msg":"compact tree index","revision":9945}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9945,"took":"883.646µs","hash":2280441894}
{"level":"info","msg":"storing new hash","hash":2280441894,"revision":9945,"compact-revision":9944}
{"level":"info","msg":"compact tree index","revision":9946}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9946,"took":"1.497346ms","hash":3106882180}
{"level":"info","msg":"storing new hash","hash":3106882180,"revision":9946,"compact-revision":9945}
{"level":"info","msg":"compact tree index","revision":9947}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9947,"took":"1.436367ms","hash":2660969144}
{"level":"info","msg":"storing new hash","hash":2660969144,"revision":9947,"compact-revision":9946}
{"level":"info","msg":"compact tree index","revision":9948}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9948,"took":"1.384136ms","hash":1316032271}
{"level":"info","msg":"storing new hash","hash":1316032271,"revision":9948,"compact-revision":9947}
{"level":"info","msg":"compact tree index","revision":9949}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9949,"took":"1.74631ms","hash":1071273429}
{"level":"info","msg":"storing new hash","hash":1071273429,"revision":9949,"compact-revision":9948}
{"level":"info","msg":"compact tree index","revision":9950}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9950,"took":"1.412412ms","hash":2035904312}
{"level":"info","msg":"storing new hash","hash":2035904312,"revision":9950,"compact-revision":9949}
{"level":"info","msg":"compact tree index","revision":9951}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9951,"took":"1.744021ms","hash":1592592132}
{"level":"info","msg":"storing new hash","hash":1592592132,"revision":9951,"compact-revision":9950}
{"level":"info","msg":"compact tree index","revision":9952}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9952,"took":"1.367685ms","hash":251895795}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9953}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9953,"took":"1.3632ms","hash":874997598}
{"level":"info","msg":"storing new hash","hash":874997598,"revision":9953,"compact-revision":9952}
{"level":"info","msg":"compact tree index","revision":9954}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9954,"took":"1.536986ms","hash":3447618275}
{"level":"info","msg":"storing new hash","hash":3447618275,"revision":9954,"compact-revision":9953}
{"level":"info","msg":"compact tree index","revision":9955}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9955,"took":"1.039593ms","hash":3939111647}
{"level":"info","msg":"storing new hash","hash":3939111647,"revision":9955,"compact-revision":9954}
{"level":"info","msg":"compact tree index","revision":9956}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9956,"took":"1.222789ms","hash":2003040440}
{"level":"info","msg":"storing new hash","hash":2003040440,"revision":9956,"compact-revision":9955}
{"level":"info","msg":"compact tree index","revision":9957}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9957,"took":"1.635176ms","hash":113812066}
{"level":"info","msg":"storing new hash","hash":113812066,"revision":9957,"compact-revision":9956}
{"level":"info","msg":"compact tree index","revision":9958}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9958,"took":"1.459733ms","hash":1967712016}
{"level":"info","msg":"storing new hash","hash":1967712016,"revision":9958,"compact-revision":9957}
{"level":"info","msg":"compact tree index","revision":9959}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9959,"took":"1.352291ms","hash":1392283436}
{"level":"info","msg":"storing new hash","hash":1392283436,"revision":9959,"compact-revision":9958}
{"level":"info","msg":"compact tree index","revision":9960}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9960,"took":"1.108506ms","hash":3546829457}
{"level":"info","msg":"storing new hash","hash":3546829457,"revision":9960,"compact-revision":9959}
{"level":"info","msg":"compact tree index","revision":9961}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9961,"took":"1.116222ms","hash":251489671}
{"level":"info","msg":"storing new hash","hash":251489671,"revision":9961,"compact-revision":9960}
{"level":"info","msg":"compact tree index","revision":9962}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9962,"took":"1.207814ms","hash":809428773}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9963}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9963,"took":"1.283039ms","hash":394956569}
{"level":"info","msg":"storing new hash","hash":394956569,"revision":9963,"compact-revision":9962}
{"level":"info","msg":"compact tree index","revision":9964}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9964,"took":"1.111371ms","hash":1304031841}
{"level":"info","msg":"storing new hash","hash":1304031841,"revision":9964,"compact-revision":9963}
{"level":"info","msg":"compact tree index","revision":9965}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9965,"took":"1.397239ms","hash":1007855803}
{"level":"info","msg":"storing new hash","hash":1007855803,"revision":9965,"compact-revision":9964}
{"level":"info","msg":"compact tree index","revision":9966}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9966,"took":"1.198022ms","hash":47260185}
{"level":"info","msg":"storing new hash","hash":47260185,"revision":9966,"compact-revision":9965}
{"level":"info","msg":"compact tree index","revision":9967}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9967,"took":"1.22456ms","hash":627395109}
{"level":"info","msg":"storing new hash","hash":627395109,"revision":9967,"compact-revision":9966}
{"level":"info","msg":"compact tree index","revision":9968}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9968,"took":"2.079633ms","hash":818329119}
{"level":"info","msg":"storing new hash","hash":818329119,"revision":9968,"compact-revision":9967}
{"level":"info","msg":"compact tree index","revision":9969}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9969,"took":"1.086749ms","hash":2965582944}
{"level":"info","msg":"storing new hash","hash":2965582944,"revision":9969,"compact-revision":9968}
{"level":"info","msg":"compact tree index","revision":9970}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9970,"took":"816.513µs","hash":4131393165}
{"level":"info","msg":"storing new hash","hash":4131393165,"revision":9970,"compact-revision":9969}
{"level":"info","msg":"compact tree index","revision":9971}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9971,"took":"1.207037ms","hash":3522455217}
{"level":"info","msg":"storing new hash","hash":3522455217,"revision":9971,"compact-revision":9970}
{"level":"info","msg":"compact tree index","revision":9972}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9972,"took":"616.295µs","hash":1281199318}
{"level":"info","msg":"storing new hash","hash":1281199318,"revision":9972,"compact-revision":9971}
{"level":"info","msg":"compact tree index","revision":9973}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9973,"took":"1.214592ms","hash":1039586828}
{"level":"info","msg":"storing new hash","hash":1039586828,"revision":9973,"compact-revision":9972}
{"level":"info","msg":"compact tree index","revision":9974}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9974,"took":"1.280349ms","hash":3299855281}
{"level":"info","msg":"storing new hash","hash":3299855281,"revision":9974,"compact-revision":9973}
{"level":"info","msg":"compact tree index","revision":9975}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9975,"took":"983.75µs","hash":3810239373}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9976}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9976,"took":"894.712µs","hash":1032519313}
{"level":"info","msg":"storing new hash","hash":1032519313,"revision":9976,"compact-revision":9975}
{"level":"info","msg":"compact tree index","revision":9977}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9977,"took":"1.171765ms","hash":3760177543}
{"level":"info","msg":"storing new hash","hash":3760177543,"revision":9977,"compact-revision":9976}
{"level":"info","msg":"compact tree index","revision":9978}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9978,"took":"876.23µs","hash":2476705013}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9979}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9979,"took":"1.025422ms","hash":3022777545}
{"level":"info","msg":"storing new hash","hash":3022777545,"revision":9979,"compact-revision":9978}
{"level":"info","msg":"compact tree index","revision":9980}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9980,"took":"889.15µs","hash":2769888363}
{"level":"info","msg":"storing new hash","hash":2769888363,"revision":9980,"compact-revision":9979}
{"level":"info","msg":"compact tree index","revision":9981}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9981,"took":"783.599µs","hash":3568467633}
{"level":"info","msg":"storing new hash","hash":3568467633,"revision":9981,"compact-revision":9980}
{"level":"info","msg":"compact tree index","revision":9982}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9982,"took":"1.045941ms","hash":3933336595}
{"level":"info","msg":"storing new hash","hash":3933336595,"revision":9982,"compact-revision":9981}
{"level":"info","msg":"compact tree index","revision":9983}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9983,"took":"886.738µs","hash":3452279855}
{"level":"info","msg":"storing new hash","hash":3452279855,"revision":9983,"compact-revision":9982}
{"level":"info","msg":"compact tree index","revision":9984}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9984,"took":"900.932µs","hash":4232457539}
{"level":"info","msg":"storing new hash","hash":4232457539,"revision":9984,"compact-revision":9983}
{"level":"info","msg":"compact tree index","revision":9985}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9985,"took":"935.061µs","hash":3973754021}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9986}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9986,"took":"770.404µs","hash":3524863495}
{"level":"info","msg":"storing new hash","hash":3524863495,"revision":9986,"compact-revision":9985}
{"level":"info","msg":"compact tree index","revision":9987}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9987,"took":"1.059606ms","hash":4121775675}
{"level":"info","msg":"storing new hash","hash":4121775675,"revision":9987,"compact-revision":9986}
{"level":"info","msg":"compact tree index","revision":9988}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9988,"took":"1.120679ms","hash":625437580}
{"level":"info","msg":"storing new hash","hash":625437580,"revision":9988,"compact-revision":9987}
{"level":"info","msg":"compact tree index","revision":9989}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9989,"took":"742.396µs","hash":1424765270}
{"level":"info","msg":"storing new hash","hash":1424765270,"revision":9989,"compact-revision":9988}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x16535e7]

goroutine 531 [running]:
go.etcd.io/bbolt.(*Tx).Bucket(...)
        /Users/kosovgrigorii/go/pkg/mod/go.etcd.io/bbolt@v1.3.8/tx.go:102
go.etcd.io/etcd/server/v3/mvcc/backend.unsafeForEach(0x0, {0x1d285c8, 0x2240780}, 0x2000?)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/batch_tx.go:235 +0x47
go.etcd.io/etcd/server/v3/mvcc/backend.(*baseReadTx).UnsafeForEach(0xc0003fe370, {0x1d285c8, 0x2240780}, 0xc0004005a0)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/read_tx.go:69 +0x199
go.etcd.io/etcd/server/v3/mvcc.unsafeHashByRev({0x1d29b18, 0xc0003fe370}, 0xc0001fdf00?, 0xc0001fdf00?, 0xc0006544e0?)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/hash.go:34 +0x1b5
go.etcd.io/etcd/server/v3/mvcc.(*store).hashByRev(0xc0003804e0, 0x2710)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/kvstore.go:194 +0x385
go.etcd.io/etcd/server/v3/mvcc.(*hashStorage).HashByRev(0xc00039c1c0, 0x2710)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/hash.go:133 +0x1d2
go.etcd.io/etcd/server/v3/mvcc.TestHashKVWhenCompacting.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/kvstore_test.go:561 +0x108
created by go.etcd.io/etcd/server/v3/mvcc.TestHashKVWhenCompacting in goroutine 327
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/kvstore_test.go:558 +0x3be
FAIL    go.etcd.io/etcd/server/v3/mvcc  34.048s
ok      go.etcd.io/etcd/server/v3/mvcc/backend  12.631s
ok      go.etcd.io/etcd/server/v3/proxy/httpproxy       2.279s
ok      go.etcd.io/etcd/server/v3/proxy/tcpproxy        2.314s
ok      go.etcd.io/etcd/server/v3/wal   34.028s
ok      go.etcd.io/etcd/server/v3/wal/walpb     0.848s
FAIL
FAIL: (code:1):
  % (cd server && 'env' 'go' 'test' '-short' '-timeout=3m' '--race=true' '--cpu=4' './...')
FAIL: 'unit' failed at Wed Jan 24 16:41:02 +05 2024

3.5.11:

Verification failed and bbolt panicking
{"level":"warn","ts":"2024-01-24T16:42:51.307082+0500","caller":"embed/config.go:679","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2024-01-24T16:42:51.307319+0500","caller":"embed/etcd.go:127","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":"2024-01-24T16:42:51.309987+0500","caller":"embed/etcd.go:375","msg":"closing etcd server","name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test3132924753","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]}
{"level":"info","ts":"2024-01-24T16:42:51.310035+0500","caller":"embed/etcd.go:377","msg":"closed etcd server","name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test3132924753","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]}
{"level":"info","ts":"2024-01-24T16:42:51.310108+0500","caller":"verify/verify.go:57","msg":"verification of persisted state","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test3132924753"}
{"level":"panic","ts":"2024-01-24T16:42:51.310206+0500","caller":"backend/backend.go:189","msg":"failed to open database","path":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test3132924753/member/snap/db","error":"open /var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test3132924753/member/snap/db: no such file or directory","stacktrace":"go.etcd.io/etcd/server/v3/mvcc/backend.newBackend\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189\ngo.etcd.io/etcd/server/v3/mvcc/backend.New\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163\ngo.etcd.io/etcd/server/v3/verify.Verify\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76\ngo.etcd.io/etcd/server/v3/verify.VerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94\ngo.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:378\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:438\ngo.etcd.io/etcd/server/v3/embed.StartEtcd.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:116\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:132\ngo.etcd.io/etcd/server/v3/embed.TestEnableAuth\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34\ntesting.tRunner\n\t/usr/local/opt/go/libexec/src/testing/testing.go:1595"}
{"level":"error","ts":"2024-01-24T16:42:51.310376+0500","caller":"verify/verify.go:64","msg":"verification of persisted state failed","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test3132924753","stacktrace":"go.etcd.io/etcd/server/v3/verify.Verify.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:64\nruntime.gopanic\n\t/usr/local/opt/go/libexec/src/runtime/panic.go:920\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write\n\t/Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/zapcore/entry.go:234\ngo.uber.org/zap.(*Logger).Panic\n\t/Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/logger.go:227\ngo.etcd.io/etcd/server/v3/mvcc/backend.newBackend\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189\ngo.etcd.io/etcd/server/v3/mvcc/backend.New\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163\ngo.etcd.io/etcd/server/v3/verify.Verify\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76\ngo.etcd.io/etcd/server/v3/verify.VerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94\ngo.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:378\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:438\ngo.etcd.io/etcd/server/v3/embed.StartEtcd.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:116\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:132\ngo.etcd.io/etcd/server/v3/embed.TestEnableAuth\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34\ntesting.tRunner\n\t/usr/local/opt/go/libexec/src/testing/testing.go:1595"}
--- FAIL: TestEnableAuth (0.02s)
panic: failed to open database [recovered]
        panic: failed to open database [recovered]
        panic: failed to open database

goroutine 147 [running]:
testing.tRunner.func1.2({0x2219fa0, 0xc0002e2f90})
        /usr/local/opt/go/libexec/src/testing/testing.go:1545 +0x366
testing.tRunner.func1()
        /usr/local/opt/go/libexec/src/testing/testing.go:1548 +0x630
panic({0x2219fa0?, 0xc0002e2f90?})
        /usr/local/opt/go/libexec/src/runtime/panic.go:920 +0x270
go.etcd.io/etcd/server/v3/verify.Verify.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:66 +0x505
panic({0x2219fa0?, 0xc0002e2f90?})
        /usr/local/opt/go/libexec/src/runtime/panic.go:920 +0x270
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc00043e240, {0xc000122300, 0x2, 0x2})
        /Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/zapcore/entry.go:234 +0x61b
go.uber.org/zap.(*Logger).Panic(0xc000170360?, {0x24115d9, 0x17}, {0xc000122300, 0x2, 0x2})
        /Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/logger.go:227 +0x68
go.etcd.io/etcd/server/v3/mvcc/backend.newBackend({{0xc000170360, 0x53}, 0x5f5e100, 0x2710, {0x0, 0x0}, 0x280000000, 0xc0000026e0, 0x0, 0x0, ...})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189 +0x6b3
go.etcd.io/etcd/server/v3/mvcc/backend.New(...)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163
go.etcd.io/etcd/server/v3/verify.Verify({{0xc000112050?, 0xb?}, 0x0?, 0xc0000026e0?})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76 +0x478
go.etcd.io/etcd/server/v3/verify.VerifyIfEnabled({{0xc000112050?, 0xc00030f180?}, 0x48?, 0xc0000026e0?})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94 +0x76
go.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled({{0xc000112050?, 0x2409e20?}, 0x12?, 0xc0000026e0?})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103 +0x66
go.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:378 +0x9e
go.etcd.io/etcd/server/v3/embed.(*Etcd).Close(0xc00015d500)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:438 +0xcf4
go.etcd.io/etcd/server/v3/embed.StartEtcd.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:116 +0x109
go.etcd.io/etcd/server/v3/embed.StartEtcd(0xc000032000)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:132 +0xa0c
go.etcd.io/etcd/server/v3/embed.TestEnableAuth(0xc000103860)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34 +0x1d0
testing.tRunner(0xc000103860, 0x246a9a8)
        /usr/local/opt/go/libexec/src/testing/testing.go:1595 +0x239
created by testing.(*T).Run in goroutine 1
        /usr/local/opt/go/libexec/src/testing/testing.go:1648 +0x82b
FAIL    go.etcd.io/etcd/server/v3/embed 0.860s
ok      go.etcd.io/etcd/server/v3/etcdmain      2.404s
?       go.etcd.io/etcd/server/v3/etcdserver/api/v2stats        [no test files]
?       go.etcd.io/etcd/server/v3/etcdserver/api/v2v3   [no test files]
?       go.etcd.io/etcd/server/v3/etcdserver/api/v3alarm        [no test files]
?       go.etcd.io/etcd/server/v3/etcdserver/api/v3client       [no test files]
?       go.etcd.io/etcd/server/v3/etcdserver/api/v3election     [no test files]
?       go.etcd.io/etcd/server/v3/etcdserver/api/v3election/v3electionpb        [no test files]
?       go.etcd.io/etcd/server/v3/etcdserver/api/v3election/v3electionpb/gw     [no test files]
?       go.etcd.io/etcd/server/v3/etcdserver/api/v3lock [no test files]
?       go.etcd.io/etcd/server/v3/etcdserver/api/v3lock/v3lockpb        [no test files]
?       go.etcd.io/etcd/server/v3/etcdserver/api/v3lock/v3lockpb/gw     [no test files]
?       go.etcd.io/etcd/server/v3/lease/leasepb [no test files]
?       go.etcd.io/etcd/server/v3/mock/mockstorage      [no test files]
?       go.etcd.io/etcd/server/v3/mock/mockstore        [no test files]
?       go.etcd.io/etcd/server/v3/mock/mockwait [no test files]
?       go.etcd.io/etcd/server/v3/mvcc/backend/testing  [no test files]
?       go.etcd.io/etcd/server/v3/mvcc/buckets  [no test files]
?       go.etcd.io/etcd/server/v3/proxy/grpcproxy       [no test files]
?       go.etcd.io/etcd/server/v3/proxy/grpcproxy/adapter       [no test files]
?       go.etcd.io/etcd/server/v3/proxy/grpcproxy/cache [no test files]
?       go.etcd.io/etcd/server/v3/storage/mvcc/testutil [no test files]
?       go.etcd.io/etcd/server/v3/verify        [no test files]
ok      go.etcd.io/etcd/server/v3/etcdserver    20.822s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/etcdhttp       9.490s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/membership     18.006s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp       3.614s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/snap   4.396s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2auth 2.284s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2discovery    1.964s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2error        2.203s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2http 1.741s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2http/httptypes       2.170s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2store        1.984s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v3compactor    3.325s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc  1.624s
ok      go.etcd.io/etcd/server/v3/etcdserver/cindex     4.057s
ok      go.etcd.io/etcd/server/v3/lease 14.430s
ok      go.etcd.io/etcd/server/v3/lease/leasehttp       6.165s
{"level":"info","msg":"compact tree index","revision":1}
{"level":"info","msg":"compact tree index","revision":2}
{"level":"info","msg":"compact tree index","revision":3}
{"level":"info","msg":"compact tree index","revision":4}
{"level":"info","msg":"compact tree index","revision":5}
{"level":"info","msg":"compact tree index","revision":6}
{"level":"info","msg":"compact tree index","revision":7}
{"level":"info","msg":"compact tree index","revision":8}
{"level":"info","msg":"compact tree index","revision":9}
{"level":"info","msg":"compact tree index","revision":10}
{"level":"info","msg":"compact tree index","revision":11}
{"level":"info","msg":"compact tree index","revision":12}
{"level":"info","msg":"compact tree index","revision":13}
{"level":"info","msg":"compact tree index","revision":14}
{"level":"info","msg":"compact tree index","revision":15}
{"level":"info","msg":"compact tree index","revision":16}
{"level":"info","msg":"compact tree index","revision":17}
{"level":"info","msg":"compact tree index","revision":18}
{"level":"info","msg":"compact tree index","revision":19}
{"level":"info","msg":"compact tree index","revision":1}
{"level":"info","msg":"compact tree index","revision":2}
{"level":"info","msg":"compact tree index","revision":3}
{"level":"info","msg":"compact tree index","revision":4}
{"level":"info","msg":"compact tree index","revision":5}
{"level":"info","msg":"compact tree index","revision":6}
{"level":"info","msg":"compact tree index","revision":7}
{"level":"info","msg":"compact tree index","revision":8}
{"level":"info","msg":"compact tree index","revision":9}
{"level":"info","msg":"compact tree index","revision":10}
{"level":"info","msg":"compact tree index","revision":11}
{"level":"info","msg":"compact tree index","revision":12}
{"level":"info","msg":"compact tree index","revision":13}
{"level":"info","msg":"compact tree index","revision":14}
{"level":"info","msg":"compact tree index","revision":15}
{"level":"info","msg":"compact tree index","revision":16}
{"level":"info","msg":"compact tree index","revision":17}
{"level":"info","msg":"compact tree index","revision":18}
{"level":"info","msg":"compact tree index","revision":19}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":4}
{"level":"info","msg":"finished scheduled compaction","compact-revision":4,"took":"249.5µs","hash":1500391675}
{"level":"info","msg":"storing new hash","hash":1500391675,"revision":4,"compact-revision":-1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":4}
{"level":"info","msg":"finished scheduled compaction","compact-revision":4,"took":"146.627µs","hash":1500391675}
{"level":"info","msg":"storing new hash","hash":1500391675,"revision":4,"compact-revision":-1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":1}
{"level":"info","msg":"finished scheduled compaction","compact-revision":1,"took":"47.163297ms","hash":1084519789}
{"level":"info","msg":"storing new hash","hash":1084519789,"revision":1,"compact-revision":-1}
{"level":"info","msg":"compact tree index","revision":2}
{"level":"info","msg":"finished scheduled compaction","compact-revision":2,"took":"133.565619ms","hash":551221740}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":3}
{"level":"info","msg":"finished scheduled compaction","compact-revision":3,"took":"64.641055ms","hash":4023298219}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":0}
{"level":"info","msg":"finished scheduled compaction","compact-revision":0,"took":"134.143906ms","hash":1084519789}
{"level":"info","msg":"storing new hash","hash":1084519789,"revision":0,"compact-revision":-1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":5}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":4}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":1}
{"level":"info","msg":"finished scheduled compaction","compact-revision":1,"took":"124.635µs","hash":1084519789}
{"level":"info","msg":"storing new hash","hash":1084519789,"revision":1,"compact-revision":-1}
{"level":"info","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":1}
{"level":"info","msg":"kvstore restored","current-rev":3}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":4}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":5}
{"level":"info","msg":"finished scheduled compaction","compact-revision":5,"took":"184.723µs","hash":873998276}
{"level":"info","msg":"storing new hash","hash":873998276,"revision":5,"compact-revision":-1}
{"level":"info","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":5}
{"level":"info","msg":"kvstore restored","current-rev":5}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"finished scheduled compaction","compact-revision":3,"took":"16.365µs","hash":3457566959}
{"level":"info","msg":"storing new hash","hash":3457566959,"revision":3,"compact-revision":-1}
{"level":"info","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":3}
{"level":"info","msg":"kvstore restored","current-rev":5}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":34}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":4}
{"level":"info","msg":"resume scheduled compaction","meta-bucket-name":"meta","meta-bucket-name-key":"scheduledCompactRev","scheduled-compact-revision":2}
{"level":"info","msg":"compact tree index","revision":2}
{"level":"info","msg":"finished scheduled compaction","compact-revision":2,"took":"78.393µs","hash":3305255506}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":9900}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9900,"took":"1.087569996s","hash":2982592828}
{"level":"info","msg":"storing new hash","hash":2982592828,"revision":9900,"compact-revision":-1}
{"level":"info","msg":"compact tree index","revision":9901}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9901,"took":"1.001155ms","hash":4092902393}
{"level":"info","msg":"storing new hash","hash":4092902393,"revision":9901,"compact-revision":9900}
{"level":"info","msg":"compact tree index","revision":9902}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9902,"took":"866.145µs","hash":3442946395}
{"level":"info","msg":"storing new hash","hash":3442946395,"revision":9902,"compact-revision":9901}
{"level":"info","msg":"compact tree index","revision":9903}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9903,"took":"1.014285ms","hash":3934476647}
{"level":"info","msg":"storing new hash","hash":3934476647,"revision":9903,"compact-revision":9902}
{"level":"info","msg":"compact tree index","revision":9904}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9904,"took":"846.096µs","hash":1979317650}
{"level":"info","msg":"storing new hash","hash":1979317650,"revision":9904,"compact-revision":9903}
{"level":"info","msg":"compact tree index","revision":9905}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9905,"took":"843.178µs","hash":4126965741}
{"level":"info","msg":"storing new hash","hash":4126965741,"revision":9905,"compact-revision":9904}
{"level":"info","msg":"compact tree index","revision":9906}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9906,"took":"669.685µs","hash":3409966415}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9907}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9907,"took":"1.079197ms","hash":3968572787}
{"level":"info","msg":"storing new hash","hash":3968572787,"revision":9907,"compact-revision":9906}
{"level":"info","msg":"compact tree index","revision":9908}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9908,"took":"1.845188ms","hash":1013037252}
{"level":"info","msg":"storing new hash","hash":1013037252,"revision":9908,"compact-revision":9907}
{"level":"info","msg":"compact tree index","revision":9909}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9909,"took":"1.798644ms","hash":1305141790}
{"level":"info","msg":"storing new hash","hash":1305141790,"revision":9909,"compact-revision":9908}
{"level":"info","msg":"compact tree index","revision":9910}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9910,"took":"1.604622ms","hash":189389043}
{"level":"info","msg":"storing new hash","hash":189389043,"revision":9910,"compact-revision":9909}
{"level":"info","msg":"compact tree index","revision":9911}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9911,"took":"1.711235ms","hash":754823375}
{"level":"info","msg":"storing new hash","hash":754823375,"revision":9911,"compact-revision":9910}
{"level":"info","msg":"compact tree index","revision":9912}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9912,"took":"1.742712ms","hash":4067225043}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9913}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9913,"took":"1.772885ms","hash":804901573}
{"level":"info","msg":"storing new hash","hash":804901573,"revision":9913,"compact-revision":9912}
{"level":"info","msg":"compact tree index","revision":9914}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9914,"took":"1.819888ms","hash":3600863096}
{"level":"info","msg":"storing new hash","hash":3600863096,"revision":9914,"compact-revision":9913}
{"level":"info","msg":"compact tree index","revision":9915}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9915,"took":"1.994351ms","hash":4044662596}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9916}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9916,"took":"1.658234ms","hash":1824345379}
{"level":"info","msg":"storing new hash","hash":1824345379,"revision":9916,"compact-revision":9915}
{"level":"info","msg":"compact tree index","revision":9917}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9917,"took":"1.935756ms","hash":487987193}
{"level":"info","msg":"storing new hash","hash":487987193,"revision":9917,"compact-revision":9916}
{"level":"info","msg":"compact tree index","revision":9918}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9918,"took":"2.117754ms","hash":1855345291}
{"level":"info","msg":"storing new hash","hash":1855345291,"revision":9918,"compact-revision":9917}
{"level":"info","msg":"compact tree index","revision":9919}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9919,"took":"1.618954ms","hash":1226980023}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9920}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9920,"took":"1.993621ms","hash":1439986907}
{"level":"info","msg":"storing new hash","hash":1439986907,"revision":9920,"compact-revision":9919}
{"level":"info","msg":"compact tree index","revision":9921}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9921,"took":"1.832831ms","hash":492703523}
{"level":"info","msg":"storing new hash","hash":492703523,"revision":9921,"compact-revision":9920}
{"level":"info","msg":"compact tree index","revision":9922}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9922,"took":"1.611261ms","hash":597525889}
{"level":"info","msg":"storing new hash","hash":597525889,"revision":9922,"compact-revision":9921}
{"level":"info","msg":"compact tree index","revision":9923}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9923,"took":"1.69157ms","hash":69856701}
{"level":"info","msg":"storing new hash","hash":69856701,"revision":9923,"compact-revision":9922}
{"level":"info","msg":"compact tree index","revision":9924}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9924,"took":"1.644955ms","hash":1578799301}
{"level":"info","msg":"storing new hash","hash":1578799301,"revision":9924,"compact-revision":9923}
{"level":"info","msg":"compact tree index","revision":9925}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9925,"took":"1.594706ms","hash":800180767}
{"level":"info","msg":"storing new hash","hash":800180767,"revision":9925,"compact-revision":9924}
{"level":"info","msg":"compact tree index","revision":9926}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9926,"took":"1.615565ms","hash":292700349}
{"level":"info","msg":"storing new hash","hash":292700349,"revision":9926,"compact-revision":9925}
{"level":"info","msg":"compact tree index","revision":9927}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9927,"took":"1.802856ms","hash":918956161}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9928}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9928,"took":"1.558381ms","hash":2769599053}
{"level":"info","msg":"storing new hash","hash":2769599053,"revision":9928,"compact-revision":9927}
{"level":"info","msg":"compact tree index","revision":9929}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9929,"took":"1.820467ms","hash":2021754203}
{"level":"info","msg":"storing new hash","hash":2021754203,"revision":9929,"compact-revision":9928}
{"level":"info","msg":"compact tree index","revision":9930}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9930,"took":"2.038145ms","hash":1040367542}
{"level":"info","msg":"storing new hash","hash":1040367542,"revision":9930,"compact-revision":9929}
{"level":"info","msg":"compact tree index","revision":9931}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9931,"took":"1.129857ms","hash":431413130}
{"level":"info","msg":"storing new hash","hash":431413130,"revision":9931,"compact-revision":9930}
{"level":"info","msg":"compact tree index","revision":9932}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9932,"took":"1.45718ms","hash":2216639981}
{"level":"info","msg":"storing new hash","hash":2216639981,"revision":9932,"compact-revision":9931}
{"level":"info","msg":"compact tree index","revision":9933}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9933,"took":"1.313208ms","hash":4122240823}
{"level":"info","msg":"storing new hash","hash":4122240823,"revision":9933,"compact-revision":9932}
{"level":"info","msg":"compact tree index","revision":9934}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9934,"took":"1.469447ms","hash":216889994}
{"level":"info","msg":"storing new hash","hash":216889994,"revision":9934,"compact-revision":9933}
{"level":"info","msg":"compact tree index","revision":9935}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9935,"took":"1.470912ms","hash":727257782}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9936}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9936,"took":"1.591057ms","hash":1941512540}
{"level":"info","msg":"storing new hash","hash":1941512540,"revision":9936,"compact-revision":9935}
{"level":"info","msg":"compact tree index","revision":9937}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9937,"took":"1.579254ms","hash":4089226019}
{"level":"info","msg":"storing new hash","hash":4089226019,"revision":9937,"compact-revision":9936}
{"level":"info","msg":"compact tree index","revision":9938}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9938,"took":"1.496778ms","hash":2151474769}
{"level":"info","msg":"storing new hash","hash":2151474769,"revision":9938,"compact-revision":9937}
{"level":"info","msg":"compact tree index","revision":9939}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9939,"took":"1.87529ms","hash":2810744429}
{"level":"info","msg":"storing new hash","hash":2810744429,"revision":9939,"compact-revision":9938}
{"level":"info","msg":"compact tree index","revision":9940}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9940,"took":"1.612337ms","hash":732857719}
{"level":"info","msg":"storing new hash","hash":732857719,"revision":9940,"compact-revision":9939}
{"level":"info","msg":"compact tree index","revision":9941}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9941,"took":"1.523768ms","hash":1510334381}
{"level":"info","msg":"storing new hash","hash":1510334381,"revision":9941,"compact-revision":9940}
{"level":"info","msg":"compact tree index","revision":9942}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9942,"took":"1.474902ms","hash":1690719503}
{"level":"info","msg":"storing new hash","hash":1690719503,"revision":9942,"compact-revision":9941}
{"level":"info","msg":"compact tree index","revision":9943}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9943,"took":"1.522196ms","hash":1131560243}
{"level":"info","msg":"storing new hash","hash":1131560243,"revision":9943,"compact-revision":9942}
{"level":"info","msg":"compact tree index","revision":9944}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9944,"took":"1.467108ms","hash":1517942576}
{"level":"info","msg":"storing new hash","hash":1517942576,"revision":9944,"compact-revision":9943}
{"level":"info","msg":"compact tree index","revision":9945}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9945,"took":"2.013407ms","hash":2280441894}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9946}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9946,"took":"1.637524ms","hash":3106882180}
{"level":"info","msg":"storing new hash","hash":3106882180,"revision":9946,"compact-revision":9945}
{"level":"info","msg":"compact tree index","revision":9947}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9947,"took":"1.447503ms","hash":2660969144}
{"level":"info","msg":"storing new hash","hash":2660969144,"revision":9947,"compact-revision":9946}
{"level":"info","msg":"compact tree index","revision":9948}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9948,"took":"1.23149ms","hash":1316032271}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9949}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9949,"took":"1.378601ms","hash":1071273429}
{"level":"info","msg":"storing new hash","hash":1071273429,"revision":9949,"compact-revision":9948}
{"level":"info","msg":"compact tree index","revision":9950}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9950,"took":"1.740325ms","hash":2035904312}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9951}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9951,"took":"906.416µs","hash":1592592132}
{"level":"info","msg":"storing new hash","hash":1592592132,"revision":9951,"compact-revision":9950}
{"level":"info","msg":"compact tree index","revision":9952}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9952,"took":"1.069281ms","hash":251895795}
{"level":"info","msg":"storing new hash","hash":251895795,"revision":9952,"compact-revision":9951}
{"level":"info","msg":"compact tree index","revision":9953}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9953,"took":"1.435635ms","hash":874997598}
{"level":"info","msg":"storing new hash","hash":874997598,"revision":9953,"compact-revision":9952}
{"level":"info","msg":"compact tree index","revision":9954}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9954,"took":"1.160752ms","hash":3447618275}
{"level":"info","msg":"storing new hash","hash":3447618275,"revision":9954,"compact-revision":9953}
{"level":"info","msg":"compact tree index","revision":9955}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9955,"took":"1.323951ms","hash":3939111647}
{"level":"info","msg":"storing new hash","hash":3939111647,"revision":9955,"compact-revision":9954}
{"level":"info","msg":"compact tree index","revision":9956}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9956,"took":"813.279µs","hash":2003040440}
{"level":"info","msg":"storing new hash","hash":2003040440,"revision":9956,"compact-revision":9955}
{"level":"info","msg":"compact tree index","revision":9957}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9957,"took":"961.277µs","hash":113812066}
{"level":"info","msg":"storing new hash","hash":113812066,"revision":9957,"compact-revision":9956}
{"level":"info","msg":"compact tree index","revision":9958}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9958,"took":"715.997µs","hash":1967712016}
{"level":"info","msg":"storing new hash","hash":1967712016,"revision":9958,"compact-revision":9957}
{"level":"info","msg":"compact tree index","revision":9959}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9959,"took":"1.185909ms","hash":1392283436}
{"level":"info","msg":"storing new hash","hash":1392283436,"revision":9959,"compact-revision":9958}
{"level":"info","msg":"compact tree index","revision":9960}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9960,"took":"1.227898ms","hash":3546829457}
{"level":"info","msg":"storing new hash","hash":3546829457,"revision":9960,"compact-revision":9959}
{"level":"info","msg":"compact tree index","revision":9961}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9961,"took":"1.200927ms","hash":251489671}
{"level":"info","msg":"storing new hash","hash":251489671,"revision":9961,"compact-revision":9960}
{"level":"info","msg":"compact tree index","revision":9962}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9962,"took":"1.197038ms","hash":809428773}
{"level":"info","msg":"storing new hash","hash":809428773,"revision":9962,"compact-revision":9961}
{"level":"info","msg":"compact tree index","revision":9963}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9963,"took":"1.213086ms","hash":394956569}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9964}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9964,"took":"895.151µs","hash":1304031841}
{"level":"info","msg":"storing new hash","hash":1304031841,"revision":9964,"compact-revision":9963}
{"level":"info","msg":"compact tree index","revision":9965}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9965,"took":"1.303672ms","hash":1007855803}
{"level":"info","msg":"storing new hash","hash":1007855803,"revision":9965,"compact-revision":9964}
{"level":"info","msg":"compact tree index","revision":9966}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9966,"took":"1.032805ms","hash":47260185}
{"level":"info","msg":"storing new hash","hash":47260185,"revision":9966,"compact-revision":9965}
{"level":"info","msg":"compact tree index","revision":9967}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9967,"took":"1.037608ms","hash":627395109}
{"level":"info","msg":"storing new hash","hash":627395109,"revision":9967,"compact-revision":9966}
{"level":"info","msg":"compact tree index","revision":9968}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9968,"took":"1.31541ms","hash":818329119}
{"level":"info","msg":"storing new hash","hash":818329119,"revision":9968,"compact-revision":9967}
{"level":"info","msg":"compact tree index","revision":9969}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9969,"took":"873.912µs","hash":2965582944}
{"level":"info","msg":"storing new hash","hash":2965582944,"revision":9969,"compact-revision":9968}
{"level":"info","msg":"compact tree index","revision":9970}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9970,"took":"690.314µs","hash":4131393165}
{"level":"info","msg":"storing new hash","hash":4131393165,"revision":9970,"compact-revision":9969}
{"level":"info","msg":"compact tree index","revision":9971}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9971,"took":"831.334µs","hash":3522455217}
{"level":"info","msg":"storing new hash","hash":3522455217,"revision":9971,"compact-revision":9970}
{"level":"info","msg":"compact tree index","revision":9972}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9972,"took":"1.274997ms","hash":1281199318}
{"level":"info","msg":"storing new hash","hash":1281199318,"revision":9972,"compact-revision":9971}
{"level":"info","msg":"compact tree index","revision":9973}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9973,"took":"1.011427ms","hash":1039586828}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9974}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9974,"took":"950.325µs","hash":3299855281}
{"level":"info","msg":"storing new hash","hash":3299855281,"revision":9974,"compact-revision":9973}
{"level":"info","msg":"compact tree index","revision":9975}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9975,"took":"1.209847ms","hash":3810239373}
{"level":"info","msg":"storing new hash","hash":3810239373,"revision":9975,"compact-revision":9974}
{"level":"info","msg":"compact tree index","revision":9976}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9976,"took":"1.08013ms","hash":1032519313}
{"level":"info","msg":"storing new hash","hash":1032519313,"revision":9976,"compact-revision":9975}
{"level":"info","msg":"compact tree index","revision":9977}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9977,"took":"1.19497ms","hash":3760177543}
{"level":"info","msg":"storing new hash","hash":3760177543,"revision":9977,"compact-revision":9976}
{"level":"info","msg":"compact tree index","revision":9978}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9978,"took":"1.022152ms","hash":2476705013}
{"level":"info","msg":"storing new hash","hash":2476705013,"revision":9978,"compact-revision":9977}
{"level":"info","msg":"compact tree index","revision":9979}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9979,"took":"1.383196ms","hash":3022777545}
{"level":"info","msg":"storing new hash","hash":3022777545,"revision":9979,"compact-revision":9978}
{"level":"info","msg":"compact tree index","revision":9980}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9980,"took":"920.158µs","hash":2769888363}
{"level":"info","msg":"storing new hash","hash":2769888363,"revision":9980,"compact-revision":9979}
{"level":"info","msg":"compact tree index","revision":9981}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9981,"took":"882.167µs","hash":3568467633}
{"level":"info","msg":"storing new hash","hash":3568467633,"revision":9981,"compact-revision":9980}
{"level":"info","msg":"compact tree index","revision":9982}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9982,"took":"461.501µs","hash":3933336595}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9983}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9983,"took":"1.051723ms","hash":3452279855}
{"level":"info","msg":"storing new hash","hash":3452279855,"revision":9983,"compact-revision":9982}
{"level":"info","msg":"compact tree index","revision":9984}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9984,"took":"677.4µs","hash":4232457539}
{"level":"info","msg":"storing new hash","hash":4232457539,"revision":9984,"compact-revision":9983}
{"level":"info","msg":"compact tree index","revision":9985}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9985,"took":"975.347µs","hash":3973754021}
{"level":"info","msg":"storing new hash","hash":3973754021,"revision":9985,"compact-revision":9984}
{"level":"info","msg":"compact tree index","revision":9986}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9986,"took":"1.348855ms","hash":3524863495}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9987}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9987,"took":"851.243µs","hash":4121775675}
{"level":"info","msg":"storing new hash","hash":4121775675,"revision":9987,"compact-revision":9986}
{"level":"info","msg":"compact tree index","revision":9988}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9988,"took":"799.211µs","hash":625437580}
{"level":"info","msg":"storing new hash","hash":625437580,"revision":9988,"compact-revision":9987}
{"level":"info","msg":"compact tree index","revision":9989}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9989,"took":"691.066µs","hash":1424765270}
{"level":"info","msg":"storing new hash","hash":1424765270,"revision":9989,"compact-revision":9988}
{"level":"info","msg":"compact tree index","revision":9990}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9990,"took":"1.042711ms","hash":309267387}
{"level":"info","msg":"storing new hash","hash":309267387,"revision":9990,"compact-revision":9989}
{"level":"info","msg":"compact tree index","revision":9991}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9991,"took":"1.044829ms","hash":903570311}
{"level":"info","msg":"storing new hash","hash":903570311,"revision":9991,"compact-revision":9990}
{"level":"info","msg":"compact tree index","revision":9992}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9992,"took":"1.013515ms","hash":3947511451}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":9993}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9993,"took":"605.93µs","hash":920584589}
{"level":"info","msg":"storing new hash","hash":920584589,"revision":9993,"compact-revision":9992}
{"level":"info","msg":"compact tree index","revision":9994}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9994,"took":"630.133µs","hash":3481672752}
{"level":"info","msg":"storing new hash","hash":3481672752,"revision":9994,"compact-revision":9993}
{"level":"info","msg":"compact tree index","revision":9995}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9995,"took":"470.804µs","hash":3895620620}
{"level":"info","msg":"storing new hash","hash":3895620620,"revision":9995,"compact-revision":9994}
{"level":"info","msg":"compact tree index","revision":9996}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9996,"took":"460.04µs","hash":1973132907}
{"level":"info","msg":"storing new hash","hash":1973132907,"revision":9996,"compact-revision":9995}
{"level":"info","msg":"compact tree index","revision":9997}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9997,"took":"791.884µs","hash":70304945}
{"level":"info","msg":"storing new hash","hash":70304945,"revision":9997,"compact-revision":9996}
{"level":"info","msg":"compact tree index","revision":9998}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9998,"took":"624.542µs","hash":2008056259}
{"level":"info","msg":"storing new hash","hash":2008056259,"revision":9998,"compact-revision":9997}
{"level":"info","msg":"compact tree index","revision":9999}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9999,"took":"1.052918ms","hash":1342499327}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x16535e7]

goroutine 445 [running]:
go.etcd.io/bbolt.(*Tx).Bucket(...)
        /Users/kosovgrigorii/go/pkg/mod/go.etcd.io/bbolt@v1.3.8/tx.go:102
go.etcd.io/etcd/server/v3/mvcc/backend.unsafeForEach(0x0, {0x1d28de8, 0x22417e0}, 0x2000?)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/batch_tx.go:235 +0x47
go.etcd.io/etcd/server/v3/mvcc/backend.(*baseReadTx).UnsafeForEach(0xc000584230, {0x1d28de8, 0x22417e0}, 0xc000344230)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/read_tx.go:69 +0x199
go.etcd.io/etcd/server/v3/mvcc.unsafeHashByRev({0x1d2a338, 0xc000584230}, 0xc000549f00?, 0xc000549f00?, 0xc00046f6c0?)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/hash.go:34 +0x1b5
go.etcd.io/etcd/server/v3/mvcc.(*store).hashByRev(0xc0000a51e0, 0x2710)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/kvstore.go:194 +0x385
go.etcd.io/etcd/server/v3/mvcc.(*hashStorage).HashByRev(0xc000236140, 0x2710)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/hash.go:133 +0x1d2
go.etcd.io/etcd/server/v3/mvcc.TestHashKVWhenCompacting.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/kvstore_test.go:561 +0x108
created by go.etcd.io/etcd/server/v3/mvcc.TestHashKVWhenCompacting in goroutine 471
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/kvstore_test.go:558 +0x3be
FAIL    go.etcd.io/etcd/server/v3/mvcc  32.338s
ok      go.etcd.io/etcd/server/v3/mvcc/backend  10.996s
ok      go.etcd.io/etcd/server/v3/proxy/httpproxy       2.252s
ok      go.etcd.io/etcd/server/v3/proxy/tcpproxy        2.506s
ok      go.etcd.io/etcd/server/v3/wal   34.655s
ok      go.etcd.io/etcd/server/v3/wal/walpb     2.422s
FAIL
FAIL: (code:1):
  % (cd server && 'env' 'go' 'test' '-short' '-timeout=3m' '--race=true' '--cpu=4' './...')
FAIL: 'unit' failed at Wed Jan 24 16:43:48 +05 2024

release-3.5:

Verification failed & bbolt panicking
{"level":"warn","ts":"2024-01-24T16:45:30.353272+0500","caller":"embed/config.go:679","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2024-01-24T16:45:30.353492+0500","caller":"embed/etcd.go:127","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":"2024-01-24T16:45:30.355412+0500","caller":"embed/etcd.go:375","msg":"closing etcd server","name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test970702714","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]}
{"level":"info","ts":"2024-01-24T16:45:30.355476+0500","caller":"embed/etcd.go:377","msg":"closed etcd server","name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test970702714","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]}
{"level":"info","ts":"2024-01-24T16:45:30.355535+0500","caller":"verify/verify.go:57","msg":"verification of persisted state","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test970702714"}
{"level":"panic","ts":"2024-01-24T16:45:30.355641+0500","caller":"backend/backend.go:189","msg":"failed to open database","path":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test970702714/member/snap/db","error":"open /var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test970702714/member/snap/db: no such file or directory","stacktrace":"go.etcd.io/etcd/server/v3/mvcc/backend.newBackend\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189\ngo.etcd.io/etcd/server/v3/mvcc/backend.New\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163\ngo.etcd.io/etcd/server/v3/verify.Verify\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76\ngo.etcd.io/etcd/server/v3/verify.VerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94\ngo.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:378\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:438\ngo.etcd.io/etcd/server/v3/embed.StartEtcd.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:116\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:132\ngo.etcd.io/etcd/server/v3/embed.TestEnableAuth\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34\ntesting.tRunner\n\t/usr/local/opt/go/libexec/src/testing/testing.go:1595"}
{"level":"error","ts":"2024-01-24T16:45:30.355821+0500","caller":"verify/verify.go:64","msg":"verification of persisted state failed","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test970702714","stacktrace":"go.etcd.io/etcd/server/v3/verify.Verify.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:64\nruntime.gopanic\n\t/usr/local/opt/go/libexec/src/runtime/panic.go:920\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write\n\t/Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/zapcore/entry.go:234\ngo.uber.org/zap.(*Logger).Panic\n\t/Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/logger.go:227\ngo.etcd.io/etcd/server/v3/mvcc/backend.newBackend\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189\ngo.etcd.io/etcd/server/v3/mvcc/backend.New\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163\ngo.etcd.io/etcd/server/v3/verify.Verify\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76\ngo.etcd.io/etcd/server/v3/verify.VerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94\ngo.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:378\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:438\ngo.etcd.io/etcd/server/v3/embed.StartEtcd.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:116\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:132\ngo.etcd.io/etcd/server/v3/embed.TestEnableAuth\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34\ntesting.tRunner\n\t/usr/local/opt/go/libexec/src/testing/testing.go:1595"}
--- FAIL: TestEnableAuth (0.00s)
panic: failed to open database [recovered]
        panic: failed to open database [recovered]
        panic: failed to open database

goroutine 134 [running]:
testing.tRunner.func1.2({0x221abc0, 0xc0002e8bb0})
        /usr/local/opt/go/libexec/src/testing/testing.go:1545 +0x366
testing.tRunner.func1()
        /usr/local/opt/go/libexec/src/testing/testing.go:1548 +0x630
panic({0x221abc0?, 0xc0002e8bb0?})
        /usr/local/opt/go/libexec/src/runtime/panic.go:920 +0x270
go.etcd.io/etcd/server/v3/verify.Verify.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:66 +0x505
panic({0x221abc0?, 0xc0002e8bb0?})
        /usr/local/opt/go/libexec/src/runtime/panic.go:920 +0x270
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc000452240, {0xc00018a580, 0x2, 0x2})
        /Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/zapcore/entry.go:234 +0x61b
go.uber.org/zap.(*Logger).Panic(0xc00039a360?, {0x2412259, 0x17}, {0xc00018a580, 0x2, 0x2})
        /Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/logger.go:227 +0x68
go.etcd.io/etcd/server/v3/mvcc/backend.newBackend({{0xc00039a360, 0x52}, 0x5f5e100, 0x2710, {0x0, 0x0}, 0x280000000, 0xc0003d07d0, 0x0, 0x0, ...})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189 +0x6b3
go.etcd.io/etcd/server/v3/mvcc/backend.New(...)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163
go.etcd.io/etcd/server/v3/verify.Verify({{0xc00004a1e0?, 0xb?}, 0x20?, 0xc0003d07d0?})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76 +0x478
go.etcd.io/etcd/server/v3/verify.VerifyIfEnabled({{0xc00004a1e0?, 0xc000317100?}, 0x48?, 0xc0003d07d0?})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94 +0x76
go.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled({{0xc00004a1e0?, 0x240aaa0?}, 0x12?, 0xc0003d07d0?})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103 +0x66
go.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:378 +0x9e
go.etcd.io/etcd/server/v3/embed.(*Etcd).Close(0xc0003c0e00)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:438 +0xcf4
go.etcd.io/etcd/server/v3/embed.StartEtcd.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:116 +0x109
go.etcd.io/etcd/server/v3/embed.StartEtcd(0xc0003ee000)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:132 +0xa0c
go.etcd.io/etcd/server/v3/embed.TestEnableAuth(0xc000366340)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34 +0x1d0
testing.tRunner(0xc000366340, 0x246b688)
        /usr/local/opt/go/libexec/src/testing/testing.go:1595 +0x239
created by testing.(*T).Run in goroutine 1
        /usr/local/opt/go/libexec/src/testing/testing.go:1648 +0x82b
FAIL    go.etcd.io/etcd/server/v3/embed 0.875s
ok      go.etcd.io/etcd/server/v3/etcdmain      2.814s
?       go.etcd.io/etcd/server/v3/lease/leasepb [no test files]
?       go.etcd.io/etcd/server/v3/mock/mockwait [no test files]
?       go.etcd.io/etcd/server/v3/mock/mockstorage      [no test files]
?       go.etcd.io/etcd/server/v3/mock/mockstore        [no test files]
?       go.etcd.io/etcd/server/v3/mvcc/backend/testing  [no test files]
?       go.etcd.io/etcd/server/v3/mvcc/buckets  [no test files]
?       go.etcd.io/etcd/server/v3/proxy/grpcproxy       [no test files]
?       go.etcd.io/etcd/server/v3/proxy/grpcproxy/adapter       [no test files]
?       go.etcd.io/etcd/server/v3/proxy/grpcproxy/cache [no test files]
?       go.etcd.io/etcd/server/v3/storage/mvcc/testutil [no test files]
?       go.etcd.io/etcd/server/v3/verify        [no test files]
ok      go.etcd.io/etcd/server/v3/etcdserver    24.060s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/etcdhttp       9.511s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/membership     17.185s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp       3.762s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/snap   4.925s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2auth 3.308s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2discovery    2.776s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2error        3.019s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2http 1.897s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2http/httptypes       2.199s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2store        2.019s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v3compactor    3.474s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc  1.932s
ok      go.etcd.io/etcd/server/v3/etcdserver/cindex     3.089s
ok      go.etcd.io/etcd/server/v3/lease 14.878s
ok      go.etcd.io/etcd/server/v3/lease/leasehttp       6.698s
{"level":"info","msg":"compact tree index","revision":1}
{"level":"info","msg":"compact tree index","revision":2}
{"level":"info","msg":"compact tree index","revision":3}
{"level":"info","msg":"compact tree index","revision":4}
{"level":"info","msg":"compact tree index","revision":5}
{"level":"info","msg":"compact tree index","revision":6}
{"level":"info","msg":"compact tree index","revision":7}
{"level":"info","msg":"compact tree index","revision":8}
{"level":"info","msg":"compact tree index","revision":9}
{"level":"info","msg":"compact tree index","revision":10}
{"level":"info","msg":"compact tree index","revision":11}
{"level":"info","msg":"compact tree index","revision":12}
{"level":"info","msg":"compact tree index","revision":13}
{"level":"info","msg":"compact tree index","revision":14}
{"level":"info","msg":"compact tree index","revision":15}
{"level":"info","msg":"compact tree index","revision":16}
{"level":"info","msg":"compact tree index","revision":17}
{"level":"info","msg":"compact tree index","revision":18}
{"level":"info","msg":"compact tree index","revision":19}
{"level":"info","msg":"compact tree index","revision":1}
{"level":"info","msg":"compact tree index","revision":2}
{"level":"info","msg":"compact tree index","revision":3}
{"level":"info","msg":"compact tree index","revision":4}
{"level":"info","msg":"compact tree index","revision":5}
{"level":"info","msg":"compact tree index","revision":6}
{"level":"info","msg":"compact tree index","revision":7}
{"level":"info","msg":"compact tree index","revision":8}
{"level":"info","msg":"compact tree index","revision":9}
{"level":"info","msg":"compact tree index","revision":10}
{"level":"info","msg":"compact tree index","revision":11}
{"level":"info","msg":"compact tree index","revision":12}
{"level":"info","msg":"compact tree index","revision":13}
{"level":"info","msg":"compact tree index","revision":14}
{"level":"info","msg":"compact tree index","revision":15}
{"level":"info","msg":"compact tree index","revision":16}
{"level":"info","msg":"compact tree index","revision":17}
{"level":"info","msg":"compact tree index","revision":18}
{"level":"info","msg":"compact tree index","revision":19}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":4}
{"level":"info","msg":"finished scheduled compaction","compact-revision":4,"took":"307.686µs","hash":1500391675,"current-db-size-bytes":24576,"current-db-size":"25 kB","current-db-size-in-use-bytes":24576,"current-db-size-in-use":"25 kB"}
{"level":"info","msg":"storing new hash","hash":1500391675,"revision":4,"compact-revision":-1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":4}
{"level":"info","msg":"finished scheduled compaction","compact-revision":4,"took":"238.018µs","hash":1500391675,"current-db-size-bytes":24576,"current-db-size":"25 kB","current-db-size-in-use-bytes":24576,"current-db-size-in-use":"25 kB"}
{"level":"info","msg":"storing new hash","hash":1500391675,"revision":4,"compact-revision":-1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":1}
{"level":"info","msg":"finished scheduled compaction","compact-revision":1,"took":"76.884617ms","hash":1084519789,"current-db-size-bytes":24576,"current-db-size":"25 kB","current-db-size-in-use-bytes":24576,"current-db-size-in-use":"25 kB"}
{"level":"info","msg":"storing new hash","hash":1084519789,"revision":1,"compact-revision":-1}
{"level":"info","msg":"compact tree index","revision":2}
{"level":"info","msg":"finished scheduled compaction","compact-revision":2,"took":"37.335388ms","hash":551221740,"current-db-size-bytes":24576,"current-db-size":"25 kB","current-db-size-in-use-bytes":24576,"current-db-size-in-use":"25 kB"}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"compact tree index","revision":3}
{"level":"info","msg":"finished scheduled compaction","compact-revision":3,"took":"114.872869ms","hash":4023298219,"current-db-size-bytes":24576,"current-db-size":"25 kB","current-db-size-in-use-bytes":24576,"current-db-size-in-use":"25 kB"}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":0}
{"level":"info","msg":"finished scheduled compaction","compact-revision":0,"took":"34.387185ms","hash":1084519789,"current-db-size-bytes":24576,"current-db-size":"25 kB","current-db-size-in-use-bytes":24576,"current-db-size-in-use":"25 kB"}
{"level":"info","msg":"storing new hash","hash":1084519789,"revision":0,"compact-revision":-1}
{"level":"info","msg":"compact tree index","revision":1}
{"level":"info","msg":"finished scheduled compaction","compact-revision":1,"took":"78.456126ms","hash":1084519789,"current-db-size-bytes":24576,"current-db-size":"25 kB","current-db-size-in-use-bytes":24576,"current-db-size-in-use":"25 kB"}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":5}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":4}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":1}
{"level":"info","msg":"finished scheduled compaction","compact-revision":1,"took":"74.771µs","hash":1084519789,"current-db-size-bytes":24576,"current-db-size":"25 kB","current-db-size-in-use-bytes":24576,"current-db-size-in-use":"25 kB"}
{"level":"info","msg":"storing new hash","hash":1084519789,"revision":1,"compact-revision":-1}
{"level":"info","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":1}
{"level":"info","msg":"kvstore restored","current-rev":3}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":4}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":5}
{"level":"info","msg":"finished scheduled compaction","compact-revision":5,"took":"36.607363ms","hash":873998276,"current-db-size-bytes":24576,"current-db-size":"25 kB","current-db-size-in-use-bytes":24576,"current-db-size-in-use":"25 kB"}
{"level":"info","msg":"storing new hash","hash":873998276,"revision":5,"compact-revision":-1}
{"level":"info","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":5}
{"level":"info","msg":"kvstore restored","current-rev":5}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"finished scheduled compaction","compact-revision":3,"took":"15.445µs","hash":3457566959,"current-db-size-bytes":0,"current-db-size":"0 B","current-db-size-in-use-bytes":0,"current-db-size-in-use":"0 B"}
{"level":"info","msg":"storing new hash","hash":3457566959,"revision":3,"compact-revision":-1}
{"level":"info","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":3}
{"level":"info","msg":"kvstore restored","current-rev":5}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":34}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"kvstore restored","current-rev":4}
{"level":"info","msg":"resume scheduled compaction","meta-bucket-name":"meta","meta-bucket-name-key":"scheduledCompactRev","scheduled-compact-revision":2}
{"level":"info","msg":"compact tree index","revision":2}
{"level":"info","msg":"finished scheduled compaction","compact-revision":2,"took":"201.184µs","hash":3305255506,"current-db-size-bytes":24576,"current-db-size":"25 kB","current-db-size-in-use-bytes":24576,"current-db-size-in-use":"25 kB"}
{"level":"info","msg":"previous compaction was interrupted, skip storing compaction hash value"}
{"level":"info","msg":"kvstore restored","current-rev":1}
{"level":"info","msg":"compact tree index","revision":9900}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9900,"took":"1.514350632s","hash":2982592828,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":102400,"current-db-size-in-use":"102 kB"}
{"level":"info","msg":"storing new hash","hash":2982592828,"revision":9900,"compact-revision":-1}
{"level":"info","msg":"compact tree index","revision":9901}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9901,"took":"53.745649ms","hash":4092902393,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":4092902393,"revision":9901,"compact-revision":9900}
{"level":"info","msg":"compact tree index","revision":9902}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9902,"took":"70.019383ms","hash":3442946395,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":3442946395,"revision":9902,"compact-revision":9901}
{"level":"info","msg":"compact tree index","revision":9903}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9903,"took":"92.122446ms","hash":3934476647,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":3934476647,"revision":9903,"compact-revision":9902}
{"level":"info","msg":"compact tree index","revision":9904}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9904,"took":"94.924767ms","hash":1979317650,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":1979317650,"revision":9904,"compact-revision":9903}
{"level":"info","msg":"compact tree index","revision":9905}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9905,"took":"69.375825ms","hash":4126965741,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":4126965741,"revision":9905,"compact-revision":9904}
{"level":"info","msg":"compact tree index","revision":9906}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9906,"took":"59.85964ms","hash":3409966415,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":3409966415,"revision":9906,"compact-revision":9905}
{"level":"info","msg":"compact tree index","revision":9907}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9907,"took":"81.279182ms","hash":3968572787,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":53248,"current-db-size-in-use":"53 kB"}
{"level":"info","msg":"storing new hash","hash":3968572787,"revision":9907,"compact-revision":9906}
{"level":"info","msg":"compact tree index","revision":9908}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9908,"took":"109.192853ms","hash":1013037252,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":1013037252,"revision":9908,"compact-revision":9907}
{"level":"info","msg":"compact tree index","revision":9909}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9909,"took":"99.233287ms","hash":1305141790,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":1305141790,"revision":9909,"compact-revision":9908}
{"level":"info","msg":"compact tree index","revision":9910}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9910,"took":"63.675689ms","hash":189389043,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":189389043,"revision":9910,"compact-revision":9909}
{"level":"info","msg":"compact tree index","revision":9911}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9911,"took":"58.514405ms","hash":754823375,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":754823375,"revision":9911,"compact-revision":9910}
{"level":"info","msg":"compact tree index","revision":9912}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9912,"took":"94.865576ms","hash":4067225043,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":4067225043,"revision":9912,"compact-revision":9911}
{"level":"info","msg":"compact tree index","revision":9913}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9913,"took":"67.393257ms","hash":804901573,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":804901573,"revision":9913,"compact-revision":9912}
{"level":"info","msg":"compact tree index","revision":9914}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9914,"took":"66.423723ms","hash":3600863096,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":3600863096,"revision":9914,"compact-revision":9913}
{"level":"info","msg":"compact tree index","revision":9915}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9915,"took":"60.973483ms","hash":4044662596,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":4044662596,"revision":9915,"compact-revision":9914}
{"level":"info","msg":"compact tree index","revision":9916}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9916,"took":"54.445635ms","hash":1824345379,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":1824345379,"revision":9916,"compact-revision":9915}
{"level":"info","msg":"compact tree index","revision":9917}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9917,"took":"74.476932ms","hash":487987193,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":487987193,"revision":9917,"compact-revision":9916}
{"level":"info","msg":"compact tree index","revision":9918}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9918,"took":"60.122598ms","hash":1855345291,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":1855345291,"revision":9918,"compact-revision":9917}
{"level":"info","msg":"compact tree index","revision":9919}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9919,"took":"92.780579ms","hash":1226980023,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":1226980023,"revision":9919,"compact-revision":9918}
{"level":"info","msg":"compact tree index","revision":9920}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9920,"took":"96.670058ms","hash":1439986907,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":1439986907,"revision":9920,"compact-revision":9919}
{"level":"info","msg":"compact tree index","revision":9921}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9921,"took":"47.021718ms","hash":492703523,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":492703523,"revision":9921,"compact-revision":9920}
{"level":"info","msg":"compact tree index","revision":9922}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9922,"took":"81.720207ms","hash":597525889,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":597525889,"revision":9922,"compact-revision":9921}
{"level":"info","msg":"compact tree index","revision":9923}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9923,"took":"126.56173ms","hash":69856701,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":69856701,"revision":9923,"compact-revision":9922}
{"level":"info","msg":"compact tree index","revision":9924}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9924,"took":"63.587122ms","hash":1578799301,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":1578799301,"revision":9924,"compact-revision":9923}
{"level":"info","msg":"compact tree index","revision":9925}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9925,"took":"36.17999ms","hash":800180767,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":49152,"current-db-size-in-use":"49 kB"}
{"level":"info","msg":"storing new hash","hash":800180767,"revision":9925,"compact-revision":9924}
{"level":"info","msg":"compact tree index","revision":9926}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9926,"took":"83.651503ms","hash":292700349,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":292700349,"revision":9926,"compact-revision":9925}
{"level":"info","msg":"compact tree index","revision":9927}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9927,"took":"83.552067ms","hash":918956161,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":918956161,"revision":9927,"compact-revision":9926}
{"level":"info","msg":"compact tree index","revision":9928}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9928,"took":"111.611889ms","hash":2769599053,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":2769599053,"revision":9928,"compact-revision":9927}
{"level":"info","msg":"compact tree index","revision":9929}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9929,"took":"66.125023ms","hash":2021754203,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":2021754203,"revision":9929,"compact-revision":9928}
{"level":"info","msg":"compact tree index","revision":9930}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9930,"took":"64.70667ms","hash":1040367542,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":1040367542,"revision":9930,"compact-revision":9929}
{"level":"info","msg":"compact tree index","revision":9931}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9931,"took":"61.824458ms","hash":431413130,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":431413130,"revision":9931,"compact-revision":9930}
{"level":"info","msg":"compact tree index","revision":9932}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9932,"took":"61.704573ms","hash":2216639981,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":2216639981,"revision":9932,"compact-revision":9931}
{"level":"info","msg":"compact tree index","revision":9933}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9933,"took":"95.017773ms","hash":4122240823,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":4122240823,"revision":9933,"compact-revision":9932}
{"level":"info","msg":"compact tree index","revision":9934}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9934,"took":"93.770454ms","hash":216889994,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":216889994,"revision":9934,"compact-revision":9933}
{"level":"info","msg":"compact tree index","revision":9935}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9935,"took":"96.871731ms","hash":727257782,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":727257782,"revision":9935,"compact-revision":9934}
{"level":"info","msg":"compact tree index","revision":9936}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9936,"took":"62.361562ms","hash":1941512540,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":1941512540,"revision":9936,"compact-revision":9935}
{"level":"info","msg":"compact tree index","revision":9937}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9937,"took":"53.982237ms","hash":4089226019,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":4089226019,"revision":9937,"compact-revision":9936}
{"level":"info","msg":"compact tree index","revision":9938}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9938,"took":"65.98004ms","hash":2151474769,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":2151474769,"revision":9938,"compact-revision":9937}
{"level":"info","msg":"compact tree index","revision":9939}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9939,"took":"33.013425ms","hash":2810744429,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":2810744429,"revision":9939,"compact-revision":9938}
{"level":"info","msg":"compact tree index","revision":9940}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9940,"took":"92.625232ms","hash":732857719,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":732857719,"revision":9940,"compact-revision":9939}
{"level":"info","msg":"compact tree index","revision":9941}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9941,"took":"76.23121ms","hash":1510334381,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":1510334381,"revision":9941,"compact-revision":9940}
{"level":"info","msg":"compact tree index","revision":9942}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9942,"took":"91.72221ms","hash":1690719503,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":1690719503,"revision":9942,"compact-revision":9941}
{"level":"info","msg":"compact tree index","revision":9943}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9943,"took":"92.362497ms","hash":1131560243,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":1131560243,"revision":9943,"compact-revision":9942}
{"level":"info","msg":"compact tree index","revision":9944}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9944,"took":"93.856244ms","hash":1517942576,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":1517942576,"revision":9944,"compact-revision":9943}
{"level":"info","msg":"compact tree index","revision":9945}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9945,"took":"93.540036ms","hash":2280441894,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":2280441894,"revision":9945,"compact-revision":9944}
{"level":"info","msg":"compact tree index","revision":9946}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9946,"took":"95.006086ms","hash":3106882180,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":3106882180,"revision":9946,"compact-revision":9945}
{"level":"info","msg":"compact tree index","revision":9947}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9947,"took":"91.048712ms","hash":2660969144,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":2660969144,"revision":9947,"compact-revision":9946}
{"level":"info","msg":"compact tree index","revision":9948}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9948,"took":"90.475911ms","hash":1316032271,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":1316032271,"revision":9948,"compact-revision":9947}
{"level":"info","msg":"compact tree index","revision":9949}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9949,"took":"93.950045ms","hash":1071273429,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":1071273429,"revision":9949,"compact-revision":9948}
{"level":"info","msg":"compact tree index","revision":9950}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9950,"took":"30.229852ms","hash":2035904312,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":2035904312,"revision":9950,"compact-revision":9949}
{"level":"info","msg":"compact tree index","revision":9951}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9951,"took":"75.781106ms","hash":1592592132,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":1592592132,"revision":9951,"compact-revision":9950}
{"level":"info","msg":"compact tree index","revision":9952}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9952,"took":"34.661147ms","hash":251895795,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":251895795,"revision":9952,"compact-revision":9951}
{"level":"info","msg":"compact tree index","revision":9953}
{"level":"info","msg":"finished scheduled compaction","compact-revision":9953,"took":"63.620636ms","hash":874997598,"current-db-size-bytes":675840,"current-db-size":"676 kB","current-db-size-in-use-bytes":45056,"current-db-size-in-use":"45 kB"}
{"level":"info","msg":"storing new hash","hash":874997598,"revision":9953,"compact-revision":9952}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x16535e7]

goroutine 647 [running]:
go.etcd.io/bbolt.(*Tx).Bucket(...)
        /Users/kosovgrigorii/go/pkg/mod/go.etcd.io/bbolt@v1.3.8/tx.go:102
go.etcd.io/etcd/server/v3/mvcc/backend.unsafeForEach(0x0, {0x1d29988, 0x22427e0}, 0x106cac5?)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/batch_tx.go:235 +0x47
go.etcd.io/etcd/server/v3/mvcc/backend.(*baseReadTx).UnsafeForEach(0xc000196500, {0x1d29988, 0x22427e0}, 0xc000360d00)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/read_tx.go:69 +0x199
go.etcd.io/etcd/server/v3/mvcc.unsafeHashByRev({0x1d2aed8, 0xc000196500}, 0xc000545f00?, 0xc000545f00?, 0xc0000196c0?)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/hash.go:34 +0x1b5
go.etcd.io/etcd/server/v3/mvcc.(*store).hashByRev(0xc0003a91e0, 0x2710)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/kvstore.go:194 +0x385
go.etcd.io/etcd/server/v3/mvcc.(*hashStorage).HashByRev(0xc00002e140, 0x2710)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/hash.go:133 +0x1d2
go.etcd.io/etcd/server/v3/mvcc.TestHashKVWhenCompacting.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/kvstore_test.go:561 +0x108
created by go.etcd.io/etcd/server/v3/mvcc.TestHashKVWhenCompacting in goroutine 577
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/kvstore_test.go:558 +0x3be
FAIL    go.etcd.io/etcd/server/v3/mvcc  37.699s
ok      go.etcd.io/etcd/server/v3/mvcc/backend  13.441s
ok      go.etcd.io/etcd/server/v3/proxy/httpproxy       2.614s
ok      go.etcd.io/etcd/server/v3/proxy/tcpproxy        2.680s
ok      go.etcd.io/etcd/server/v3/wal   38.411s
ok      go.etcd.io/etcd/server/v3/wal/walpb     2.531s
FAIL
FAIL: (code:1):
  % (cd server && 'env' 'go' 'test' '-short' '-timeout=3m' '--race=true' '--cpu=4' './...')
FAIL: 'unit' failed at Wed Jan 24 16:46:28 +05 2024

main:

Panicking
{"level":"warn","ts":"2024-01-24T16:48:41.936655+0500","caller":"embed/config.go:893","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"warn","ts":"2024-01-24T16:48:41.936868+0500","caller":"embed/config.go:995","msg":"it isn't recommended to use default name, please set a value for --name. Note that etcd might run into issue when multiple members have the same default name","name":"default"}
{"level":"info","ts":"2024-01-24T16:48:41.936917+0500","caller":"embed/etcd.go:125","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":"2024-01-24T16:48:41.938777+0500","caller":"embed/etcd.go:393","msg":"closing etcd server","name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/TestEnableAuth1192049644/001","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]}
{"level":"info","ts":"2024-01-24T16:48:41.93883+0500","caller":"embed/etcd.go:395","msg":"closed etcd server","name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/TestEnableAuth1192049644/001","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]}
{"level":"info","ts":"2024-01-24T16:48:41.938916+0500","caller":"verify/verify.go:58","msg":"verification skipped due to non exist db file"}
--- FAIL: TestEnableAuth (0.00s)
    auth_test.go:30: listen tcp 127.0.0.1:2380: bind: address already in use
{"level":"warn","ts":"2024-01-24T16:48:41.947853+0500","caller":"embed/config.go:893","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2024-01-24T16:48:41.947903+0500","caller":"embed/config.go:1096","msg":"get cluster for etcd-server-ssl SRV","service-scheme":"https","service-name":"etcd-server-ssl","server-name":"1.example.com","discovery-srv":"example.com","advertise-peer-urls":["http://localhost:2380"],"found-cluster":[],"error":"lookup _etcd-server-ssl._tcp.example.com on 10.0.0.53:53: no such host"}
{"level":"info","ts":"2024-01-24T16:48:41.947955+0500","caller":"embed/config.go:1111","msg":"get cluster for etcd-server SRV","service-scheme":"http","service-name":"etcd-server","server-name":"1.example.com","discovery-srv":"example.com","advertise-peer-urls":["http://localhost:2380"],"found-cluster":[],"error":"lookup _etcd-server._tcp.example.com on 10.0.0.53:53: no such host"}
{"level":"warn","ts":"2024-01-24T16:48:41.94799+0500","caller":"embed/config.go:1045","msg":"failed to resolve during SRV discovery","error":"lookup _etcd-server-ssl._tcp.example.com on 10.0.0.53:53: no such host; lookup _etcd-server._tcp.example.com on 10.0.0.53:53: no such host","errorCauses":[{"error":"lookup _etcd-server-ssl._tcp.example.com on 10.0.0.53:53: no such host"},{"error":"lookup _etcd-server._tcp.example.com on 10.0.0.53:53: no such host"}]}
{"level":"warn","ts":"2024-01-24T16:48:41.948782+0500","caller":"embed/config.go:893","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2024-01-24T16:48:41.948826+0500","caller":"embed/config.go:1096","msg":"get cluster for etcd-server-ssl SRV","service-scheme":"https","service-name":"etcd-server-ssl","server-name":"1.example.com","discovery-srv":"example.com","advertise-peer-urls":["https://1.example.com:2380"],"found-cluster":["1.example.com=https://1.example.com:2380","0=https://2.example.com:2380","1=https://3.example.com:2380"]}
{"level":"info","ts":"2024-01-24T16:48:41.948876+0500","caller":"embed/config.go:1111","msg":"get cluster for etcd-server SRV","service-scheme":"http","service-name":"etcd-server","server-name":"1.example.com","discovery-srv":"example.com","advertise-peer-urls":["https://1.example.com:2380"],"found-cluster":["1.example.com=https://1.example.com:2380","0=https://2.example.com:2380","1=https://3.example.com:2380"],"error":"lookup _etcd-server._tcp.example.com on 10.0.0.53:53: no such host"}
{"level":"warn","ts":"2024-01-24T16:48:41.948915+0500","caller":"embed/config.go:1045","msg":"failed to resolve during SRV discovery","error":"lookup _etcd-server._tcp.example.com on 10.0.0.53:53: no such host"}
{"level":"info","ts":"2024-01-24T16:48:41.948947+0500","caller":"embed/config.go:1051","msg":"got bootstrap from DNS for etcd-server","node":"1.example.com=https://1.example.com:2380"}
{"level":"info","ts":"2024-01-24T16:48:41.948977+0500","caller":"embed/config.go:1051","msg":"got bootstrap from DNS for etcd-server","node":"0=https://2.example.com:2380"}
{"level":"info","ts":"2024-01-24T16:48:41.948999+0500","caller":"embed/config.go:1051","msg":"got bootstrap from DNS for etcd-server","node":"1=https://3.example.com:2380"}
{"level":"warn","ts":"2024-01-24T16:48:41.949793+0500","caller":"embed/config.go:893","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2024-01-24T16:48:41.949848+0500","caller":"embed/config.go:1096","msg":"get cluster for etcd-server-ssl SRV","service-scheme":"https","service-name":"etcd-server-ssl","server-name":"1.example.com","discovery-srv":"example.com","advertise-peer-urls":["https://1.example.com:2380"],"found-cluster":["1.example.com=https://1.example.com:2380"]}
{"level":"info","ts":"2024-01-24T16:48:41.949888+0500","caller":"embed/config.go:1111","msg":"get cluster for etcd-server SRV","service-scheme":"http","service-name":"etcd-server","server-name":"1.example.com","discovery-srv":"example.com","advertise-peer-urls":["https://1.example.com:2380"],"found-cluster":["1.example.com=https://1.example.com:2380","0=http://2.example.com:2380","1=http://3.example.com:2380"]}
{"level":"info","ts":"2024-01-24T16:48:41.949921+0500","caller":"embed/config.go:1051","msg":"got bootstrap from DNS for etcd-server","node":"1.example.com=https://1.example.com:2380"}
{"level":"info","ts":"2024-01-24T16:48:41.949956+0500","caller":"embed/config.go:1051","msg":"got bootstrap from DNS for etcd-server","node":"0=http://2.example.com:2380"}
{"level":"info","ts":"2024-01-24T16:48:41.949982+0500","caller":"embed/config.go:1051","msg":"got bootstrap from DNS for etcd-server","node":"1=http://3.example.com:2380"}
{"level":"warn","ts":"2024-01-24T16:48:41.951272+0500","caller":"embed/config.go:893","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2024-01-24T16:48:41.951335+0500","caller":"embed/config.go:1096","msg":"get cluster for etcd-server-ssl SRV","service-scheme":"https","service-name":"etcd-server-ssl","server-name":"1.example.com","discovery-srv":"example.com","advertise-peer-urls":["http://1.example.com:2380"],"found-cluster":[],"error":"lookup _etcd-server-ssl._tcp.example.com on 10.0.0.53:53: no such host"}
{"level":"info","ts":"2024-01-24T16:48:41.95138+0500","caller":"embed/config.go:1111","msg":"get cluster for etcd-server SRV","service-scheme":"http","service-name":"etcd-server","server-name":"1.example.com","discovery-srv":"example.com","advertise-peer-urls":["http://1.example.com:2380"],"found-cluster":["1.example.com=http://1.example.com:2380","0=http://2.example.com:2380","1=http://3.example.com:2380"]}
{"level":"warn","ts":"2024-01-24T16:48:41.951451+0500","caller":"embed/config.go:1045","msg":"failed to resolve during SRV discovery","error":"lookup _etcd-server-ssl._tcp.example.com on 10.0.0.53:53: no such host"}
{"level":"info","ts":"2024-01-24T16:48:41.951478+0500","caller":"embed/config.go:1051","msg":"got bootstrap from DNS for etcd-server","node":"1.example.com=http://1.example.com:2380"}
{"level":"info","ts":"2024-01-24T16:48:41.951506+0500","caller":"embed/config.go:1051","msg":"got bootstrap from DNS for etcd-server","node":"0=http://2.example.com:2380"}
{"level":"info","ts":"2024-01-24T16:48:41.951526+0500","caller":"embed/config.go:1051","msg":"got bootstrap from DNS for etcd-server","node":"1=http://3.example.com:2380"}
{"level":"warn","ts":"2024-01-24T16:48:41.952257+0500","caller":"embed/config.go:893","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"warn","ts":"2024-01-24T16:48:41.952291+0500","caller":"embed/config.go:995","msg":"it isn't recommended to use default name, please set a value for --name. Note that etcd might run into issue when multiple members have the same default name","name":"default"}
{"level":"warn","ts":"2024-01-24T16:48:41.953007+0500","caller":"embed/config.go:893","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"warn","ts":"2024-01-24T16:48:41.953043+0500","caller":"embed/config.go:978","msg":"Detected that checkpointing is enabled without persistence. Consider enabling experimental-enable-lease-checkpoint-persist"}
{"level":"warn","ts":"2024-01-24T16:48:41.953072+0500","caller":"embed/config.go:995","msg":"it isn't recommended to use default name, please set a value for --name. Note that etcd might run into issue when multiple members have the same default name","name":"default"}
{"level":"warn","ts":"2024-01-24T16:48:41.953827+0500","caller":"embed/config.go:893","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"warn","ts":"2024-01-24T16:48:41.953866+0500","caller":"embed/config.go:995","msg":"it isn't recommended to use default name, please set a value for --name. Note that etcd might run into issue when multiple members have the same default name","name":"default"}
{"level":"warn","ts":"2024-01-24T16:48:41.954669+0500","caller":"embed/config.go:893","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"warn","ts":"2024-01-24T16:48:41.956026+0500","caller":"embed/config.go:893","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"warn","ts":"2024-01-24T16:48:41.957245+0500","caller":"embed/config.go:995","msg":"it isn't recommended to use default name, please set a value for --name. Note that etcd might run into issue when multiple members have the same default name","name":"default"}
{"level":"info","ts":"2024-01-24T16:48:41.957295+0500","caller":"embed/config_test.go:433","msg":"test log"}
{"level":"warn","ts":"2024-01-24T16:48:41.958114+0500","caller":"embed/config.go:893","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"warn","ts":"2024-01-24T16:48:41.958164+0500","caller":"embed/config.go:995","msg":"it isn't recommended to use default name, please set a value for --name. Note that etcd might run into issue when multiple members have the same default name","name":"default"}
{"level":"info","ts":"2024-01-24T16:48:41.958202+0500","caller":"embed/config_test.go:433","msg":"test log"}
{"level":"warn","ts":"2024-01-24T16:48:41.960032+0500","caller":"embed/config.go:893","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"warn","ts":"2024-01-24T16:48:41.960064+0500","caller":"embed/config.go:995","msg":"it isn't recommended to use default name, please set a value for --name. Note that etcd might run into issue when multiple members have the same default name","name":"default"}
{"level":"warn","ts":"2024-01-24T16:48:41.960728+0500","caller":"embed/config.go:893","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"warn","ts":"2024-01-24T16:48:41.960768+0500","caller":"embed/config.go:995","msg":"it isn't recommended to use default name, please set a value for --name. Note that etcd might run into issue when multiple members have the same default name","name":"default"}
{"level":"warn","ts":"2024-01-24T16:48:41.961877+0500","caller":"embed/config.go:893","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"warn","ts":"2024-01-24T16:48:41.962056+0500","caller":"embed/config.go:995","msg":"it isn't recommended to use default name, please set a value for --name. Note that etcd might run into issue when multiple members have the same default name","name":"default"}
{"level":"warn","ts":"2024-01-24T16:48:41.962748+0500","caller":"embed/config.go:893","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"warn","ts":"2024-01-24T16:48:41.962785+0500","caller":"embed/config.go:995","msg":"it isn't recommended to use default name, please set a value for --name. Note that etcd might run into issue when multiple members have the same default name","name":"default"}
{"level":"warn","ts":"2024-01-24T16:48:41.963444+0500","caller":"embed/config.go:893","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"warn","ts":"2024-01-24T16:48:41.963493+0500","caller":"embed/config.go:995","msg":"it isn't recommended to use default name, please set a value for --name. Note that etcd might run into issue when multiple members have the same default name","name":"default"}
{"level":"warn","ts":"2024-01-24T16:48:41.964134+0500","caller":"embed/config.go:893","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"warn","ts":"2024-01-24T16:48:41.964167+0500","caller":"embed/config.go:995","msg":"it isn't recommended to use default name, please set a value for --name. Note that etcd might run into issue when multiple members have the same default name","name":"default"}
{"level":"warn","ts":"2024-01-24T16:48:41.964828+0500","caller":"embed/config.go:893","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"warn","ts":"2024-01-24T16:48:41.964861+0500","caller":"embed/config.go:995","msg":"it isn't recommended to use default name, please set a value for --name. Note that etcd might run into issue when multiple members have the same default name","name":"default"}
{"level":"warn","ts":"2024-01-24T16:48:41.966031+0500","caller":"embed/config.go:893","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"warn","ts":"2024-01-24T16:48:41.967595+0500","caller":"embed/config.go:893","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"warn","ts":"2024-01-24T16:48:41.96764+0500","caller":"embed/config.go:995","msg":"it isn't recommended to use default name, please set a value for --name. Note that etcd might run into issue when multiple members have the same default name","name":"default"}
{"level":"info","ts":"2024-01-24T16:48:41.967676+0500","caller":"embed/etcd.go:125","msg":"configuring peer listeners","listen-peer-urls":["unix://localhost:5084000001"]}
{"level":"info","ts":"2024-01-24T16:48:41.967844+0500","caller":"embed/etcd.go:133","msg":"configuring client listeners","listen-client-urls":["unix://localhost:5084000000"]}
{"level":"info","ts":"2024-01-24T16:48:41.968126+0500","caller":"embed/etcd.go:311","msg":"starting an etcd server","etcd-version":"3.6.0-alpha.0","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.21.3","go-os":"darwin","go-arch":"amd64","max-cpu-set":4,"max-cpu-available":12,"member-initialized":false,"name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/TestStartEtcdWrongToken4148590059/001","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/TestStartEtcdWrongToken4148590059/001/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":10000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["unix://localhost:5084000001"],"listen-peer-urls":["unix://localhost:5084000001"],"advertise-client-urls":["unix://localhost:5084000000"],"listen-client-urls":["unix://localhost:5084000000"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"default=unix://localhost:5084000001","initial-cluster-state":"new","initial-cluster-token":"etcd-cluster","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","discovery-token":"","discovery-endpoints":"","discovery-dial-timeout":"2s","discovery-request-timeout":"5s","discovery-keepalive-time":"2s","discovery-keepalive-timeout":"6s","discovery-insecure-transport":false,"discovery-insecure-skip-tls-verify":false,"discovery-cert":"","discovery-key":"","discovery-cacert":"","discovery-user":"","downgrade-check-interval":"5s","max-learners":1}
{"level":"warn","ts":"2024-01-24T16:48:41.968297+0500","caller":"fileutil/fileutil.go:56","msg":"check file permission","error":"directory \"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/TestStartEtcdWrongToken4148590059/001\" exist, but the permission is \"drwxr-xr-x\". The recommended permission is \"-rwx------\" to prevent possible unprivileged access to the data"}
{"level":"info","ts":"2024-01-24T16:48:41.969222+0500","logger":"bbolt","caller":"backend/backend.go:185","msg":"Opening db file (/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/TestStartEtcdWrongToken4148590059/001/member/snap/db) with mode 2d72772d2d2d2d2d2d2d and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: false, PreLoadFreelist: false, FreelistType: hashmap, ReadOnly: false, MmapFlags: 0, InitialMmapSize: 10737418240, PageSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0xc00039c468}"}
{"level":"info","ts":"2024-01-24T16:48:42.002244+0500","logger":"bbolt","caller":"bbolt@v1.4.0-alpha.0/db.go:319","msg":"Opening bbolt db (/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/TestStartEtcdWrongToken4148590059/001/member/snap/db) successfully"}
{"level":"info","ts":"2024-01-24T16:48:42.002385+0500","caller":"storage/backend.go:80","msg":"opened backend db","path":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/TestStartEtcdWrongToken4148590059/001/member/snap/db","took":"33.303158ms"}
{"level":"info","ts":"2024-01-24T16:48:42.002618+0500","caller":"etcdserver/bootstrap.go:93","msg":"bootstrapping cluster"}
{"level":"info","ts":"2024-01-24T16:48:42.00274+0500","caller":"etcdserver/bootstrap.go:100","msg":"bootstrapping storage"}
{"level":"info","ts":"2024-01-24T16:48:42.071737+0500","caller":"etcdserver/bootstrap.go:108","msg":"bootstrapping raft"}
{"level":"info","ts":"2024-01-24T16:48:42.071952+0500","caller":"etcdserver/bootstrap.go:493","msg":"starting local member","local-member-id":"476f8ca1a44984f8","cluster-id":"29f87ac8031f5d62"}
{"level":"info","ts":"2024-01-24T16:48:42.071999+0500","caller":"etcdserver/server.go:304","msg":"bootstrap successfully"}
{"level":"info","ts":"2024-01-24T16:48:42.072168+0500","logger":"raft","caller":"v3@v3.6.0-alpha.0/raft.go:1920","msg":"476f8ca1a44984f8 switched to configuration voters=()"}
{"level":"info","ts":"2024-01-24T16:48:42.07226+0500","logger":"raft","caller":"v3@v3.6.0-alpha.0/raft.go:870","msg":"476f8ca1a44984f8 became follower at term 0"}
{"level":"info","ts":"2024-01-24T16:48:42.072312+0500","logger":"raft","caller":"v3@v3.6.0-alpha.0/raft.go:483","msg":"newRaft 476f8ca1a44984f8 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]"}
{"level":"info","ts":"2024-01-24T16:48:42.072357+0500","logger":"raft","caller":"v3@v3.6.0-alpha.0/raft.go:870","msg":"476f8ca1a44984f8 became follower at term 1"}
{"level":"info","ts":"2024-01-24T16:48:42.072459+0500","logger":"raft","caller":"v3@v3.6.0-alpha.0/raft.go:1920","msg":"476f8ca1a44984f8 switched to configuration voters=(5147487524981671160)"}
{"level":"warn","ts":"2024-01-24T16:48:42.134105+0500","caller":"auth/store.go:1146","msg":"unknown token type","type":"wrong-token","error":"auth: invalid auth options"}
{"level":"warn","ts":"2024-01-24T16:48:42.134169+0500","caller":"etcdserver/server.go:361","msg":"failed to create token provider","error":"auth: invalid auth options"}
{"level":"info","ts":"2024-01-24T16:48:42.172194+0500","caller":"embed/etcd.go:393","msg":"closing etcd server","name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/TestStartEtcdWrongToken4148590059/001","advertise-peer-urls":["unix://localhost:5084000001"],"advertise-client-urls":["unix://localhost:5084000000"]}
{"level":"info","ts":"2024-01-24T16:48:42.172485+0500","caller":"embed/etcd.go:395","msg":"closed etcd server","name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/TestStartEtcdWrongToken4148590059/001","advertise-peer-urls":["unix://localhost:5084000001"],"advertise-client-urls":["unix://localhost:5084000000"]}
{"level":"info","ts":"2024-01-24T16:48:42.172589+0500","caller":"verify/verify.go:63","msg":"verification of persisted state","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/TestStartEtcdWrongToken4148590059/001"}
{"level":"info","ts":"2024-01-24T16:48:42.172669+0500","logger":"bbolt","caller":"backend/backend.go:185","msg":"Opening db file (/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/TestStartEtcdWrongToken4148590059/001/member/snap/db) with mode 2d72772d2d2d2d2d2d2d and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: false, PreLoadFreelist: false, FreelistType: , ReadOnly: false, MmapFlags: 0, InitialMmapSize: 10737418240, PageSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0xc00039c520}"}
{"level":"info","ts":"2024-01-24T16:48:42.172818+0500","logger":"bbolt","caller":"bbolt@v1.4.0-alpha.0/db.go:319","msg":"Opening bbolt db (/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/TestStartEtcdWrongToken4148590059/001/member/snap/db) successfully"}
{"level":"info","ts":"2024-01-24T16:48:42.173363+0500","caller":"verify/verify.go:131","msg":"verification: consistentIndex OK","backend-consistent-index":0,"hardstate-commit":0}
{"level":"info","ts":"2024-01-24T16:48:42.209266+0500","caller":"verify/verify.go:74","msg":"verification of persisted state successful","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/TestStartEtcdWrongToken4148590059/001"}
FAIL
FAIL    go.etcd.io/etcd/server/v3/embed 1.777s
ok      go.etcd.io/etcd/server/v3/etcdmain      3.216s
?       go.etcd.io/etcd/server/v3/etcdserver/api/v3election     [no test files]
?       go.etcd.io/etcd/server/v3/etcdserver/api/v3election/v3electionpb        [no test files]
?       go.etcd.io/etcd/server/v3/etcdserver/api/v3election/v3electionpb/gw     [no test files]
?       go.etcd.io/etcd/server/v3/etcdserver/api/v3lock [no test files]
?       go.etcd.io/etcd/server/v3/etcdserver/api/v3lock/v3lockpb        [no test files]
?       go.etcd.io/etcd/server/v3/etcdserver/api/v3lock/v3lockpb/gw     [no test files]
?       go.etcd.io/etcd/server/v3/etcdserver/errors     [no test files]
?       go.etcd.io/etcd/server/v3/lease/leasepb [no test files]
?       go.etcd.io/etcd/server/v3/mock/mockstorage      [no test files]
?       go.etcd.io/etcd/server/v3/mock/mockstore        [no test files]
?       go.etcd.io/etcd/server/v3/mock/mockwait [no test files]
?       go.etcd.io/etcd/server/v3/proxy/grpcproxy       [no test files]
?       go.etcd.io/etcd/server/v3/proxy/grpcproxy/adapter       [no test files]
?       go.etcd.io/etcd/server/v3/proxy/grpcproxy/cache [no test files]
?       go.etcd.io/etcd/server/v3/storage       [no test files]
?       go.etcd.io/etcd/server/v3/storage/backend/testing       [no test files]
?       go.etcd.io/etcd/server/v3/storage/mvcc/testutil [no test files]
?       go.etcd.io/etcd/server/v3/storage/wal/testing   [no test files]
?       go.etcd.io/etcd/server/v3/verify        [no test files]
ok      go.etcd.io/etcd/server/v3/etcdserver    41.972s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/etcdhttp       5.968s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/etcdhttp/types 4.387s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/membership     4.179s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp       5.145s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/snap   6.231s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2discovery    5.965s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2error        6.098s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2store        4.654s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v3compactor    7.078s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v3discovery    22.212s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc  4.736s
ok      go.etcd.io/etcd/server/v3/etcdserver/apply      32.756s
ok      go.etcd.io/etcd/server/v3/etcdserver/cindex     7.087s
ok      go.etcd.io/etcd/server/v3/etcdserver/txn        23.039s
ok      go.etcd.io/etcd/server/v3/etcdserver/version    4.252s
ok      go.etcd.io/etcd/server/v3/lease 22.018s
ok      go.etcd.io/etcd/server/v3/lease/leasehttp       8.967s
ok      go.etcd.io/etcd/server/v3/proxy/tcpproxy        4.438s
ok      go.etcd.io/etcd/server/v3/storage/backend       24.645s
ok      go.etcd.io/etcd/server/v3/storage/datadir       2.251s
panic: schedule: schedule to stopped scheduler

goroutine 684 [running]:
go.etcd.io/etcd/pkg/v3/schedule.(*fifo).Schedule(0xc0001f2cb0, {0x1e0e5c0?, 0xc00061ad38})
        /Users/kosovgrigorii/Documents/GitHub/etcd/pkg/schedule/schedule.go:111 +0x265
go.etcd.io/etcd/server/v3/storage/mvcc.(*store).compact(0xc0003a1ee0, 0xc000723f68, 0x26e3, 0x26e2, 0x1)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/storage/mvcc/kvstore.go:252 +0x217
go.etcd.io/etcd/server/v3/storage/mvcc.(*store).Compact(0xc0003a1ee0, 0xc000723f68, 0xc00044efc0?)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/storage/mvcc/kvstore.go:278 +0x312
go.etcd.io/etcd/server/v3/storage/mvcc.TestHashKVWhenCompacting.func3()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/storage/mvcc/kvstore_test.go:610 +0x173
created by go.etcd.io/etcd/server/v3/storage/mvcc.TestHashKVWhenCompacting in goroutine 634
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/storage/mvcc/kvstore_test.go:607 +0x7bd
FAIL    go.etcd.io/etcd/server/v3/storage/mvcc  58.322s
ok      go.etcd.io/etcd/server/v3/storage/schema        42.572s
ok      go.etcd.io/etcd/server/v3/storage/wal   55.609s
ok      go.etcd.io/etcd/server/v3/storage/wal/walpb     1.790s
FAIL
FAIL: (code:1):
  % (cd server && 'env' 'ETCD_VERIFY=all' 'go' 'test' './...' '-short' '-timeout=3m' '--race=true' '--cpu=4')
There was a Failure in module server, aborting...
FAIL: 'unit' FAILED at Wed Jan 24 16:49:50 +05 2024

@KosovGrigorii
Copy link
Author

tests also failed on test_3.5.9 this time:

{"level":"warn","ts":"2024-01-24T16:52:57.834787+0500","caller":"embed/config.go:673","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2024-01-24T16:52:57.835069+0500","caller":"embed/etcd.go:127","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]}
{"level":"error","ts":"2024-01-24T16:52:57.837938+0500","caller":"embed/etcd.go:537","msg":"########## transport.NewListenerWithOpts failed","error":"case: lnOpts.IsTimeout(), lnOpts.IsSocketOpts(): listen tcp 127.0.0.1:2380: bind: address already in use","stacktrace":"go.etcd.io/etcd/server/v3/embed.configurePeerListeners\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:537\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:131\ngo.etcd.io/etcd/server/v3/embed.TestEnableAuth\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34\ntesting.tRunner\n\t/usr/local/opt/go/libexec/src/testing/testing.go:1595"}
{"level":"info","ts":"2024-01-24T16:52:57.838116+0500","caller":"embed/etcd.go:376","msg":"closing etcd server","name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1395285965","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]}
{"level":"info","ts":"2024-01-24T16:52:57.838201+0500","caller":"embed/etcd.go:378","msg":"closed etcd server","name":"default","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1395285965","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]}
{"level":"info","ts":"2024-01-24T16:52:57.838265+0500","caller":"verify/verify.go:57","msg":"verification of persisted state","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1395285965"}
{"level":"panic","ts":"2024-01-24T16:52:57.838393+0500","caller":"backend/backend.go:189","msg":"failed to open database","path":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1395285965/member/snap/db","error":"open /var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1395285965/member/snap/db: no such file or directory","stacktrace":"go.etcd.io/etcd/server/v3/mvcc/backend.newBackend\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189\ngo.etcd.io/etcd/server/v3/mvcc/backend.New\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163\ngo.etcd.io/etcd/server/v3/verify.Verify\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76\ngo.etcd.io/etcd/server/v3/verify.VerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94\ngo.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:379\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:439\ngo.etcd.io/etcd/server/v3/embed.StartEtcd.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:116\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:132\ngo.etcd.io/etcd/server/v3/embed.TestEnableAuth\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34\ntesting.tRunner\n\t/usr/local/opt/go/libexec/src/testing/testing.go:1595"}
{"level":"error","ts":"2024-01-24T16:52:57.838633+0500","caller":"verify/verify.go:64","msg":"verification of persisted state failed","data-dir":"/var/folders/82/0hky4sf567l1n8vzb310vftj0822br/T/auth-test1395285965","stacktrace":"go.etcd.io/etcd/server/v3/verify.Verify.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:64\nruntime.gopanic\n\t/usr/local/opt/go/libexec/src/runtime/panic.go:920\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write\n\t/Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/zapcore/entry.go:234\ngo.uber.org/zap.(*Logger).Panic\n\t/Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/logger.go:227\ngo.etcd.io/etcd/server/v3/mvcc/backend.newBackend\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189\ngo.etcd.io/etcd/server/v3/mvcc/backend.New\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163\ngo.etcd.io/etcd/server/v3/verify.Verify\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76\ngo.etcd.io/etcd/server/v3/verify.VerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94\ngo.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:379\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).Close\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:439\ngo.etcd.io/etcd/server/v3/embed.StartEtcd.func1\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:116\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:132\ngo.etcd.io/etcd/server/v3/embed.TestEnableAuth\n\t/Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34\ntesting.tRunner\n\t/usr/local/opt/go/libexec/src/testing/testing.go:1595"}
--- FAIL: TestEnableAuth (0.01s)
panic: failed to open database [recovered]
        panic: failed to open database [recovered]
        panic: failed to open database

goroutine 151 [running]:
testing.tRunner.func1.2({0x219d4e0, 0xc000588770})
        /usr/local/opt/go/libexec/src/testing/testing.go:1545 +0x366
testing.tRunner.func1()
        /usr/local/opt/go/libexec/src/testing/testing.go:1548 +0x630
panic({0x219d4e0?, 0xc000588770?})
        /usr/local/opt/go/libexec/src/runtime/panic.go:920 +0x270
go.etcd.io/etcd/server/v3/verify.Verify.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:66 +0x505
panic({0x219d4e0?, 0xc000588770?})
        /usr/local/opt/go/libexec/src/runtime/panic.go:920 +0x270
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc000554180, {0xc00020e180, 0x2, 0x2})
        /Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/zapcore/entry.go:234 +0x61b
go.uber.org/zap.(*Logger).Panic(0xc00004e360?, {0x237da00, 0x17}, {0xc00020e180, 0x2, 0x2})
        /Users/kosovgrigorii/go/pkg/mod/go.uber.org/zap@v1.17.0/logger.go:227 +0x68
go.etcd.io/etcd/server/v3/mvcc/backend.newBackend({{0xc00004e360, 0x53}, 0x5f5e100, 0x2710, {0x0, 0x0}, 0x280000000, 0xc0005a0aa0, 0x0, 0x0, ...})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:189 +0x6b3
go.etcd.io/etcd/server/v3/mvcc/backend.New(...)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/mvcc/backend/backend.go:163
go.etcd.io/etcd/server/v3/verify.Verify({{0xc00004a0f0?, 0xb?}, 0xc0?, 0xc0005a0aa0?})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:76 +0x478
go.etcd.io/etcd/server/v3/verify.VerifyIfEnabled({{0xc00004a0f0?, 0xc00065eb40?}, 0xc8?, 0xc0005a0aa0?})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:94 +0x76
go.etcd.io/etcd/server/v3/verify.MustVerifyIfEnabled({{0xc00004a0f0?, 0x2376630?}, 0x12?, 0xc0005a0aa0?})
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/verify/verify.go:103 +0x66
go.etcd.io/etcd/server/v3/embed.(*Etcd).Close.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:379 +0x9e
go.etcd.io/etcd/server/v3/embed.(*Etcd).Close(0xc0004df100)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:439 +0xcf4
go.etcd.io/etcd/server/v3/embed.StartEtcd.func1()
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:116 +0x109
go.etcd.io/etcd/server/v3/embed.StartEtcd(0xc000542000)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/etcd.go:132 +0xa2c
go.etcd.io/etcd/server/v3/embed.TestEnableAuth(0xc000007ba0)
        /Users/kosovgrigorii/Documents/GitHub/etcd/server/embed/auth_test.go:34 +0x1d0
testing.tRunner(0xc000007ba0, 0x23d58c0)
        /usr/local/opt/go/libexec/src/testing/testing.go:1595 +0x239
created by testing.(*T).Run in goroutine 1
        /usr/local/opt/go/libexec/src/testing/testing.go:1648 +0x82b
FAIL    go.etcd.io/etcd/server/v3/embed 2.499s
ok      go.etcd.io/etcd/server/v3/etcdmain      2.700s
ok      go.etcd.io/etcd/server/v3/etcdserver    32.237s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/etcdhttp       6.158s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/membership     22.046s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp       4.613s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/snap   6.625s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2auth 8.693s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2discovery    6.530s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2error        6.257s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2http 6.085s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2http/httptypes       5.837s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v2store        5.330s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v3compactor    6.981s
ok      go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc  4.162s
ok      go.etcd.io/etcd/server/v3/etcdserver/cindex     6.934s
ok      go.etcd.io/etcd/server/v3/lease 18.227s
ok      go.etcd.io/etcd/server/v3/lease/leasehttp       7.191s
ok      go.etcd.io/etcd/server/v3/mvcc  52.241s
ok      go.etcd.io/etcd/server/v3/mvcc/backend  18.645s
ok      go.etcd.io/etcd/server/v3/proxy/httpproxy       2.735s
ok      go.etcd.io/etcd/server/v3/proxy/tcpproxy        2.928s
ok      go.etcd.io/etcd/server/v3/wal   43.849s
ok      go.etcd.io/etcd/server/v3/wal/walpb     2.007s
FAIL
FAIL: (code:1):
  % (cd server && 'env' 'go' 'test' '-short' '-timeout=3m' '--race=true' '--cpu=4' './...')
FAIL: 'unit' failed at Wed Jan 24 16:53:59 +05 2024

@ahrtr
Copy link
Member

ahrtr commented Jan 24, 2024

Thanks @KosovGrigorii for the feedback.

There are two issues:

  • Your environment's issue, the port 2380 is already being used. Please double check that.
$ lsof -iTCP -sTCP:LISTEN -P -n

@KosovGrigorii
Copy link
Author

Yes, forgot to turn off etcd while running tests, therefore environment issue. Sorry for that

On PR everything works great! Thank you

@ahrtr
Copy link
Member

ahrtr commented Jan 25, 2024

Yes, forgot to turn off etcd while running tests, therefore environment issue. Sorry for that

On PR everything works great! Thank you

Thanks for the confirmation. Will close this ticket once #17313 and the backporting PRs get merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging a pull request may close this issue.

2 participants