Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

failed to recover v3 backend from snapshot ,failed to find database snapshot file #14569

Closed
IamSatyaonline opened this issue Oct 10, 2022 · 5 comments
Closed

failed to recover v3 backend from snapshot ,failed to find database snapshot file #14569

IamSatyaonline opened this issue Oct 10, 2022 · 5 comments
Labels
type/bug

Comments

@IamSatyaonline
Copy link

What happened?

We have a ETCD cluster of kubernetes which has one member with ETCD version - 3.5.3 and it was up and running from a long time of 25 days. Suddendly something happended in ETCD and getting restarted frequently,ETCD is throwing the error below as "panic: failed to recover v3 backend from snapshot"

"message" : "{"level":"panic","ts":"2022-10-06T09:18:02.022-0500","caller":"etcdserver/server.go:515","msg":"failed to recover v3 backend from snapshot","error":"failed to find database snapshot file (snap: snapshot file doesn'
"message" : "

{"level":"info","ts":"2022-10-06T09:18:01.910-0500","caller":"etcdserver/server.go:508","msg":
{*}

"recovered v2 store from snapshot{*}","snapshot-index":3420684,"snapshot-size":"23 kB"}",
"Restarting ETCD",
"message" : "{"level":"warn","ts":"2022-10-06T09:18:02.022-0500","caller":"snap/db.go:88","msg":"failed to find [SNAPSHOT-INDEX].snap.db","snapshot-index":3420684,"snapshot-file-path":"/data/member/snap/000000000034320c.snap.db","error":"snap: snapshot file doesn't exist"}",
"message" : "panic: failed to recover v3 backend from snapshot"

What did you expect to happen?

There should not be any issue in ETCD service and ETCD should be up and running instead of complaining about the missing snapshot file.
Is there any specific reason due to that ETCD had missed the snapshot file , Could you please help us to find the root cause of the issue ?

How can we reproduce it (as minimally and precisely as possible)?

We don't know the steps to reproduce it as it is not reproducible.

Anything else we need to know?

No response

Etcd version (please run commands below)

$ etcd --version
# paste output here

$ etcdctl version

etcd-version: 3.5.3
git-sha: 0452feec7
go-version: go1.16.15
go-os: linux
go-arch: amd64

Etcd configuration (command line flags or environment variables)

ETCD_AUTO_COMPACTION_RETENTION: 100
ETCD_CERT_FILE: /run/sec/certs/server/srvcert.pem
ETCD_PEER_AUTO_TLS: true
ETCD_INITIAL_CLUSTER: etcd-0=https
ETCD_INITIAL_CLUSTER_TOKEN: etcd
ETCD_LISTEN_CLIENT_URLS: https
ETCD_MAX_SNAPSHOTS: 3
ETCD_MAX_WALS: 3
ETCD_ADVERTISE:CLIENT_URLS: https
ETCD_ENABLE_V2 false:
ETCD_KEY_FILE: /run/sec/certs/server/srvprivkey.pem
ETCD_SNAPSHOT_COUNT: 5000
ETCD_AUTO_COMPACTION_MODE: revision
ETCD_CLIENT_CERT_AUTH: true
ETCD_ELECTION_TIMEOUT: 1000
ETCD_HEARTBEAT_INTERVAL: 100
ETCD_INITIAL_CLUSTER_STATE: new
ETCD_NAME: etcd-0
ETCD_QUOTA_BACKEND_BYTES: 268435456
ETCD_TRUSTED_CA_FILE: /data/combinedca/cacertbundle.pem
ETCD_DATA_DIR: /data
ETCD_INITIAL_ADVERTISE_PEER_URLS: https
ETCD_LISTEN_PEER_URLS: https
ETCD_METRICS: basic

Etcd debug information (please run commands blow, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

message" : "\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdserver/server.go:515 +0x1656",
          "message" : "go.etcd.io/etcd/server/v3/embed.StartEtcd(0xc0001c4000, 0xc000562000, 0x0, 0x0)",
          "message" : "\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/etcd.go:123 +0x257a",
          "message" : "\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/main.go:40 +0x13f",
          "message" : "main.main()",
          "message" : "\t/go/src/go.etcd.io/etcd/release/etcd/server/main.go:32 +0x45",
          "message" : "panic: failed to recover v3 backend from snapshot",
          "message" : "go.uber.org/zap.(*Logger).Panic(0xc0007fa3c0, 0x12346a6, 0x2a, 0xc00038a140, 0x1, 0x1)",
          "message" : "\t/go/src/go.etcd.io/etcd/release/etcd/server/embed/etcd.go:245 +0xef8",
          "message" : "\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/etcd.go:228 +0x32",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:48.917-0500\",\"caller\":\"etcdserver/server.go:508\",\"msg\":\"recovered v2 store from snapshot\",\"snapshot-index\":3420684,\"snapshot-size\":\"23 kB\"}",
          "message" : "go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc00034a780, 0xc00038a140, 0x1, 0x1)",
          "message" : "\t/go/pkg/mod/go.uber.org/zap@v1.17.0/zapcore/entry.go:234 +0x58d",
          "message" : "\t/go/pkg/mod/go.uber.org/zap@v1.17.0/logger.go:227 +0x85",
          "message" : "{\"level\":\"warn\",\"ts\":\"2022-09-29T13:19:49.114-0500\",\"caller\":\"snap/db.go:88\",\"msg\":\"failed to find [SNAPSHOT-INDEX].snap.db\",\"snapshot-index\":3420684,\"snapshot-file-path\":\"/data/member/snap/000000000034320c.snap.db\",\"error\":\"snap: snapshot file doesn't exist\"}",
          "message" : "go.etcd.io/etcd/server/v3/etcdmain.startEtcd(0xc0001c4000, 0x120893e, 0x6, 0xc0002b2801, 0x2)",
          "message" : "go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2(0xc00021e1d0, 0x1, 0x1)",
          "message" : "go.etcd.io/etcd/server/v3/etcdmain.Main(0xc00021e1d0, 0x1, 0x1)",
          "message" : "{\"level\":\"panic\",\"ts\":\"2022-09-29T13:19:49.114-0500\",\"caller\":\"etcdserver/server.go:515\",\"msg\":\"failed to recover v3 backend from snapshot\",\"error\":\"failed to find database snapshot file (snap: snapshot file doesn't exist)\",\"stacktrace\":\"go.etcd.io/etcd/server/v3/etcdserver.NewServer\\n\\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdserver/server.go:515\\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\\n\\t/go/src/go.etcd.io/etcd/release/etcd/server/embed/etcd.go:245\\ngo.etcd.io/etcd/server/v3/etcdmain.startEtcd\\n\\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/etcd.go:228\\ngo.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\\n\\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/etcd.go:123\\ngo.etcd.io/etcd/server/v3/etcdmain.Main\\n\\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/main.go:40\\nmain.main\\n\\t/go/src/go.etcd.io/etcd/release/etcd/server/main.go:32\\nruntime.main\\n\\t/go/gos/go1.16.15/src/runtime/proc.go:225\"}",
          "message" : "goroutine 1 [running]:",
          "message" : "go.etcd.io/etcd/server/v3/etcdserver.NewServer(0xc00004f4da, 0x26, 0x0, 0x0, 0x0, 0x0, 0xc0003f6a20, 0x1, 0x1, 0xc0003f6c60, ...)",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:42.613-0500\",\"caller\":\"etcdserver/backend.go:81\",\"msg\":\"opened backend db\",\"path\":\"/data/member/snap/db\",\"took\":\"2.59842235s\"}",
          "message" : "{\"level\":\"warn\",\"ts\":\"2022-09-29T13:19:39.809-0500\",\"caller\":\"etcdmain/etcd.go:446\",\"msg\":\"found invalid file under data directory\",\"filename\":\"auth_successful\",\"data-dir\":\"/data\"}",
          "message" : "{\"level\":\"warn\",\"ts\":\"2022-09-29T13:19:39.809-0500\",\"caller\":\"etcdmain/etcd.go:446\",\"msg\":\"found invalid file under data directory\",\"filename\":\"fixtures\",\"data-dir\":\"/data\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.809-0500\",\"caller\":\"etcdmain/etcd.go:116\",\"msg\":\"server has been already initialized\",\"data-dir\":\"/data\",\"dir-type\":\"member\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.811-0500\",\"caller\":\"embed/etcd.go:479\",\"msg\":\"starting with peer TLS\",\"tls-info\":\"cert = /data/fixtures/peer/cert.pem, key = /data/fixtures/peer/key.pem, client-cert=/data/fixtures/peer/cert.pem, client-key=/data/fixtures/peer/key.pem, trusted-ca = , client-cert-auth = false, crl-file = \",\"cipher-suites\":[]}",
          "message" : "{\"level\":\"warn\",\"ts\":\"2022-09-29T13:19:39.809-0500\",\"caller\":\"etcdmain/etcd.go:446\",\"msg\":\"found invalid file under data directory\",\"filename\":\"combinedca\",\"data-dir\":\"/data\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:40.010-0500\",\"caller\":\"embed/etcd.go:139\",\"msg\":\"configuring client listeners\",\"listen-client-urls\":[\"https://0.0.0.0:2379\"]}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:40.010-0500\",\"caller\":\"embed/etcd.go:308\",\"msg\":\"starting an etcd server\",\"etcd-version\":\"3.5.3\",\"git-sha\":\"0452feec7\",\"go-version\":\"go1.16.15\",\"go-os\":\"linux\",\"go-arch\":\"amd64\",\"max-cpu-set\":112,\"max-cpu-available\":112,\"member-initialized\":true,\"name\":\"etcd-0\",\"data-dir\":\"/data\",\"wal-dir\":\"\",\"wal-dir-dedicated\":\"\",\"member-dir\":\"/data/member\",\"force-new-cluster\":false,\"heartbeat-interval\":\"100ms\",\"election-timeout\":\"1s\",\"initial-election-tick-advance\":true,\"snapshot-count\":5000,\"snapshot-catchup-entries\":5000,\"initial-advertise-peer-urls\":[\"https://etcd-0.etcd-peer.omalnexuvzwcgnb-y-ec-vcuup-001.svc.cluster.local:2380\"],\"listen-peer-urls\":[\"https://0.0.0.0:2380\"],\"advertise-client-urls\":[\"https://etcd-0.etcd.omalnexuvzwcgnb-y-ec-vcuup-001:2379\"],\"listen-client-urls\":[\"https://0.0.0.0:2379\"],\"listen-metrics-urls\":[],\"cors\":[\"*\"],\"host-whitelist\":[\"*\"],\"initial-cluster\":\"\",\"initial-cluster-state\":\"new\",\"initial-cluster-token\":\"\",\"quota-size-bytes\":268435456,\"pre-vote\":true,\"initial-corrupt-check\":false,\"corrupt-check-time-interval\":\"0s\",\"auto-compaction-mode\":\"revision\",\"auto-compaction-retention\":\"100ns\",\"auto-compaction-interval\":\"100ns\",\"discovery-url\":\"\",\"discovery-proxy\":\"\",\"downgrade-check-interval\":\"5s\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.809-0500\",\"caller\":\"embed/etcd.go:131\",\"msg\":\"configuring peer listeners\",\"listen-peer-urls\":[\"https://0.0.0.0:2380\"]}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"etcdmain/etcd.go:73\",\"msg\":\"Running: \",\"args\":[\"/usr/local/bin/etcd\"]}",
          "message" : "{\"level\":\"warn\",\"ts\":\"2022-09-29T13:19:39.809-0500\",\"caller\":\"etcdmain/etcd.go:446\",\"msg\":\"found invalid file under data directory\",\"filename\":\"etcd.liveness\",\"data-dir\":\"/data\"}",
          "message" : "{\"level\":\"warn\",\"ts\":1664475580.0112433,\"caller\":\"fileutil/fileutil.go:57\",\"msg\":\"check file permission\",\"error\":\"directory \\\"/data\\\" exist, but the permission is \\\"drwxrwxrwx\\\". The recommended permission is \\\"-rwx------\\\" to prevent possible unprivileged access to the data\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_AUTO_COMPACTION_RETENTION\",\"variable-value\":\"100\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_CERT_FILE\",\"variable-value\":\"/run/sec/certs/server/srvcert.pem\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_PEER_AUTO_TLS\",\"variable-value\":\"true\"}",
          "message" : "++ grep -ic begin /data/combinedca/cacertbundle.pem",
          "message" : "+ [[ 3 -gt 1 ]]",
          "message" : "+ echo 'Is bootstraping ? false'",
          "message" : "+ [[ 0 == 0 ]]",
          "message" : "Auto TLS enable , using ETCD auto generated certs for peer communication",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_INITIAL_CLUSTER\",\"variable-value\":\"etcd-0=https://etcd-0.etcd-peer.omalnexuvzwcgnb-y-ec-vcuup-001.svc.cluster.local:2380\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_INITIAL_CLUSTER_TOKEN\",\"variable-value\":\"etcd\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_LISTEN_CLIENT_URLS\",\"variable-value\":\"https://0.0.0.0:2379\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_MAX_SNAPSHOTS\",\"variable-value\":\"3\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_MAX_WALS\",\"variable-value\":\"3\"}",
          "message" : "{\"level\":\"warn\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:93\",\"msg\":\"unrecognized environment variable\",\"environment-variable\":\"ETCD_LOG_PACKAGE_LEVELS=etcdserver=INFO,security=INFO\"}",
          "message" : "+ numberofcerts=3",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_ADVERTISE_CLIENT_URLS\",\"variable-value\":\"https://etcd-0.etcd.omalnexuvzwcgnb-y-ec-vcuup-001:2379\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_ENABLE_V2\",\"variable-value\":\"false\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_KEY_FILE\",\"variable-value\":\"/run/sec/certs/server/srvprivkey.pem\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_SNAPSHOT_COUNT\",\"variable-value\":\"5000\"}",
          "message" : "{\"level\":\"warn\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:93\",\"msg\":\"unrecognized environment variable\",\"environment-variable\":\"ETCD_DEBUG=false\"}",
          "message" : "2022-09-29 13:19:39.524249 I |  [[ ! -f /data/combinedca/cacertbundle.pem ]]",
          "message" : "+ [[ false == \\f\\a\\l\\s\\e ]]",
          "message" : "+ echo 'Setup peer certs for etcd-0 '",
          "message" : "+ export ETCD_LISTEN_PEER_URLS",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_AUTO_COMPACTION_MODE\",\"variable-value\":\"revision\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_CLIENT_CERT_AUTH\",\"variable-value\":\"true\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_ELECTION_TIMEOUT\",\"variable-value\":\"1000\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_HEARTBEAT_INTERVAL\",\"variable-value\":\"100\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_INITIAL_CLUSTER_STATE\",\"variable-value\":\"new\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_NAME\",\"variable-value\":\"etcd-0\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_QUOTA_BACKEND_BYTES\",\"variable-value\":\"268435456\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_TRUSTED_CA_FILE\",\"variable-value\":\"/data/combinedca/cacertbundle.pem\"}",
          "message" : "{\"level\":\"warn\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:93\",\"msg\":\"unrecognized environment variable\",\"environment-variable\":\"ETCD_FIFO_DIR=/fifo\"}",
          "message" : "+ isBootstrapping=false",
          "message" : "+ ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380",
          "message" : "+ [[ true == \\f\\a\\l\\s\\e ]]",
          "message" : "+ echo 'Auto TLS enable , using ETCD auto generated certs for peer communication'",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_DATA_DIR\",\"variable-value\":\"/data\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_INITIAL_ADVERTISE_PEER_URLS\",\"variable-value\":\"https://etcd-0.etcd-peer.omalnexuvzwcgnb-y-ec-vcuup-001.svc.cluster.local:2380\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_LISTEN_PEER_URLS\",\"variable-value\":\"https://0.0.0.0:2380\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:39.715-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_METRICS\",\"variable-value\":\"basic\"}",
          "message" : "+ [[ -f /data/combinedca/cacertbundle.pem ]]",
          "message" : "Is bootstraping ? false",
          "message" : "Setup peer certs for etcd-0 ",
          "message" : "+ [[ true == \\f\\a\\l\\s\\e ]]",
          "message" : "+ /usr/local/bin/etcd",
          "message" : "++ echo etcd-0",
          "message" : "+ export ETCD_INITIAL_CLUSTER_STATE=new",
          "message" : "+ ETCD_INITIAL_CLUSTER=etcd-0=https://etcd-0.etcd-peer.omalnexuvzwcgnb-y-ec-vcuup-001.svc.cluster.local:2380",
          "message" : "+ /usr/local/bin/scripts/switch_ca_cert.sh -cs",
          "message" : "2022-09-29 13:19:39.411699 I |  CHECK=true",
          "message" : "2022-09-29 13:19:39.415065 I |  getopts cs option",
          "message" : "2022-09-29 13:19:39.420629 I |  [[ -v SUICIDE ]]",
          "message" : "2022-09-29 13:19:39.516183 I |  [[ ! -s /data/combinedca/cacertbundle.pem ]]",
          "message" : "2022-09-29 13:19:39.517329 I |  awk 1 /run/sec/cas/clientca/client-cacertbundle.pem /run/sec/cas/pmca/client-cacertbundle.pem /run/sec/cas/siptlsca/cacertbundle.pem",
          "message" : "++ sed 's/-0$//g'",
          "message" : "+ AUTO_TLS=true",
          "message" : "+ CA_FILE=/data/combinedca/cacertbundle.pem",
          "message" : "2022-09-29 13:19:39.523311 I |  [[ -v CHECK ]]",
          "message" : "+ initial_cluster_base=etcd-0=https://etcd-0.etcd-peer.omalnexuvzwcgnb-y-ec-vcuup-001.svc.cluster.local:2380",
          "message" : "+ export ETCD_INITIAL_CLUSTER",
          "message" : "2022-09-29 13:19:39.419720 I |  [[ ! -f /run/sec/cas/clientca/client-cacertbundle.pem ]]",
          "message" : "22022-09-29 13:19:39.422180 I |  ps -ef",
          "message" : "22022-09-29 13:19:39.422465 I |  grep -v grep",
          "message" : "2022-09-29 13:19:39.513774 I |  rm -f /data/combinedca/cacertbundle.pem",
          "message" : "+ [[ 0 = \\0 ]]",
          "message" : "+ replace_string=XYZ",
          "message" : "+ initial_cluster_base=XYZ=https://XYZ.etcd-peer.omalnexuvzwcgnb-y-ec-vcuup-001.svc.cluster.local:2380",
          "message" : "+ (( c=1 ))",
          "message" : "+ ETCD_PEER_CERT_FILE=/run/sec/certs/peer/srvcert.pem",
          "message" : "2022-09-29 13:19:39.409914 I |  getopts cs option",
          "message" : "2022-09-29 13:19:39.410818 I |  case \"${option}\" in",
          "message" : "2022-09-29 13:19:39.413438 I |  case \"${option}\" in",
          "message" : "2022-09-29 13:19:39.414256 I |  SUICIDE=true",
          "message" : "2022-09-29 13:19:39.418877 I |  [[ ! -f /run/sec/cas/bootstrap/cacertbundle.pem ]]",
          "message" : "+ base_name=etcd",
          "message" : "+ ETCD_INITIAL_CLUSTER_STATE=new",
          "message" : "+ (( c<=0 ))",
          "message" : "+ ETCD_PEER_KEY_FILE=/run/sec/certs/peer/srvprivkey.pem",
          "message" : "2022-09-29 13:19:39.412580 I |  getopts cs option",
          "message" : "2022-09-29 13:19:39.415875 I |  mkdir -p /data/combinedca/",
          "message" : "2022-09-29 13:19:39.418069 I |  [[ 0 == 0 ]]",
          "message" : "22022-09-29 13:19:39.422262 I |  grep watch_cert",
          "message" : "2022-09-29 13:19:39.512405 I |  [[ -z 250422        77       1  0 13:17 ?        00:00:00 /bin/bash /usr/local/bin/scripts/watch_cert.sh /run/sec/cas/clientca /run/sec/cas/pmca /run/sec/cas/siptlsca ]]",
          "message" : "2022-09-29 13:19:39.521247 I |  [[ ! -s /data/combinedca/cacertbundle.pem ]]",
          "message" : "2022-09-29 13:19:39.522247 I |  [[ ! -s /data/combinedca/cacertbundle.pem ]]",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:38.516-0500\",\"caller\":\"etcdserver/server.go:508\",\"msg\":\"recovered v2 store from snapshot\",\"snapshot-index\":3420684,\"snapshot-size\":\"23 kB\"}",
          "message" : "\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/etcd.go:228 +0x32",
          "message" : "\t/go/src/go.etcd.io/etcd/release/etcd/server/main.go:32 +0x45",
          "message" : "+ ordinal=0",
          "message" : "go.uber.org/zap.(*Logger).Panic(0xc00025c5a0, 0x12346a6, 0x2a, 0xc0005bd600, 0x1, 0x1)",
          "message" : "go.etcd.io/etcd/server/v3/etcdmain.startEtcd(0xc000020000, 0x120893e, 0x6, 0xc0002e3101, 0x2)",
          "message" : "go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2(0xc00003c1f0, 0x1, 0x1)",
          "message" : "\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/etcd.go:123 +0x257a",
          "message" : "go.etcd.io/etcd/server/v3/etcdmain.Main(0xc00003c1f0, 0x1, 0x1)",
          "message" : "panic: failed to recover v3 backend from snapshot",
          "message" : "goroutine 1 [running]:",
          "message" : "\t/go/src/go.etcd.io/etcd/release/etcd/server/embed/etcd.go:245 +0xef8",
          "message" : "\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/main.go:40 +0x13f",
          "message" : "\t/go/pkg/mod/go.uber.org/zap@v1.17.0/zapcore/entry.go:234 +0x58d",
          "message" : "\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdserver/server.go:515 +0x1656",
          "message" : "go.etcd.io/etcd/server/v3/embed.StartEtcd(0xc000020000, 0xc000020600, 0x0, 0x0)",
          "message" : "Restarting ETCD",
          "message" : "{\"level\":\"warn\",\"ts\":\"2022-09-29T13:19:38.809-0500\",\"caller\":\"snap/db.go:88\",\"msg\":\"failed to find [SNAPSHOT-INDEX].snap.db\",\"snapshot-index\":3420684,\"snapshot-file-path\":\"/data/member/snap/000000000034320c.snap.db\",\"error\":\"snap: snapshot file doesn't exist\"}",
          "message" : "{\"level\":\"panic\",\"ts\":\"2022-09-29T13:19:38.809-0500\",\"caller\":\"etcdserver/server.go:515\",\"msg\":\"failed to recover v3 backend from snapshot\",\"error\":\"failed to find database snapshot file (snap: snapshot file doesn't exist)\",\"stacktrace\":\"go.etcd.io/etcd/server/v3/etcdserver.NewServer\\n\\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdserver/server.go:515\\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\\n\\t/go/src/go.etcd.io/etcd/release/etcd/server/embed/etcd.go:245\\ngo.etcd.io/etcd/server/v3/etcdmain.startEtcd\\n\\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/etcd.go:228\\ngo.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\\n\\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/etcd.go:123\\ngo.etcd.io/etcd/server/v3/etcdmain.Main\\n\\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/main.go:40\\nmain.main\\n\\t/go/src/go.etcd.io/etcd/release/etcd/server/main.go:32\\nruntime.main\\n\\t/go/gos/go1.16.15/src/runtime/proc.go:225\"}",
          "message" : "go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0001ec3c0, 0xc0005bd600, 0x1, 0x1)",
          "message" : "\t/go/pkg/mod/go.uber.org/zap@v1.17.0/logger.go:227 +0x85",
          "message" : "go.etcd.io/etcd/server/v3/etcdserver.NewServer(0xc00004f4da, 0x26, 0x0, 0x0, 0x0, 0x0, 0xc00088ca20, 0x1, 0x1, 0xc00088cc60, ...)",
          "message" : "main.main()",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:32.114-0500\",\"caller\":\"etcdserver/backend.go:81\",\"msg\":\"opened backend db\",\"path\":\"/data/member/snap/db\",\"took\":\"2.396737053s\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.516-0500\",\"caller\":\"embed/etcd.go:479\",\"msg\":\"starting with peer TLS\",\"tls-info\":\"cert = /data/fixtures/peer/cert.pem, key = /data/fixtures/peer/key.pem, client-cert=/data/fixtures/peer/cert.pem, client-key=/data/fixtures/peer/key.pem, trusted-ca = , client-cert-auth = false, crl-file = \",\"cipher-suites\":[]}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_SNAPSHOT_COUNT\",\"variable-value\":\"5000\"}",
          "message" : "{\"level\":\"warn\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:93\",\"msg\":\"unrecognized environment variable\",\"environment-variable\":\"ETCD_LOG_PACKAGE_LEVELS=etcdserver=INFO,security=INFO\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"etcdmain/etcd.go:73\",\"msg\":\"Running: \",\"args\":[\"/usr/local/bin/etcd\"]}",
          "message" : "{\"level\":\"warn\",\"ts\":\"2022-09-29T13:19:29.515-0500\",\"caller\":\"etcdmain/etcd.go:446\",\"msg\":\"found invalid file under data directory\",\"filename\":\"combinedca\",\"data-dir\":\"/data\"}",
          "message" : "{\"level\":\"warn\",\"ts\":\"2022-09-29T13:19:29.515-0500\",\"caller\":\"etcdmain/etcd.go:446\",\"msg\":\"found invalid file under data directory\",\"filename\":\"etcd.liveness\",\"data-dir\":\"/data\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.713-0500\",\"caller\":\"embed/etcd.go:139\",\"msg\":\"configuring client listeners\",\"listen-client-urls\":[\"https://0.0.0.0:2379\"]}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_QUOTA_BACKEND_BYTES\",\"variable-value\":\"268435456\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_TRUSTED_CA_FILE\",\"variable-value\":\"/data/combinedca/cacertbundle.pem\"}",
          "message" : "{\"level\":\"warn\",\"ts\":\"2022-09-29T13:19:29.515-0500\",\"caller\":\"etcdmain/etcd.go:446\",\"msg\":\"found invalid file under data directory\",\"filename\":\"fixtures\",\"data-dir\":\"/data\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.515-0500\",\"caller\":\"etcdmain/etcd.go:116\",\"msg\":\"server has been already initialized\",\"data-dir\":\"/data\",\"dir-type\":\"member\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.515-0500\",\"caller\":\"embed/etcd.go:131\",\"msg\":\"configuring peer listeners\",\"listen-peer-urls\":[\"https://0.0.0.0:2380\"]}",
          "message" : "{\"level\":\"warn\",\"ts\":1664475569.714364,\"caller\":\"fileutil/fileutil.go:57\",\"msg\":\"check file permission\",\"error\":\"directory \\\"/data\\\" exist, but the permission is \\\"drwxrwxrwx\\\". The recommended permission is \\\"-rwx------\\\" to prevent possible unprivileged access to the data\"}",
          "message" : "{\"level\":\"warn\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:93\",\"msg\":\"unrecognized environment variable\",\"environment-variable\":\"ETCD_DEBUG=false\"}",
          "message" : "{\"level\":\"warn\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:93\",\"msg\":\"unrecognized environment variable\",\"environment-variable\":\"ETCD_FIFO_DIR=/fifo\"}",
          "message" : "{\"level\":\"warn\",\"ts\":\"2022-09-29T13:19:29.515-0500\",\"caller\":\"etcdmain/etcd.go:446\",\"msg\":\"found invalid file under data directory\",\"filename\":\"auth_successful\",\"data-dir\":\"/data\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.713-0500\",\"caller\":\"embed/etcd.go:308\",\"msg\":\"starting an etcd server\",\"etcd-version\":\"3.5.3\",\"git-sha\":\"0452feec7\",\"go-version\":\"go1.16.15\",\"go-os\":\"linux\",\"go-arch\":\"amd64\",\"max-cpu-set\":112,\"max-cpu-available\":112,\"member-initialized\":true,\"name\":\"etcd-0\",\"data-dir\":\"/data\",\"wal-dir\":\"\",\"wal-dir-dedicated\":\"\",\"member-dir\":\"/data/member\",\"force-new-cluster\":false,\"heartbeat-interval\":\"100ms\",\"election-timeout\":\"1s\",\"initial-election-tick-advance\":true,\"snapshot-count\":5000,\"snapshot-catchup-entries\":5000,\"initial-advertise-peer-urls\":[\"https://etcd-0.etcd-peer.omalnexuvzwcgnb-y-ec-vcuup-001.svc.cluster.local:2380\"],\"listen-peer-urls\":[\"https://0.0.0.0:2380\"],\"advertise-client-urls\":[\"https://etcd-0.etcd.omalnexuvzwcgnb-y-ec-vcuup-001:2379\"],\"listen-client-urls\":[\"https://0.0.0.0:2379\"],\"listen-metrics-urls\":[],\"cors\":[\"*\"],\"host-whitelist\":[\"*\"],\"initial-cluster\":\"\",\"initial-cluster-state\":\"new\",\"initial-cluster-token\":\"\",\"quota-size-bytes\":268435456,\"pre-vote\":true,\"initial-corrupt-check\":false,\"corrupt-check-time-interval\":\"0s\",\"auto-compaction-mode\":\"revision\",\"auto-compaction-retention\":\"100ns\",\"auto-compaction-interval\":\"100ns\",\"discovery-url\":\"\",\"discovery-proxy\":\"\",\"downgrade-check-interval\":\"5s\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_ELECTION_TIMEOUT\",\"variable-value\":\"1000\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_ENABLE_V2\",\"variable-value\":\"false\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_INITIAL_CLUSTER_STATE\",\"variable-value\":\"new\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_LISTEN_PEER_URLS\",\"variable-value\":\"https://0.0.0.0:2380\"}",
          "message" : "2022-09-29 13:19:29.321517 I |  [[ ! -s /data/combinedca/cacertbundle.pem ]]",
          "message" : "+ isBootstrapping=false",
          "message" : "+ [[ false == \\f\\a\\l\\s\\e ]]",
          "message" : "+ echo 'Auto TLS enable , using ETCD auto generated certs for peer communication'",
          "message" : "+ /usr/local/bin/etcd",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_AUTO_COMPACTION_MODE\",\"variable-value\":\"revision\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_AUTO_COMPACTION_RETENTION\",\"variable-value\":\"100\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_CLIENT_CERT_AUTH\",\"variable-value\":\"true\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_HEARTBEAT_INTERVAL\",\"variable-value\":\"100\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_INITIAL_CLUSTER\",\"variable-value\":\"etcd-0=https://etcd-0.etcd-peer.omalnexuvzwcgnb-y-ec-vcuup-001.svc.cluster.local:2380\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_INITIAL_CLUSTER_TOKEN\",\"variable-value\":\"etcd\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_LISTEN_CLIENT_URLS\",\"variable-value\":\"https://0.0.0.0:2379\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_METRICS\",\"variable-value\":\"basic\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_NAME\",\"variable-value\":\"etcd-0\"}",
          "message" : "2022-09-29 13:19:29.317073 I |  [[ ! -s /data/combinedca/cacertbundle.pem ]]",
          "message" : "2022-09-29 13:19:29.318208 I |  awk 1 /run/sec/cas/clientca/client-cacertbundle.pem /run/sec/cas/pmca/client-cacertbundle.pem /run/sec/cas/siptlsca/cacertbundle.pem",
          "message" : "2022-09-29 13:19:29.322330 I |  [[ ! -s /data/combinedca/cacertbundle.pem ]]",
          "message" : "+ numberofcerts=3",
          "message" : "+ [[ -f /data/combinedca/cacertbundle.pem ]]",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_INITIAL_ADVERTISE_PEER_URLS\",\"variable-value\":\"https://etcd-0.etcd-peer.omalnexuvzwcgnb-y-ec-vcuup-001.svc.cluster.local:2380\"}",
          "message" : "2022-09-29 13:19:29.314793 I |  rm -f /data/combinedca/cacertbundle.pem",
          "message" : "+ echo 'Setup peer certs for etcd-0 '",
          "message" : "Setup peer certs for etcd-0 ",
          "message" : "+ [[ true == \\f\\a\\l\\s\\e ]]",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_CERT_FILE\",\"variable-value\":\"/run/sec/certs/server/srvcert.pem\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_DATA_DIR\",\"variable-value\":\"/data\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_KEY_FILE\",\"variable-value\":\"/run/sec/certs/server/srvprivkey.pem\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_MAX_SNAPSHOTS\",\"variable-value\":\"3\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_MAX_WALS\",\"variable-value\":\"3\"}",
          "message" : "2022-09-29 13:19:29.323173 I |  [[ -v CHECK ]]",
          "message" : "++ grep -ic begin /data/combinedca/cacertbundle.pem",
          "message" : "+ [[ 3 -gt 1 ]]",
          "message" : "Is bootstraping ? false",
          "message" : "+ [[ 0 == 0 ]]",
          "message" : "+ export ETCD_LISTEN_PEER_URLS",
          "message" : "+ [[ true == \\f\\a\\l\\s\\e ]]",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_ADVERTISE_CLIENT_URLS\",\"variable-value\":\"https://etcd-0.etcd.omalnexuvzwcgnb-y-ec-vcuup-001:2379\"}",
          "message" : "{\"level\":\"info\",\"ts\":\"2022-09-29T13:19:29.514-0500\",\"caller\":\"flags/flag.go:113\",\"msg\":\"recognized and used environment variable\",\"variable-name\":\"ETCD_PEER_AUTO_TLS\",\"variable-value\":\"true\"}",
          "message" : "2022-09-29 13:19:29.409573 I |  [[ ! -f /data/combinedca/cacertbundle.pem ]]",
          "message" : "+ echo 'Is bootstraping ? false'",
          "message" : "+ ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380",
          "message" : "Auto TLS enable , using ETCD auto generated certs for peer communication",
          "message" : "\t/go/src/go.etcd.io/etcd/release/etcd/server/embed/etcd.go:245 +0xef8",
          "message" : "\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/etcd.go:228 +0x32",
          "message" : "\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/etcd.go:123 +0x257a",
          "message" : "\t/go/src/go.etcd.io/etcd/release/etcd/server/main.go:32 +0x45",
@serathius
Copy link
Member

Looks like a duplicate of #14382
Please upgrade to v3.5.5

cc @ahrtr Looks like fix for #14382 is not mentioned in v3.5.5 changelog or I missed it?

@ahrtr
Copy link
Member

ahrtr commented Oct 10, 2022

ETCD_QUOTA_BACKEND_BYTES: 268435456

Why do you configure such a small value (about 268MB) ? Please check the db size. I suspect it ran out of the quota. If yes, then most likely the same issue as #14382.

cc @ahrtr Looks like fix for #14382 is not mentioned in v3.5.5 changelog or I missed it?

It's included in the 3.5.5 changelog. Fix [etcd fails to start after performing alarm list operation and then power off/on]

@serathius
Copy link
Member

It's included in the 3.5.5 changelog. Fix [etcd fails to start after performing alarm list operation and then power off/on]

Thanks for confirming. I must have searched it incorrectly.
Closing the issue as it was already fixed in the newest release.

@IamSatyaonline
Copy link
Author

ETCD_QUOTA_BACKEND_BYTES: 268435456

Why do you configure such a small value (about 268MB) ? Please check the db size. I suspect it ran out of the quota. If yes, then most likely the same issue as #14382.

cc @ahrtr Looks like fix for #14382 is not mentioned in v3.5.5 changelog or I missed it?

It's included in the 3.5.5 changelog. Fix [etcd fails to start after performing alarm list operation and then power off/on]

Hi @ahrtr
is the panic error 'failed to find database snapshot file' in the etcd log due to insufficient db size ?
The ticket #14382 is indicating that a hard reboot happened on underlying worker nodes and due to this reboot ETCD service went down.
Could you please help to understand the root cause of this panic error ? This panic message issue is due to insufficient db size or due to hard reboot ?

Thanks,
Satya

@ahrtr
Copy link
Member

ahrtr commented Oct 13, 2022

@IamSatyaonline
Please refer to #14382 (comment) and https://github.com/ahrtr/etcd-issues/blob/b221ffdee411e9dc1715d329f5e67f41366012b3/etcd/etcd-db-editor/main.go#L16-L28

@serathius serathius mentioned this issue Nov 14, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug
Milestone
No milestone
Development

No branches or pull requests
3 participants
@serathius @ahrtr @IamSatyaonline