etcd-member 3 node cluster setup

[darktall]

I am trying to configure etcd 3 node cluster in coreos . Below is my configuration file

/usr/lib/systemd/system/etcd-member.service

[Unit]
Description=etcd (System Application Container)
Documentation=https://github.com/coreos/etcd
Wants=network.target
Conflicts=etcd.service
Conflicts=etcd2.service

[Service]
Type=notify
Restart=on-failure
RestartSec=10s
TimeoutStartSec=0
LimitNOFILE=40000

Environment="ETCD_IMAGE_TAG=v3.1.8"
Environment="ETCD_NAME=%m"
Environment="ETCD_USER=etcd"
Environment="ETCD_DATA_DIR=/var/lib/etcd"
Environment="RKT_RUN_ARGS=--uuid-file-save=/var/lib/coreos/etcd-member-wrapper.uuid"

ExecStartPre=/usr/bin/mkdir --parents /var/lib/coreos
ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/lib/coreos/etcd-member-wrapper.uuid
ExecStart=/usr/lib/coreos/etcd-wrapper $ETCD_OPTS
ExecStop=-/usr/bin/rkt stop --uuid-file=/var/lib/coreos/etcd-member-wrapper.uuid

[Install]
WantedBy=multi-user.target

/etc/systemd/system/etcd-member.service.d/20-cl-etcd-member.conf

[Service]
[Service]
Environment="ETCD_IMAGE_TAG=v3.1.8"
Environment="ETCD_DATA_DIR=/var/lib/etcd"
Environment="ETCD_OPTS=--name s1 --listen-client-urls http://10.98.8.205:2379 --advertise-client-urls http://10.98.8.205:2379 --listen-peer-urls http://10.98.8.205:2380 --initial-advertise-peer-urls http://10.98.8.205:2380 --initial-cluster s1=http://10.98.8.205:2380,s2=http://10.98.8.206:2380,s3=http://10.98.8.207:2380 --initial-cluster-token mytoken --initial-cluster-state new

output when i run etcdctl cluster-health

member 8e9e05c52164694d is healthy: got healthy result from http://localhost:2379
cluster is healthy

coreos version 1465.6.0

How to configure 3 node cluster properly.

[gyuho]

Have you tried https://github.com/coreos/etcd/blob/master/Documentation/platforms/container-linux-systemd.md?

[gyuho]

Try etcdctl endpoint --cluster health?

[darktall]

exec /usr/bin/rkt run --uuid-file-save=/var/lib/coreos/etcd-member-wrapper.uuid --trust-keys-from-https --mount volume=coreos-systemd-dir,targ
017-08-23 12:13:41.922682 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd
017-08-23 12:13:41.922775 I | pkg/flags: recognized and used environment variable ETCD_NAME=e4051993f61a4271b96b24f9e5a157f7
017-08-23 12:13:41.922808 W | pkg/flags: unrecognized environment variable ETCD_USER=etcd
017-08-23 12:13:41.922816 W | pkg/flags: unrecognized environment variable ETCD_IMAGE_TAG=v3.1.8
017-08-23 12:13:41.922828 I | etcdmain: etcd Version: 3.1.8
017-08-23 12:13:41.922834 I | etcdmain: Git SHA: d267ca9
017-08-23 12:13:41.922839 I | etcdmain: Go Version: go1.7.5
017-08-23 12:13:41.922844 I | etcdmain: Go OS/Arch: linux/amd64
017-08-23 12:13:41.922850 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
017-08-23 12:13:41.922894 N | etcdmain: the server is already initialized as member before, starting as etcd member...
017-08-23 12:13:41.923177 I | embed: listening for peers on http://localhost:2380
017-08-23 12:13:41.923283 I | embed: listening for client requests on localhost:2379
017-08-23 12:13:41.924120 I | etcdserver: name = e4051993f61a4271b96b24f9e5a157f7
017-08-23 12:13:41.924130 I | etcdserver: data dir = /var/lib/etcd
017-08-23 12:13:41.924137 I | etcdserver: member dir = /var/lib/etcd/member
017-08-23 12:13:41.924145 I | etcdserver: heartbeat = 100ms
017-08-23 12:13:41.924150 I | etcdserver: election = 1000ms
017-08-23 12:13:41.924156 I | etcdserver: snapshot count = 10000
017-08-23 12:13:41.924165 I | etcdserver: advertise client URLs = http://localhost:2379
017-08-23 12:13:41.983618 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 21
017-08-23 12:13:41.983730 I | raft: 8e9e05c52164694d became follower at term 6
017-08-23 12:13:41.983753 I | raft: newRaft 8e9e05c52164694d [peers: [], term: 6, commit: 21, applied: 0, lastindex: 21, lastterm: 6]
017-08-23 12:13:41.992733 I | etcdserver: starting server... [version: 3.1.8, cluster version: to_be_decided]
017-08-23 12:13:41.993958 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
017-08-23 12:13:41.994050 N | etcdserver/membership: set the initial cluster version to 3.1
017-08-23 12:13:41.994081 I | etcdserver/api: enabled capabilities for version 3.1
017-08-23 12:13:42.384140 I | raft: 8e9e05c52164694d is starting a new election at term 6
017-08-23 12:13:42.384247 I | raft: 8e9e05c52164694d became candidate at term 7
017-08-23 12:13:42.384276 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 7
017-08-23 12:13:42.384298 I | raft: 8e9e05c52164694d became leader at term 7
017-08-23 12:13:42.384311 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 7
017-08-23 12:13:42.384876 I | etcdserver: published {Name:e4051993f61a4271b96b24f9e5a157f7 ClientURLs:[http://localhost:2379]} to cluster cdf81
017-08-23 12:13:42.385465 I | embed: ready to serve client requests
tcd (System Application Container).
017-08-23 12:13:42.386815 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!

Am i doing wrong configuration ?

[darktall]

ETCDCTL_API=3 etcdctl cluster-health
member 8e9e05c52164694d is healthy: got healthy result from http://localhost:2379
cluster is healthy

[gyuho]

@darktall Above log just shows a single node cluster, so everything is expected.
What was your command to start your 3-node cluster? You need

systemctl enable --now etcd-member.service

Or

systemctl start etcd-member.service

[darktall]

Systemctl enable etcd-member.service

Systemctl start etcd-member.service

Is how I did

Thank you.

[gyuho]

When you run etcdctl cluster-health, add --cluster flag.

So try etcdctl cluster-health --cluster?

[darktall]

ETCDCTL_API=3 etcdctl cluster-health
member 8e9e05c52164694d is healthy: got healthy result from http://localhost:2379
cluster is healthy

I got above out put.

[gyuho]

ETCDCTL_API=3 etcdctl cluster-health

Again, try etcdctl cluster-health --cluster?

[darktall]

my cat 20-cl-etcd-member.conf look like below
[Service]
Environment="ETCD_IMAGE_TAG=v3.1.8"
Environment="ETCD_DATA_DIR=/var/lib/etcd"
Environment="ETCD_OPTS=--name s1
--listen-client-urls http://10.98.8.205:2379
--advertise-client-urls http://10.98.8.205:2379
--listen-peer-urls http://10.98.8.205:2380
--initial-advertise-peer-urls http://10.98.8.205:2380
--initial-cluster s1=http://10.98.8.205:2380,s2=http://10.98.8.206:2380,s3=http://10.98.8.207:2380
--initial-cluster-token mytoken
--initial-cluster-state new
--auto-compaction-retention 1"

with out --auto-compaction-retention flag also its not working for me

[gyuho]

@darktall Sorry, --cluster flag has not been release yet.
And also let's ignore --auto-compaction-retention 1 for now (just remove it).

After you copy your 20-cl-etcd-member.conf to /etc/systemd/system/etcd-member.service.d/20-cl-etcd-member.conf, did you run

systemd-delta --type=extended
systemctl daemon-reload

[gyuho]

Also try etcdctl --endpoints=YOUR_ENDPOINTS cluster-health with --endpoints?

[darktall]

I just dropped you mail ( gyuhox@gmail.com ). please check once.

[darktall]

ENDPOINTS=10.98.8.205:2379,10.98.8.206:2379,10.98.8.207:2379
ETCDCTL_API=3 etcdctl --endpoints=$ENDPOINTS cluster-health
member 8e9e05c52164694d is healthy: got healthy result from http://10.98.8.205:2379
cluster is healthy

[darktall]

Thank you very much gyuho

etcdctl --endpoints=$ENDPOINTS member list
36a9a5c1b5f51721: name=s2 peerURLs=http://10.98.8.206:2380 clientURLs=http://10.98.8.206:2379 isLeader=true
bf084ef134b8f29b: name=s3 peerURLs=http://10.98.8.207:2380 clientURLs=http://10.98.8.207:2379 isLeader=false
c2229566a5ab0f28: name=s1 peerURLs=http://10.98.8.205:2380 clientURLs=http://10.98.8.205:2379 isLeader=false

Its working now.

[darktall]

Thank you for your support.