Etcd can't be used a datastore on multihost

[tomdee]

I'm unable to use etcd as a backend datastore. This is a long standing issue - AFAIK, libnetwork/libkv has never worked with etcd.

$ docker version
Client:
 Version:      1.8.0-dev
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   2fe51ec
 Built:        Wed Jul  8 01:33:10 UTC 2015
 OS/Arch:      linux/amd64
 Experimental: true

Server:
 Version:      1.8.0-dev
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   2fe51ec
 Built:        Wed Jul  8 01:33:10 UTC 2015
 OS/Arch:      linux/amd64
 Experimental: true

Docker output

$ sudo docker --kv-store=etcd:127.0.0.1:4001 -Dd
WARN[0000] Running experimental build                   
DEBU[0000] Registering DELETE, /containers/{name:.*}    
DEBU[0000] Registering DELETE, /images/{name:.*}        
DEBU[0000] Registering OPTIONS,                         
DEBU[0000] Registering GET, /containers/{name:.*}/stats 
DEBU[0000] Registering GET, /info                       
DEBU[0000] Registering GET, /images/json                
DEBU[0000] Registering GET, /images/search              
DEBU[0000] Registering GET, /images/get                 
DEBU[0000] Registering GET, /images/{name:.*}/history   
DEBU[0000] Registering GET, /containers/{name:.*}/export 
DEBU[0000] Registering GET, /events                     
DEBU[0000] Registering GET, /images/{name:.*}/json      
DEBU[0000] Registering GET, /containers/{name:.*}/json  
DEBU[0000] Registering GET, /containers/{name:.*}/logs  
DEBU[0000] Registering GET, /containers/{name:.*}/attach/ws 
DEBU[0000] Registering GET, /exec/{id:.*}/json          
DEBU[0000] Registering GET, /_ping                      
DEBU[0000] Registering GET, /containers/ps              
DEBU[0000] Registering GET, /version                    
DEBU[0000] Registering GET, /images/{name:.*}/get       
DEBU[0000] Registering GET, /containers/json            
DEBU[0000] Registering GET, /containers/{name:.*}/changes 
DEBU[0000] Registering GET, /containers/{name:.*}/top   
DEBU[0000] Registering POST, /images/load               
DEBU[0000] Registering POST, /images/{name:.*}/tag      
DEBU[0000] Registering POST, /containers/{name:.*}/unpause 
DEBU[0000] Registering POST, /containers/{name:.*}/start 
DEBU[0000] Registering POST, /containers/{name:.*}/stop 
DEBU[0000] Registering POST, /containers/{name:.*}/exec 
DEBU[0000] Registering POST, /containers/{name:.*}/rename 
DEBU[0000] Registering POST, /images/create             
DEBU[0000] Registering POST, /containers/create         
DEBU[0000] Registering POST, /containers/{name:.*}/kill 
DEBU[0000] Registering POST, /containers/{name:.*}/copy 
DEBU[0000] Registering POST, /auth                      
DEBU[0000] Registering POST, /build                     
DEBU[0000] Registering POST, /containers/{name:.*}/pause 
DEBU[0000] Registering POST, /containers/{name:.*}/restart 
DEBU[0000] Registering POST, /containers/{name:.*}/wait 
DEBU[0000] Registering POST, /containers/{name:.*}/attach 
DEBU[0000] Registering POST, /commit                    
DEBU[0000] Registering POST, /containers/{name:.*}/resize 
DEBU[0000] Registering POST, /exec/{name:.*}/start      
DEBU[0000] Registering POST, /exec/{name:.*}/resize     
DEBU[0000] Registering POST, /images/{name:.*}/push     
DEBU[0000] docker group found. gid: 131                 
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock) 
INFO[0000] [graphdriver] using prior storage driver "aufs" 
DEBU[0000] Using graph driver aufs                      
DEBU[0000] Using default logging driver json-file       
DEBU[0000] Migrating existing containers                
DEBU[0000] Creating images graph                        
DEBU[0000] Restored 89 elements                         
DEBU[0000] Reloaded graph with 3 grants expiring at 2017-03-22 19:04:46.713978458 +0000 UTC 
DEBU[0000] Creating repository list                     
INFO[0000] Option DefaultDriver: bridge                 
INFO[0000] Option DefaultNetwork: bridge                
INFO[0000] Option OptionKVProvider: etcd                
INFO[0000] Option OptionKVProviderURL: 127.0.0.1:4001   
INFO[0000] Firewalld running: false                     
DEBU[0000] /sbin/iptables, [--wait -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER] 
DEBU[0000] /sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER] 
DEBU[0000] /sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER] 
DEBU[0000] /sbin/iptables, [--wait -t nat -D PREROUTING] 
DEBU[0000] /sbin/iptables, [--wait -t nat -D OUTPUT]    
DEBU[0000] /sbin/iptables, [--wait -t nat -F DOCKER]    
DEBU[0000] /sbin/iptables, [--wait -t nat -X DOCKER]    
FATA[0000] Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to initialize vxlan id manager: failed to initialize bit sequence handler: 100: Key not found (/docker) [2]

etcd was started with

~/etcd-v2.0.11-linux-amd64$ ./etcd
2015/07/13 11:07:46 etcd: no data-dir provided, using default data-dir ./default.etcd
2015/07/13 11:07:46 etcd: listening for peers on http://localhost:2380
2015/07/13 11:07:46 etcd: listening for peers on http://localhost:7001
2015/07/13 11:07:46 etcd: listening for client requests on http://localhost:2379
2015/07/13 11:07:46 etcd: listening for client requests on http://localhost:4001
2015/07/13 11:07:46 etcdserver: datadir is valid for the 2.0.1 format
2015/07/13 11:07:46 etcdserver: name = default
2015/07/13 11:07:46 etcdserver: data dir = default.etcd
2015/07/13 11:07:46 etcdserver: member dir = default.etcd/member
2015/07/13 11:07:46 etcdserver: heartbeat = 100ms
2015/07/13 11:07:46 etcdserver: election = 1000ms
2015/07/13 11:07:46 etcdserver: snapshot count = 10000
2015/07/13 11:07:46 etcdserver: advertise client URLs = http://localhost:2379,http://localhost:4001
2015/07/13 11:07:46 etcdserver: initial advertise peer URLs = http://localhost:2380,http://localhost:7001
2015/07/13 11:07:46 etcdserver: initial cluster = default=http://localhost:2380,default=http://localhost:7001
2015/07/13 11:07:46 etcdserver: start member ce2a822cea30bfca in cluster 7e27652122e8b2ae
2015/07/13 11:07:46 raft: ce2a822cea30bfca became follower at term 0
2015/07/13 11:07:46 raft: newRaft ce2a822cea30bfca [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2015/07/13 11:07:46 raft: ce2a822cea30bfca became follower at term 1
2015/07/13 11:07:46 etcdserver: added local member ce2a822cea30bfca [http://localhost:2380 http://localhost:7001] to cluster 7e27652122e8b2ae
2015/07/13 11:07:47 raft: ce2a822cea30bfca is starting a new election at term 1
2015/07/13 11:07:47 raft: ce2a822cea30bfca became candidate at term 2
2015/07/13 11:07:47 raft: ce2a822cea30bfca received vote from ce2a822cea30bfca at term 2
2015/07/13 11:07:47 raft: ce2a822cea30bfca became leader at term 2
2015/07/13 11:07:47 raft.node: ce2a822cea30bfca elected leader ce2a822cea30bfca at term 2
2015/07/13 11:07:47 etcdserver: published {Name:default ClientURLs:[http://localhost:2379 http://localhost:4001]} to cluster 7e27652122e8b2ae

[tomdee]

@dave-tucker I would also argue that this needs to be fixed for 0.4, or the docs updating to say it doesn't work. It's a bit disingenuous to advertise etcd support when it has never worked. I'm happy to provide a docs PR if that's the way you think this should go?

[dave-tucker]

@tomdee - a fix for 0.4 is unrealistic as afaik we are blocked by docker/libkv#20

I agree - it's odd to advertise support for something that doesn't work - but the whole idea of libkv is to abstract away the complexity of dealing with multiple kv implementations. Our docs are trusting that the upstream repo does as it claims.

I'd only recommend removing etcd from the docs if libkv plans to drop support for it.
However, I would support a PR to add a note about the current issue with etcd to the libnetwork docs if we think that this won't be resolved in the near future.

[springi99]

Hi,

there is an issue in docker/libkv#26

I started that thread because i had problems of working with etcd.

The solution proposals there seems solve the problem, so you can use etcd for kv storage. (at least I can use etcd with overlay and remote ovn) One more additional issue is listed there related to network delete and cascade deleting of empty endpoints belonging to the network. I was also asked there to reaise the problem in libnetwork. Thanks, br,

robert.

[dave-tucker]

@tomdee are your issues resolved with the above mentioned patch?

[abronan]

@dave-tucker It's probably time to update libkv in Godeps here? We merged small fixes here and there that were useful in Swarm for issues with the discovery and Leader Election mechanisms. I also merged the patch from @springi99.

So far, the 3 backends are working on our side with Swarm because we stay in the limits imposed by docker/libkv#20 (we should document it better and add a warning section until this is fixed).

Unfortunately the core issues with etcd are not going to be fixed until etcd API v3 is out. See etcd-io/etcd#2675 (comment)

[springi99]

Hi,

I started again the investigation with etcd as kv store. I have the following problems:

1. if you start with empty etcd then the watch/watchtree fails, because watchNetworks() and watchEndpoints() are called before the entries are created
2. when you create network, endpoint abd restart docker then these directories are existing dirs, so for the second start there is no problem

According to @abronan watch/watchtree should be called when entries are there. If entries are not there then create it.

The following changes are needed (besides changes listed in docker/libkv#26 including the change in libnetwork related to DeleteTree):

@@ -174,6 +176,9 @@ func (c *controller) watchNetworks() error {
        cs := c.store
        c.Unlock()

+       if _, err := cs.KVStore().Get(datastore.Key(datastore.NetworkKeyPrefix)); err != nil {
+           cs.KVStore().CreateTree(datastore.Key(datastore.NetworkKeyPrefix))
+       }
        nwPairs, err := cs.KVStore().WatchTree(datastore.Key(datastore.NetworkKeyPrefix), nil)
        if err != nil {
                return err
@@ -228,6 +233,9 @@ func (n *network) watchEndpoints() error {
        stopCh := n.stopWatchCh
        n.Unlock()

+       if _, err := cs.KVStore().Get(datastore.Key(tmp.KeyPrefix()...)); err != nil {
+           cs.KVStore().CreateTree(datastore.Key(tmp.KeyPrefix()...))
+       }
        epPairs, err := cs.KVStore().WatchTree(datastore.Key(tmp.KeyPrefix()...), stopCh)
        if err != nil {
                return err

The CreateTree is just creating directory in libkv, it could be replaced with an existing API which can create directory.

These changes are affecting only watchtree part in libnetwork, maybe watch should be also checked.

Can anybody from here can comment it? Thanks in advance!

[chenchun]

@springi99 yes, I've open #462 to fix it. For both etcd and zookeeper, it's not allowed to watch a not exist node.

[springi99]

@chenchun Thanks for your update! I have my temporary solution which is working. it has a little fluctuation at the beginning (local nets endpoint watchtree, but disappeared after few seconds from etcd). Because i "introduced" CreateTree() in libkv, i think my solution is not so good, however there is DeleteTree() in libkv as well (there is createDirectory in libkv, my CreateTree just wraps that). I cannot test my solution with zookeeper and consul. I try to understand those pull 466 commits.

[mavenugo]

@springi99 @chenchun yes, we are waiting on #466 to be merged. am looking at some issues related to #466 to be addressed before merging it and letting #462 to follow. pls watch these PRs for updates.

[mavenugo]

@tomdee @springi99 we decided to unblock these PRs by isolating #466 away from libkv upgrade.
Now that #462 is merged, can you please check if this Issue can be closed ?

[springi99]

Hi,

sorry for the delay. The "fix for zookeeper..." contains ensureKeys, and it puts docker/libnetwork/network into the datastore as an entry and not as a directory in the case of etcd. After this it is not possible to add any network to docker because it requires directory. I think same applies to endpoint. Both are there because of watchtree.

[chenchun]

Yes, but that is a problem of etcd and in docker/libkv#61 as @abronan mentioned

Etcd version >= 2.0 because it uses the 2.0.0 branch of the coreos/go-etcd client, this might change in the future as the support for APIv3 comes along.

I suggest you to use consul/zk instead or wait for APIv3 of etcd.

[springi99]

Hi, i don't fully understand, maybe i am wrong, but i have some problems: if you create a key (docker/libnetwork/network) and run watchtree, then it will watch the parent (docker/libnetwork/) and not the directory which will be created later (docker/libnetwork/network/ if apiv3 allows it at all). In this case watchnetworks are also triggered when there is a change in endpoint tree as it is under docker/libnetwork/.

[chenchun]

if you create a key (docker/libnetwork/network) and run watchtree, then it will watch the parent (docker/libnetwork/)

I don't understand. Can you please explain?

[springi99]

I made a mistake, sorry for that. The current situation on code level:

libnetwork's store.go has watchNetworks(). it is creating key docker/libnetwork/network (ensureKeys()). it is not a directory. Then it calls WatchTree(). But WatchTree() requires directory and not a key (libkv's etcd.go has the WatchTree() expecting directory).

If i can propose here something is that try to create something like ensureDirectory() instead of ensureKeys().

[chenchun]

@springi99 You can give it a try and make a PR if that works.

[abronan]

@tomdee Let us know if you find any issue related to etcd. Should work fine now, just tested :)

Also we now have integration tests in #657 so this should ensure we see regressions right away.

/cc @mavenugo

[abronan]

Closing as this is working and because we have tests checking the default behavior now. Please open another issue if you still have any more problems with etcd. This will help to focus on more specific errors if there are still issues. Thanks!