Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed to update store for object type *libnetwork.sbState: timeout #300

Closed
2 of 3 tasks
daniloradenovic opened this issue May 4, 2018 · 5 comments
Closed
2 of 3 tasks

Failed to update store for object type *libnetwork.sbState: timeout #300

daniloradenovic opened this issue May 4, 2018 · 5 comments
Labels
area/networking status/more-info-needed

Comments

@daniloradenovic
Copy link

  • This is a bug report
  • This is a feature request
  • I searched existing issues before opening this one

Expected behavior

Container starts normally

Actual behavior

Container fails to start with message:

Handler for POST /containers/f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144/start returned error: failed to update the store state of sandbox: failed to update store for object type *libnetwork.sbState: timeout
Handler for POST /containers/631ca9eadfccdfc43f1537451f4c978a7c20bd7b792f8c1683c3018be1cd123e/start returned error: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:402: container init caused \\\"process_linux.go:385: running prestart hook 0 caused \\\\\\\"error running hook: exit status 1, stdout: , stderr: time=\\\\\\\\\\\\\\\"2018-05-03T12:57:32Z\\\\\\\\\\\\\\\" level=fatal msg=\\\\\\\\\\\\\\\"failed to update store for object type *libnetwork.sbState: timeout\\\\\\\\\\\\\\\"\\\\\\\\n\\\\\\\"\\\"\": unknown

Steps to reproduce the behavior

It seems to be happening randomly.

There are different workflows that are ran and they use different images with slightly different configuration (the only difference is the command they execute). However, the images and commands themselves should not be the problem, since they work 99% of the time. They all have the following in common:

  • Create a container
  • Start a container
  • If starting the container failed, retry up to 5 times

Here are the full logs:

 time="2018-05-03T12:19:11.280760099Z" level=info msg="libcontainerd: started new docker-containerd process" pid=3334
 time="2018-05-03T12:19:12Z" level=info msg="starting containerd" module=containerd revision=9b55aab90508bd389d7654c4baf173a981477d55 version=v1.0.1
 time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.content.v1.content"..." module=containerd type=io.containerd.content.v1
 time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." module=containerd type=io.containerd.snapshotter.v1
 time="2018-05-03T12:19:12Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /mnt/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containerd
 time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." module=containerd type=io.containerd.snapshotter.v1
 time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." module=containerd type=io.containerd.metadata.v1
 time="2018-05-03T12:19:12Z" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /mnt/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module="containerd/io.containerd.metadata.v1.bolt"
 time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." module=containerd type=io.containerd.differ.v1
 time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." module=containerd type=io.containerd.gc.v1
 time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." module=containerd type=io.containerd.grpc.v1
 time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." module=containerd type=io.containerd.grpc.v1
 time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." module=containerd type=io.containerd.grpc.v1
 time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." module=containerd type=io.containerd.grpc.v1
 time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." module=containerd type=io.containerd.grpc.v1
 time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." module=containerd type=io.containerd.grpc.v1
 time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." module=containerd type=io.containerd.grpc.v1
 time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." module=containerd type=io.containerd.grpc.v1
 time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." module=containerd type=io.containerd.grpc.v1
 time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." module=containerd type=io.containerd.monitor.v1
 time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." module=containerd type=io.containerd.runtime.v1
 time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." module=containerd type=io.containerd.grpc.v1
 time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." module=containerd type=io.containerd.grpc.v1
 time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." module=containerd type=io.containerd.grpc.v1
 time="2018-05-03T12:19:12Z" level=info msg=serving... address="/var/run/docker/containerd/docker-containerd-debug.sock" module="containerd/debug"
 time="2018-05-03T12:19:12Z" level=info msg=serving... address="/var/run/docker/containerd/docker-containerd.sock" module="containerd/grpc"
 time="2018-05-03T12:19:12Z" level=info msg="containerd successfully booted in 0.089212s" module=containerd
 time="2018-05-03T12:19:12.284828500Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
 time="2018-05-03T12:19:12.285045125Z" level=warning msg="Your kernel does not support swap memory limit"
 time="2018-05-03T12:19:12.285101076Z" level=warning msg="Your kernel does not support cgroup rt period"
 time="2018-05-03T12:19:12.285110903Z" level=warning msg="Your kernel does not support cgroup rt runtime"
 time="2018-05-03T12:19:12.285704364Z" level=info msg="Loading containers: start."
 time="2018-05-03T12:19:12.823183830Z" level=info msg="Loading containers: done."
[   63.888085] aufs au_opts_verify:1597:dockerd[3331]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:19:13.330127079Z" level=info msg="Docker daemon" commit=7390fc6 graphdriver(s)=aufs version=17.12.1-ce
 time="2018-05-03T12:19:13.331615457Z" level=info msg="Daemon has completed initialization"
 time="2018-05-03T12:19:13.354929865Z" level=info msg="API listen on /var/run/docker.sock"
[  143.048870] aufs au_opts_verify:1597:dockerd[4553]: dirperm1 breaks the protection by the permission bits on the lower branch
[  143.155169] aufs au_opts_verify:1597:dockerd[4553]: dirperm1 breaks the protection by the permission bits on the lower branch
[  143.457290] aufs au_opts_verify:1597:dockerd[4476]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:20:32Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/aba881de89494c2dd283fb8ed2abbaba0f640616dfaf4a709eb1b8c7fd41dc29/shim.sock" debug=false module="containerd/tasks" pid=9473
 time="2018-05-03T12:20:33Z" level=info msg="shim reaped" id=aba881de89494c2dd283fb8ed2abbaba0f640616dfaf4a709eb1b8c7fd41dc29 module="containerd/tasks"
 time="2018-05-03T12:20:33.867804679Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[  147.235783] aufs au_opts_verify:1597:dockerd[3368]: dirperm1 breaks the protection by the permission bits on the lower branch
[  147.303928] aufs au_opts_verify:1597:dockerd[3368]: dirperm1 breaks the protection by the permission bits on the lower branch
[  147.397198] aufs au_opts_verify:1597:dockerd[4468]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:20:36Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/3feb7c5b83daf36bf8e5d1e8622600a0155ce069ec93b773e7d6fb2d93743095/shim.sock" debug=false module="containerd/tasks" pid=10323
 time="2018-05-03T12:20:36Z" level=info msg="shim reaped" id=3feb7c5b83daf36bf8e5d1e8622600a0155ce069ec93b773e7d6fb2d93743095 module="containerd/tasks"
 time="2018-05-03T12:20:36.974804718Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[  155.481175] aufs au_opts_verify:1597:dockerd[9475]: dirperm1 breaks the protection by the permission bits on the lower branch
[  155.549655] aufs au_opts_verify:1597:dockerd[9475]: dirperm1 breaks the protection by the permission bits on the lower branch
[  155.589105] aufs au_opts_verify:1597:dockerd[3348]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:20:44Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/e227edd391250ada7e68a97f3554366a979929c5ace3bbd0f34a1b15bd05af0d/shim.sock" debug=false module="containerd/tasks" pid=11131
[  155.698247] aufs au_opts_verify:1597:dockerd[4502]: dirperm1 breaks the protection by the permission bits on the lower branch
[  155.742128] aufs au_opts_verify:1597:dockerd[4467]: dirperm1 breaks the protection by the permission bits on the lower branch
[  155.782631] aufs au_opts_verify:1597:dockerd[4502]: dirperm1 breaks the protection by the permission bits on the lower branch
[  155.782877] aufs au_opts_verify:1597:dockerd[3348]: dirperm1 breaks the protection by the permission bits on the lower branch
[  155.817885] aufs au_opts_verify:1597:dockerd[4467]: dirperm1 breaks the protection by the permission bits on the lower branch
[  155.818299] aufs au_opts_verify:1597:dockerd[9815]: dirperm1 breaks the protection by the permission bits on the lower branch
[  155.858138] aufs au_opts_verify:1597:dockerd[3348]: dirperm1 breaks the protection by the permission bits on the lower branch
[  155.890452] aufs au_opts_verify:1597:dockerd[9475]: dirperm1 breaks the protection by the permission bits on the lower branch
[  155.950695] aufs au_opts_verify:1597:dockerd[4532]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:20:45Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/f2f3aee1e75d7bf72fcb55e951cedb443602b36a94d1e6ee489032b6d67c6cb8/shim.sock" debug=false module="containerd/tasks" pid=11345
[  156.163358] aufs au_opts_verify:1597:dockerd[3329]: dirperm1 breaks the protection by the permission bits on the lower branch
[  156.163886] aufs au_opts_verify:1597:dockerd[10464]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:20:45Z" level=info msg="shim reaped" id=e227edd391250ada7e68a97f3554366a979929c5ace3bbd0f34a1b15bd05af0d module="containerd/tasks"
[  156.192095] aufs au_opts_verify:1597:dockerd[4440]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:20:45.139550485Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T12:20:45Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/92e9b02be5d4c801a8fa849eec57fd01610317fe2a61f43bfae6c31eec4dd3f2/shim.sock" debug=false module="containerd/tasks" pid=11534
 time="2018-05-03T12:20:45Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/fb2809ffef46d84ee0e5cb1a7bc42c0d50f262892962a2aedd31bd6fcf5b43de/shim.sock" debug=false module="containerd/tasks" pid=11549
 time="2018-05-03T12:20:45Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/cb239003a4e5705c2376366ee33d15535b2ed39995abace7a090c19b64f067b2/shim.sock" debug=false module="containerd/tasks" pid=11564
 time="2018-05-03T12:20:45Z" level=info msg="shim reaped" id=cb239003a4e5705c2376366ee33d15535b2ed39995abace7a090c19b64f067b2 module="containerd/tasks"
 time="2018-05-03T12:20:45.864568389Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[  157.248851] aufs au_opts_verify:1597:dockerd[4465]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:20:46Z" level=info msg="shim reaped" id=92e9b02be5d4c801a8fa849eec57fd01610317fe2a61f43bfae6c31eec4dd3f2 module="containerd/tasks"
 time="2018-05-03T12:20:46.242389901Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[  157.303463] aufs au_opts_verify:1597:dockerd[4465]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:20:46Z" level=info msg="shim reaped" id=fb2809ffef46d84ee0e5cb1a7bc42c0d50f262892962a2aedd31bd6fcf5b43de module="containerd/tasks"
 time="2018-05-03T12:20:46.293935744Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[  157.406227] aufs au_opts_verify:1597:dockerd[3333]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:20:46Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/b752f08c0a5a0931326da147dbe9e9a53f34089265f5f787c675aa9a6c44b892/shim.sock" debug=false module="containerd/tasks" pid=12692
[  158.051907] aufs au_opts_verify:1597:dockerd[9475]: dirperm1 breaks the protection by the permission bits on the lower branch
[  158.089208] aufs au_opts_verify:1597:dockerd[9475]: dirperm1 breaks the protection by the permission bits on the lower branch
[  158.157777] aufs au_opts_verify:1597:dockerd[3333]: dirperm1 breaks the protection by the permission bits on the lower branch
[  167.159891] aufs au_opts_verify:1597:dockerd[11548]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:20:58Z" level=info msg="shim reaped" id=f2f3aee1e75d7bf72fcb55e951cedb443602b36a94d1e6ee489032b6d67c6cb8 module="containerd/tasks"
 time="2018-05-03T12:21:08.014300951Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[  179.081653] aufs au_opts_verify:1597:dockerd[3333]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:21:21Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/6201fc70104cb91542fa7820b1de372c097855c15f3a9a2d958b039c5a696404/shim.sock" debug=false module="containerd/tasks" pid=16330
 time="2018-05-03T12:22:27Z" level=info msg="shim reaped" id=b752f08c0a5a0931326da147dbe9e9a53f34089265f5f787c675aa9a6c44b892 module="containerd/tasks"
 time="2018-05-03T12:22:27.164550966Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T12:22:57.637004871Z" level=warning msg="Failed to remove bridge endpoint dfff152 from store: timeout"
 time="2018-05-03T12:24:01.548564064Z" level=warning msg="driver failed revoking external connectivity on endpoint frosty_edison (5ee3366e026c97d6aad592d43125798f2177fe6ae213bfe53704c42c69de9030): failed to update bridge endpoint 5ee3366 to store: failed to update bridge store for object type *bridge.bridgeEndpoint: timeout"
 time="2018-05-03T12:32:27Z" level=info msg="shim reaped" id=6201fc70104cb91542fa7820b1de372c097855c15f3a9a2d958b039c5a696404 module="containerd/tasks"
 time="2018-05-03T12:32:28.065619262Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 1212.977634] aufs au_opts_verify:1597:dockerd[4465]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1212.985499] aufs au_opts_verify:1597:dockerd[4507]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1213.168296] aufs au_opts_verify:1597:dockerd[4440]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1213.201052] aufs au_opts_verify:1597:dockerd[4561]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1222.718083] aufs au_opts_verify:1597:dockerd[4465]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1222.719501] aufs au_opts_verify:1597:dockerd[4561]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1222.724999] aufs au_opts_verify:1597:dockerd[4440]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1223.673502] aufs au_opts_verify:1597:dockerd[4440]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:38:45Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/8059579d4a3dac4662caffd622ba3447fb8ba0837bdba762ea15982a34b999a9/shim.sock" debug=false module="containerd/tasks" pid=19969
 time="2018-05-03T12:39:38Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/ce2067dafad5a43e282ba997d26b5f3fa1a48b8f40e839caaeeb649a2da755c7/shim.sock" debug=false module="containerd/tasks" pid=20188
 time="2018-05-03T12:40:33Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/576f305e407ca4a7b79fb98e7f7b32c8fb078129461479dbaef2b02806a9de28/shim.sock" debug=false module="containerd/tasks" pid=20392
[ 1378.754396] aufs au_opts_verify:1597:dockerd[3428]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1378.782306] aufs au_opts_verify:1597:dockerd[4556]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1378.791138] aufs au_opts_verify:1597:dockerd[4139]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:41:07Z" level=info msg="shim reaped" id=ce2067dafad5a43e282ba997d26b5f3fa1a48b8f40e839caaeeb649a2da755c7 module="containerd/tasks"
 time="2018-05-03T12:41:07Z" level=info msg="shim reaped" id=8059579d4a3dac4662caffd622ba3447fb8ba0837bdba762ea15982a34b999a9 module="containerd/tasks"
 time="2018-05-03T12:41:07.750518186Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T12:41:07.751598900Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 1378.851049] aufs au_opts_verify:1597:dockerd[3348]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1378.853413] aufs au_opts_verify:1597:dockerd[4468]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1378.905590] aufs au_opts_verify:1597:dockerd[4139]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1378.907240] aufs au_opts_verify:1597:dockerd[4468]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1378.918943] aufs au_opts_verify:1597:dockerd[4556]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:41:07Z" level=info msg="shim reaped" id=576f305e407ca4a7b79fb98e7f7b32c8fb078129461479dbaef2b02806a9de28 module="containerd/tasks"
 time="2018-05-03T12:41:07.895838247Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 1379.409865] aufs au_opts_verify:1597:dockerd[3346]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:41:08Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/09297ad1266c38112fa91451c1aeba29f94a320a5e64eecb4ecde4cfefede43f/shim.sock" debug=false module="containerd/tasks" pid=20818
[ 1379.488907] aufs au_opts_verify:1597:dockerd[4217]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.489596] aufs au_opts_verify:1597:dockerd[3346]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.523135] aufs au_opts_verify:1597:dockerd[4468]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.553384] aufs au_opts_verify:1597:dockerd[3406]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.576458] aufs au_opts_verify:1597:dockerd[4139]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.576981] aufs au_opts_verify:1597:dockerd[10370]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:41:08Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/c328c0c5256256445a83c2208df79b983eb9c260e778ebc26a70a4c5fca32e3e/shim.sock" debug=false module="containerd/tasks" pid=20917
 time="2018-05-03T12:41:08Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/69adbe0e43a0a68594c03276d764c80639090ed479208e63c3ff51a84fd40680/shim.sock" debug=false module="containerd/tasks" pid=20937
[ 1379.612506] aufs au_opts_verify:1597:dockerd[3346]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.636188] aufs au_opts_verify:1597:dockerd[4139]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.636657] aufs au_opts_verify:1597:dockerd[4450]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:41:08Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/f62eca4961c87162adf89653adba13c346ac23021903dbef6ef61fe9af1f6224/shim.sock" debug=false module="containerd/tasks" pid=21022
[ 1379.660105] aufs au_opts_verify:1597:dockerd[9076]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.660596] aufs au_opts_verify:1597:dockerd[3406]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.700206] aufs au_opts_verify:1597:dockerd[10370]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.700639] aufs au_opts_verify:1597:dockerd[4139]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.709506] aufs au_opts_verify:1597:dockerd[4483]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.768229] aufs au_opts_verify:1597:dockerd[3346]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.768714] aufs au_opts_verify:1597:dockerd[4450]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.769186] aufs au_opts_verify:1597:dockerd[3348]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:41:08Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/4e6af0c3a060ec819eb75cc18fb3a48aa96d6446aacb8e1db8c5cdf0ad318ff5/shim.sock" debug=false module="containerd/tasks" pid=21337
 time="2018-05-03T12:41:08Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/2a37667e4e17106b503137d77fdacc90ac6bb4a4c1424f847cb6ec55248f0aa7/shim.sock" debug=false module="containerd/tasks" pid=21355
[ 1379.861948] aufs au_opts_verify:1597:dockerd[3346]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.880696] aufs au_opts_verify:1597:dockerd[4216]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.881208] aufs au_opts_verify:1597:dockerd[3406]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.908039] aufs au_opts_verify:1597:dockerd[4450]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.928463] aufs au_opts_verify:1597:dockerd[4216]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.929806] aufs au_opts_verify:1597:dockerd[4440]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.930317] aufs au_opts_verify:1597:dockerd[4216]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.959715] aufs au_opts_verify:1597:dockerd[3406]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.979409] aufs au_opts_verify:1597:dockerd[3406]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.979873] aufs au_opts_verify:1597:dockerd[3348]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1380.008926] aufs au_opts_verify:1597:dockerd[3333]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1380.012334] aufs au_opts_verify:1597:dockerd[4450]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:41:08Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/2409464b0e1d05204777991d6f38cd33e43baa27a4d94b60809d36411f344d15/shim.sock" debug=false module="containerd/tasks" pid=21527
[ 1380.059511] aufs au_opts_verify:1597:dockerd[3406]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1380.059752] aufs au_opts_verify:1597:dockerd[3406]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1380.124273] aufs au_opts_verify:1597:dockerd[3333]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1380.152168] aufs au_opts_verify:1597:dockerd[4451]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1380.153079] aufs au_opts_verify:1597:dockerd[4216]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1380.153500] aufs au_opts_verify:1597:dockerd[4451]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:41:09Z" level=info msg="shim reaped" id=09297ad1266c38112fa91451c1aeba29f94a320a5e64eecb4ecde4cfefede43f module="containerd/tasks"
 time="2018-05-03T12:41:09Z" level=info msg="shim reaped" id=c328c0c5256256445a83c2208df79b983eb9c260e778ebc26a70a4c5fca32e3e module="containerd/tasks"
[ 1380.163374] aufs au_opts_verify:1597:dockerd[3406]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:41:09.081396504Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T12:41:09Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/cabe9ad6086046f00f2e8f87107f50a1013b25867f68ab16ac987cda32ee7e31/shim.sock" debug=false module="containerd/tasks" pid=21771
 time="2018-05-03T12:41:09.089590325Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 1380.288256] aufs au_opts_verify:1597:dockerd[3329]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1380.298497] aufs au_opts_verify:1597:dockerd[9815]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:41:09Z" level=info msg="shim reaped" id=69adbe0e43a0a68594c03276d764c80639090ed479208e63c3ff51a84fd40680 module="containerd/tasks"
 time="2018-05-03T12:41:09.243156513Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T12:41:09Z" level=info msg="shim reaped" id=f62eca4961c87162adf89653adba13c346ac23021903dbef6ef61fe9af1f6224 module="containerd/tasks"
 time="2018-05-03T12:41:09.309550812Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 1380.867928] aufs au_opts_verify:1597:dockerd[9824]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1380.971829] aufs au_opts_verify:1597:dockerd[9824]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1381.156442] aufs au_opts_verify:1597:dockerd[4483]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1381.324949] aufs au_opts_verify:1597:dockerd[4466]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1381.380544] aufs au_opts_verify:1597:dockerd[4438]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1381.932019] aufs au_opts_verify:1597:dockerd[4216]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1382.299501] aufs au_opts_verify:1597:dockerd[3348]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1382.455690] aufs au_opts_verify:1597:dockerd[4438]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:41:11Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/863ed3cdd2b39dce772f5e318a8ae9cf7ce6768308ee2a86cfacdefd2d2bd6e3/shim.sock" debug=false module="containerd/tasks" pid=22330
[ 1382.488160] aufs au_opts_verify:1597:dockerd[3428]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1382.488878] aufs au_opts_verify:1597:dockerd[3348]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1382.542161] aufs au_opts_verify:1597:dockerd[3348]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:41:11Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/0a845d16c6057edaa75a541923c69fe57c66ef2fe0657ca7a8689c721a97ee9e/shim.sock" debug=false module="containerd/tasks" pid=22433
[ 1382.570087] aufs au_opts_verify:1597:dockerd[3406]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:41:11Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/54c266c226c1eef82d481d82c5c0fb0fa6e8c1d96bff0f123a8a037e1995da41/shim.sock" debug=false module="containerd/tasks" pid=22471
 time="2018-05-03T12:43:24Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/4929d8e75a56ca2e2d8abfe94f8245bd4f84a19309018b177df9db0c0125e910/shim.sock" debug=false module="containerd/tasks" pid=22762
 time="2018-05-03T12:45:20Z" level=info msg="shim reaped" id=4929d8e75a56ca2e2d8abfe94f8245bd4f84a19309018b177df9db0c0125e910 module="containerd/tasks"
 time="2018-05-03T12:45:21.085680521Z" level=error msg="stream copy error: reading from a closed fifo"
 time="2018-05-03T12:45:21.085684731Z" level=error msg="stream copy error: reading from a closed fifo"
 time="2018-05-03T12:52:43.924774337Z" level=warning msg="Failed to remove bridge endpoint c61c5de from store: timeout"
 time="2018-05-03T12:56:11Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/631ca9eadfccdfc43f1537451f4c978a7c20bd7b792f8c1683c3018be1cd123e/shim.sock" debug=false module="containerd/tasks" pid=23228
 time="2018-05-03T12:56:32.730379177Z" level=warning msg="Could not rollback container connection to network bridge"
 time="2018-05-03T12:56:41.930879110Z" level=error msg="53fd32c54195dcb961a0c5eba182c0bd9088dd6c7043511e6f5b66153fec8469 cleanup: failed to delete container from containerd: no such container"
 time="2018-05-03T12:56:41.934737735Z" level=error msg="Handler for POST /containers/53fd32c54195dcb961a0c5eba182c0bd9088dd6c7043511e6f5b66153fec8469/start returned error: failed to create endpoint practical_fermi on network bridge: failed to save bridge endpoint d4455f1 to store: failed to update bridge store for object type *bridge.bridgeEndpoint: timeout"
 time="2018-05-03T12:57:33Z" level=info msg="shim reaped" id=631ca9eadfccdfc43f1537451f4c978a7c20bd7b792f8c1683c3018be1cd123e module="containerd/tasks"
 time="2018-05-03T12:57:33.594668367Z" level=error msg="stream copy error: reading from a closed fifo"
 time="2018-05-03T12:57:33.594678064Z" level=error msg="stream copy error: reading from a closed fifo"
 time="2018-05-03T12:58:29.298547311Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"
[ 2420.401941] aufs au_opts_verify:1597:dockerd[10464]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T12:58:29.334262030Z" level=error msg="Handler for POST /containers/f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144/start returned error: failed to update the store state of sandbox: failed to update store for object type *libnetwork.sbState: timeout"
[ 2481.125782] aufs au_opts_verify:1597:dockerd[10370]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:04:09.562807189Z" level=error msg="455b514149e4e6d6f2ac9f83e18794790f35bae4a2be651eeaf0ad417d0c5565 cleanup: failed to delete container from containerd: no such container"
 time="2018-05-03T13:04:09.629206028Z" level=error msg="Handler for POST /containers/455b514149e4e6d6f2ac9f83e18794790f35bae4a2be651eeaf0ad417d0c5565/start returned error: failed to update store for object type *libnetwork.endpoint: timeout"
[ 2824.225697] aufs au_opts_verify:1597:dockerd[4438]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:05:22Z" level=info msg="shim reaped" id=4e6af0c3a060ec819eb75cc18fb3a48aa96d6446aacb8e1db8c5cdf0ad318ff5 module="containerd/tasks"
 time="2018-05-03T13:05:22Z" level=info msg="shim reaped" id=863ed3cdd2b39dce772f5e318a8ae9cf7ce6768308ee2a86cfacdefd2d2bd6e3 module="containerd/tasks"
 time="2018-05-03T13:05:22Z" level=info msg="shim reaped" id=cabe9ad6086046f00f2e8f87107f50a1013b25867f68ab16ac987cda32ee7e31 module="containerd/tasks"
 time="2018-05-03T13:05:22Z" level=info msg="shim reaped" id=2a37667e4e17106b503137d77fdacc90ac6bb4a4c1424f847cb6ec55248f0aa7 module="containerd/tasks"
 time="2018-05-03T13:05:22Z" level=info msg="shim reaped" id=2409464b0e1d05204777991d6f38cd33e43baa27a4d94b60809d36411f344d15 module="containerd/tasks"
 time="2018-05-03T13:05:22Z" level=info msg="shim reaped" id=54c266c226c1eef82d481d82c5c0fb0fa6e8c1d96bff0f123a8a037e1995da41 module="containerd/tasks"
 time="2018-05-03T13:05:22Z" level=info msg="shim reaped" id=0a845d16c6057edaa75a541923c69fe57c66ef2fe0657ca7a8689c721a97ee9e module="containerd/tasks"
 time="2018-05-03T13:05:22.718232461Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:22.723032603Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:22.732704995Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:22.737503001Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:22.739781128Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:22.739992401Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:22.743739512Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:24.922085385Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"
 time="2018-05-03T13:05:24.981522266Z" level=error msg="4929d8e75a56ca2e2d8abfe94f8245bd4f84a19309018b177df9db0c0125e910 cleanup: failed to delete container from containerd: no such container"
 time="2018-05-03T13:05:24.986465355Z" level=error msg="Handler for POST /containers/4929d8e75a56ca2e2d8abfe94f8245bd4f84a19309018b177df9db0c0125e910/start returned error: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:402: container init caused \\\"process_linux.go:385: running prestart hook 0 caused \\\\\\\"error running hook: exit status 1, stdout: , stderr: time=\\\\\\\\\\\\\\\"2018-05-03T12:45:20Z\\\\\\\\\\\\\\\" level=fatal msg=\\\\\\\\\\\\\\\"failed to update store for object type *libnetwork.sbState: timeout\\\\\\\\\\\\\\\"\\\\\\\\n\\\\\\\"\\\"\": unknown"
 time="2018-05-03T13:05:25Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/3fd7c0fa95e8a1740c7a7acee4303fe29ad3f2d13c8f617c3b041eb59f2909cc/shim.sock" debug=false module="containerd/tasks" pid=23860
 time="2018-05-03T13:05:25Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/c0bae577c72fb15a6762d8a252fe11ba1457fa5c20e22643e838d43b846478da/shim.sock" debug=false module="containerd/tasks" pid=23861
 time="2018-05-03T13:05:25.281302482Z" level=error msg="631ca9eadfccdfc43f1537451f4c978a7c20bd7b792f8c1683c3018be1cd123e cleanup: failed to delete container from containerd: no such container"
 time="2018-05-03T13:05:25.281357469Z" level=error msg="Handler for POST /containers/631ca9eadfccdfc43f1537451f4c978a7c20bd7b792f8c1683c3018be1cd123e/start returned error: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:402: container init caused \\\"process_linux.go:385: running prestart hook 0 caused \\\\\\\"error running hook: exit status 1, stdout: , stderr: time=\\\\\\\\\\\\\\\"2018-05-03T12:57:32Z\\\\\\\\\\\\\\\" level=fatal msg=\\\\\\\\\\\\\\\"failed to update store for object type *libnetwork.sbState: timeout\\\\\\\\\\\\\\\"\\\\\\\\n\\\\\\\"\\\"\": unknown"
 time="2018-05-03T13:05:25Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/60d4c30d5fa1be2bdcf552a523c90d17b9cff23eaa2ec0117d9752e27f64bba9/shim.sock" debug=false module="containerd/tasks" pid=23887
 time="2018-05-03T13:05:26Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/5cb33d794fc97bda5d47d9b4c3cfac40ac845ad2e64512cb6cb7bab4f7fe420b/shim.sock" debug=false module="containerd/tasks" pid=24229
 time="2018-05-03T13:05:26Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/53fd32c54195dcb961a0c5eba182c0bd9088dd6c7043511e6f5b66153fec8469/shim.sock" debug=false module="containerd/tasks" pid=24244
 time="2018-05-03T13:05:26Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/455b514149e4e6d6f2ac9f83e18794790f35bae4a2be651eeaf0ad417d0c5565/shim.sock" debug=false module="containerd/tasks" pid=24252
 time="2018-05-03T13:05:28Z" level=info msg="shim reaped" id=5cb33d794fc97bda5d47d9b4c3cfac40ac845ad2e64512cb6cb7bab4f7fe420b module="containerd/tasks"
 time="2018-05-03T13:05:28Z" level=info msg="shim reaped" id=60d4c30d5fa1be2bdcf552a523c90d17b9cff23eaa2ec0117d9752e27f64bba9 module="containerd/tasks"
 time="2018-05-03T13:05:28Z" level=info msg="shim reaped" id=3fd7c0fa95e8a1740c7a7acee4303fe29ad3f2d13c8f617c3b041eb59f2909cc module="containerd/tasks"
 time="2018-05-03T13:05:28Z" level=info msg="shim reaped" id=c0bae577c72fb15a6762d8a252fe11ba1457fa5c20e22643e838d43b846478da module="containerd/tasks"
 time="2018-05-03T13:05:28.143305737Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:28Z" level=info msg="shim reaped" id=455b514149e4e6d6f2ac9f83e18794790f35bae4a2be651eeaf0ad417d0c5565 module="containerd/tasks"
 time="2018-05-03T13:05:28.145919096Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:28Z" level=info msg="shim reaped" id=53fd32c54195dcb961a0c5eba182c0bd9088dd6c7043511e6f5b66153fec8469 module="containerd/tasks"
 time="2018-05-03T13:05:28.148071974Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:28.152767470Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:28.158309261Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:28.158373997Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 2839.943261] aufs au_opts_verify:1597:dockerd[9824]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2839.993879] aufs au_opts_verify:1597:dockerd[9824]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2840.041513] aufs au_opts_verify:1597:dockerd[9824]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:05:28Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/106ec50c4c12f2b948a4827d0ff0e1838295f491bc09887f36863d564909f94f/shim.sock" debug=false module="containerd/tasks" pid=25041
[ 2840.591499] aufs au_opts_verify:1597:dockerd[4217]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2840.624314] aufs au_opts_verify:1597:dockerd[3346]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2840.661426] aufs au_opts_verify:1597:dockerd[3346]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2840.662053] aufs au_opts_verify:1597:dockerd[4553]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2840.728855] aufs au_opts_verify:1597:dockerd[4556]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2840.729300] aufs au_opts_verify:1597:dockerd[3404]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2840.760177] aufs au_opts_verify:1597:dockerd[10548]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:05:29Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/d8083e074a477f8918abbb888b32b43c3b3bab844fb3dcf56a4b51d416fb4bc1/shim.sock" debug=false module="containerd/tasks" pid=25199
[ 2840.793708] aufs au_opts_verify:1597:dockerd[10199]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:05:29Z" level=info msg="shim reaped" id=106ec50c4c12f2b948a4827d0ff0e1838295f491bc09887f36863d564909f94f module="containerd/tasks"
[ 2840.824784] aufs au_opts_verify:1597:dockerd[10199]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:05:29.735931372Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:29Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/32a7db2ff75e9564522ffe66e87e9548b26be1abcd244c7e9ae43d68bd657fbd/shim.sock" debug=false module="containerd/tasks" pid=25310
[ 2840.837456] aufs au_opts_verify:1597:dockerd[10548]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2840.868224] aufs au_opts_verify:1597:dockerd[10199]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2840.916801] aufs au_opts_verify:1597:dockerd[10548]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2840.917439] aufs au_opts_verify:1597:dockerd[9076]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:05:29Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/1809be1799f5ac14d299eaaa7e72616d17e25bb81dbe9f56524228136b073680/shim.sock" debug=false module="containerd/tasks" pid=25404
[ 2841.083373] aufs au_opts_verify:1597:dockerd[3404]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2841.083952] aufs au_opts_verify:1597:dockerd[4216]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2841.101359] aufs au_opts_verify:1597:dockerd[4217]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2841.103338] aufs au_opts_verify:1597:dockerd[4467]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:05:30Z" level=info msg="shim reaped" id=d8083e074a477f8918abbb888b32b43c3b3bab844fb3dcf56a4b51d416fb4bc1 module="containerd/tasks"
[ 2841.165140] aufs au_opts_verify:1597:dockerd[4467]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2841.165367] aufs au_opts_verify:1597:dockerd[4467]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:05:30.086709122Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 2841.184674] aufs au_opts_verify:1597:dockerd[10548]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2841.206312] aufs au_opts_verify:1597:dockerd[9076]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2841.229293] aufs au_opts_verify:1597:dockerd[4556]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2841.243856] aufs au_opts_verify:1597:dockerd[9076]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2841.282163] aufs au_opts_verify:1597:dockerd[10548]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:05:30Z" level=info msg="shim reaped" id=32a7db2ff75e9564522ffe66e87e9548b26be1abcd244c7e9ae43d68bd657fbd module="containerd/tasks"
 time="2018-05-03T13:05:30.216258406Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:30Z" level=info msg="shim reaped" id=1809be1799f5ac14d299eaaa7e72616d17e25bb81dbe9f56524228136b073680 module="containerd/tasks"
 time="2018-05-03T13:05:30.390560846Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:30Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/f6b55fc3c592db3b5102dec234ff9a8ae12237e2c5ebece365dc809031423bbb/shim.sock" debug=false module="containerd/tasks" pid=25794
[ 2841.565448] aufs au_opts_verify:1597:dockerd[10548]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:05:30Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/88ad1363242aba92d1dacfb4a3105c4a41c410bff74c855c7ea78a518829a142/shim.sock" debug=false module="containerd/tasks" pid=25860
[ 2841.626850] aufs au_opts_verify:1597:dockerd[10548]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:05:30Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/6acbc487be0926aae2de82c8114da4c5c84c52f8aac078d2081ea8eabdb823a9/shim.sock" debug=false module="containerd/tasks" pid=25875
[ 2841.671400] aufs au_opts_verify:1597:dockerd[3406]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:05:30Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/95e37c5daee30c0a09be118877a7f5051345b55b737cbbe809e8a12aff2ecf60/shim.sock" debug=false module="containerd/tasks" pid=26043
 time="2018-05-03T13:05:30Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/8d5c7b01b44ba99198dff801a03483ce9c3f450959272f2a8fb73583e74f85dd/shim.sock" debug=false module="containerd/tasks" pid=26174
 time="2018-05-03T13:05:30Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/a073921268b279cf6e4f95cee7680c3a442f2aef9ddf0e410e1d4a453ac37216/shim.sock" debug=false module="containerd/tasks" pid=26179
 time="2018-05-03T13:05:31Z" level=info msg="shim reaped" id=f6b55fc3c592db3b5102dec234ff9a8ae12237e2c5ebece365dc809031423bbb module="containerd/tasks"
 time="2018-05-03T13:05:31.249550819Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:31Z" level=info msg="shim reaped" id=6acbc487be0926aae2de82c8114da4c5c84c52f8aac078d2081ea8eabdb823a9 module="containerd/tasks"
 time="2018-05-03T13:05:31.277938818Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:31Z" level=info msg="shim reaped" id=88ad1363242aba92d1dacfb4a3105c4a41c410bff74c855c7ea78a518829a142 module="containerd/tasks"
 time="2018-05-03T13:05:31.354977244Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:31Z" level=info msg="shim reaped" id=95e37c5daee30c0a09be118877a7f5051345b55b737cbbe809e8a12aff2ecf60 module="containerd/tasks"
 time="2018-05-03T13:05:31.705973417Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 2842.858578] aufs au_opts_verify:1597:dockerd[3346]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:05:31Z" level=info msg="shim reaped" id=8d5c7b01b44ba99198dff801a03483ce9c3f450959272f2a8fb73583e74f85dd module="containerd/tasks"
 time="2018-05-03T13:05:31.779888691Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:31Z" level=info msg="shim reaped" id=a073921268b279cf6e4f95cee7680c3a442f2aef9ddf0e410e1d4a453ac37216 module="containerd/tasks"
[ 2842.893561] aufs au_opts_verify:1597:dockerd[9076]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:05:31.809327571Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 2842.932558] aufs au_opts_verify:1597:dockerd[9076]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2842.953433] aufs au_opts_verify:1597:dockerd[9076]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.005402] aufs au_opts_verify:1597:dockerd[9076]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.005615] aufs au_opts_verify:1597:dockerd[9076]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.084807] aufs au_opts_verify:1597:dockerd[10548]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.157128] aufs au_opts_verify:1597:dockerd[4451]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.202115] aufs au_opts_verify:1597:dockerd[4451]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.277667] aufs au_opts_verify:1597:dockerd[4438]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.278244] aufs au_opts_verify:1597:dockerd[9076]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.278740] aufs au_opts_verify:1597:dockerd[4553]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.350171] aufs au_opts_verify:1597:dockerd[4553]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.350698] aufs au_opts_verify:1597:dockerd[4553]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.351271] aufs au_opts_verify:1597:dockerd[9076]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.401704] aufs au_opts_verify:1597:dockerd[4553]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:05:32Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/a150a284f2e4d7c13e5a01a677beed481c9476675b112242dc83cd15fd79e9fc/shim.sock" debug=false module="containerd/tasks" pid=26991
[ 2843.440154] aufs au_opts_verify:1597:dockerd[10548]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.450104] aufs au_opts_verify:1597:dockerd[3346]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:05:32Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/3b0445e0f42c528c3f171cdcce12ef266ee1316439590a6ca404990afa55e20c/shim.sock" debug=false module="containerd/tasks" pid=27053
 time="2018-05-03T13:05:32Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/61f470a5765d2bc3e00e2e63d8f5c05c4c7e495a97e8581d40fec26564104b5e/shim.sock" debug=false module="containerd/tasks" pid=27100
 time="2018-05-03T13:05:32Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/3704e7e21501a702889c37174e2776205d3cb4cba7a2e0e245b53d7b129c978f/shim.sock" debug=false module="containerd/tasks" pid=27143
 time="2018-05-03T13:05:32Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/4ee08b443e2d4842de3f71e68cb3741a33f8d480b0ecb7f11f4d27c12c048f16/shim.sock" debug=false module="containerd/tasks" pid=27162
 time="2018-05-03T13:05:32Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/7d7aa80cd7243185ceb02008d180cd9a05ac5424e38c40d1a4cbfb9531e9912d/shim.sock" debug=false module="containerd/tasks" pid=27167
 time="2018-05-03T13:05:33Z" level=info msg="shim reaped" id=a150a284f2e4d7c13e5a01a677beed481c9476675b112242dc83cd15fd79e9fc module="containerd/tasks"
 time="2018-05-03T13:05:33.131708056Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:33Z" level=info msg="shim reaped" id=3704e7e21501a702889c37174e2776205d3cb4cba7a2e0e245b53d7b129c978f module="containerd/tasks"
 time="2018-05-03T13:05:33.210302701Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:33Z" level=info msg="shim reaped" id=61f470a5765d2bc3e00e2e63d8f5c05c4c7e495a97e8581d40fec26564104b5e module="containerd/tasks"
 time="2018-05-03T13:05:33.288034359Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:33Z" level=info msg="shim reaped" id=7d7aa80cd7243185ceb02008d180cd9a05ac5424e38c40d1a4cbfb9531e9912d module="containerd/tasks"
 time="2018-05-03T13:05:33Z" level=info msg="shim reaped" id=4ee08b443e2d4842de3f71e68cb3741a33f8d480b0ecb7f11f4d27c12c048f16 module="containerd/tasks"
 time="2018-05-03T13:05:33Z" level=info msg="shim reaped" id=3b0445e0f42c528c3f171cdcce12ef266ee1316439590a6ca404990afa55e20c module="containerd/tasks"
 time="2018-05-03T13:05:33.342540748Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:33.343263591Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:05:33.343715988Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 2844.722691] aufs au_opts_verify:1597:dockerd[4561]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2844.786017] aufs au_opts_verify:1597:dockerd[10370]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2844.828578] aufs au_opts_verify:1597:dockerd[4532]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:05:33Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/e9512f9a0e6305712e944bf79d2c18cc4dba2e86c0ab867b2d8431ad7b6e1910/shim.sock" debug=false module="containerd/tasks" pid=27949
 time="2018-05-03T13:05:34Z" level=info msg="shim reaped" id=e9512f9a0e6305712e944bf79d2c18cc4dba2e86c0ab867b2d8431ad7b6e1910 module="containerd/tasks"
 time="2018-05-03T13:05:34.609214165Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 2896.528035] aufs au_opts_verify:1597:dockerd[10199]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2896.846865] aufs au_opts_verify:1597:dockerd[4483]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:06:32Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/631ca9eadfccdfc43f1537451f4c978a7c20bd7b792f8c1683c3018be1cd123e/shim.sock" debug=false module="containerd/tasks" pid=28345
 time="2018-05-03T13:06:32Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/4929d8e75a56ca2e2d8abfe94f8245bd4f84a19309018b177df9db0c0125e910/shim.sock" debug=false module="containerd/tasks" pid=28346
[ 2955.720124] aufs au_opts_verify:1597:dockerd[3404]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:07:27.683705899Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"
[ 3079.979644] aufs au_opts_verify:1597:dockerd[28486]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3082.613489] aufs au_opts_verify:1597:dockerd[28486]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3087.228776] aufs au_opts_verify:1597:dockerd[3404]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:09:45Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/24d7d21c2576e8ec2199461ef5d5ed8df3ea6d5e0a8c5a7defc56505532e215b/shim.sock" debug=false module="containerd/tasks" pid=28778
[ 3198.490768] aufs au_opts_verify:1597:dockerd[3346]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:11:27.894633572Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"
 time="2018-05-03T13:18:12Z" level=info msg="shim reaped" id=24d7d21c2576e8ec2199461ef5d5ed8df3ea6d5e0a8c5a7defc56505532e215b module="containerd/tasks"
 time="2018-05-03T13:18:12.455765863Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 3606.677118] aufs au_opts_verify:1597:dockerd[4216]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3607.809325] aufs au_opts_verify:1597:dockerd[4216]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3607.975205] aufs au_opts_verify:1597:dockerd[10370]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:18:17Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/67acef3ec8ff73639a8d36133a2c19be84d20236c9d71776a98847a541be732e/shim.sock" debug=false module="containerd/tasks" pid=29117
 time="2018-05-03T13:18:20Z" level=info msg="shim reaped" id=67acef3ec8ff73639a8d36133a2c19be84d20236c9d71776a98847a541be732e module="containerd/tasks"
 time="2018-05-03T13:18:20.470768902Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 3612.685662] aufs au_opts_verify:1597:dockerd[4502]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3612.913251] aufs au_opts_verify:1597:dockerd[4502]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3613.214634] aufs au_opts_verify:1597:dockerd[3406]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:18:22Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/36783f320c5e2ee7f90d0c92d9abf2c0812be019b1270e40e804b7c1753aa090/shim.sock" debug=false module="containerd/tasks" pid=29302
 time="2018-05-03T13:18:23Z" level=info msg="shim reaped" id=36783f320c5e2ee7f90d0c92d9abf2c0812be019b1270e40e804b7c1753aa090 module="containerd/tasks"
 time="2018-05-03T13:18:23.889381131Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 3615.976256] aufs au_opts_verify:1597:dockerd[10370]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3616.278263] aufs au_opts_verify:1597:dockerd[10370]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3616.332718] aufs au_opts_verify:1597:dockerd[4216]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:18:25Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/a4e96f4099a736507693cd40444c1dbf8129a2d058295a04a89c7bef3364f404/shim.sock" debug=false module="containerd/tasks" pid=29491
 time="2018-05-03T13:18:27Z" level=info msg="shim reaped" id=a4e96f4099a736507693cd40444c1dbf8129a2d058295a04a89c7bef3364f404 module="containerd/tasks"
 time="2018-05-03T13:18:27.220928709Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 3619.539129] aufs au_opts_verify:1597:dockerd[10464]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3619.782189] aufs au_opts_verify:1597:dockerd[10464]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3619.832195] aufs au_opts_verify:1597:dockerd[4216]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:18:29Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/48a80757c4692226a6ce67e1beb0542330c1b742e2fb7cb230df21db863a1afb/shim.sock" debug=false module="containerd/tasks" pid=29683
 time="2018-05-03T13:18:30Z" level=info msg="shim reaped" id=48a80757c4692226a6ce67e1beb0542330c1b742e2fb7cb230df21db863a1afb module="containerd/tasks"
 time="2018-05-03T13:18:30.669794951Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 3679.032472] aufs au_opts_verify:1597:dockerd[10464]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:19:28.174034845Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"
 time="2018-05-03T13:34:50Z" level=info msg="shim reaped" id=631ca9eadfccdfc43f1537451f4c978a7c20bd7b792f8c1683c3018be1cd123e module="containerd/tasks"
 time="2018-05-03T13:34:50Z" level=info msg="shim reaped" id=4929d8e75a56ca2e2d8abfe94f8245bd4f84a19309018b177df9db0c0125e910 module="containerd/tasks"
 time="2018-05-03T13:34:50.719177543Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:34:50.726367010Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 4603.089720] aufs au_opts_verify:1597:dockerd[4217]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4603.128367] aufs au_opts_verify:1597:dockerd[28416]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4603.157483] aufs au_opts_verify:1597:dockerd[28416]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4603.177768] aufs au_opts_verify:1597:dockerd[4217]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4603.202339] aufs au_opts_verify:1597:dockerd[4217]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4603.214106] aufs au_opts_verify:1597:dockerd[28416]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:34:52Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/da55011ecbcd7581d812b8a92d5072abb613b62b893a49e9b12238ad5363fda7/shim.sock" debug=false module="containerd/tasks" pid=30201
 time="2018-05-03T13:34:52Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/358316942a7d744a69dcfdd34ca4beac0a737eebd3264cd349fd7b3c8ec3b5d0/shim.sock" debug=false module="containerd/tasks" pid=30202
 time="2018-05-03T13:34:54Z" level=info msg="shim reaped" id=358316942a7d744a69dcfdd34ca4beac0a737eebd3264cd349fd7b3c8ec3b5d0 module="containerd/tasks"
 time="2018-05-03T13:34:54Z" level=info msg="shim reaped" id=da55011ecbcd7581d812b8a92d5072abb613b62b893a49e9b12238ad5363fda7 module="containerd/tasks"
 time="2018-05-03T13:34:54.669952996Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:34:54.671912740Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 4606.038885] aufs au_opts_verify:1597:dockerd[9510]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4606.074956] aufs au_opts_verify:1597:dockerd[4468]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4606.105321] aufs au_opts_verify:1597:dockerd[9510]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4606.133173] aufs au_opts_verify:1597:dockerd[9510]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4606.149764] aufs au_opts_verify:1597:dockerd[9510]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4606.159521] aufs au_opts_verify:1597:dockerd[4468]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:34:55Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/04f6c2f50cb959dfbd87e6482f1b57edcfd2de50ae59e989b59759d5c8ded896/shim.sock" debug=false module="containerd/tasks" pid=30556
 time="2018-05-03T13:34:55Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/4b0c4ca425ae63aec65981b91049ded677e77a774a79ec42ab9cf82f58a53472/shim.sock" debug=false module="containerd/tasks" pid=30561
 time="2018-05-03T13:34:55Z" level=info msg="shim reaped" id=4b0c4ca425ae63aec65981b91049ded677e77a774a79ec42ab9cf82f58a53472 module="containerd/tasks"
 time="2018-05-03T13:34:55.741303482Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 time="2018-05-03T13:34:55Z" level=info msg="shim reaped" id=04f6c2f50cb959dfbd87e6482f1b57edcfd2de50ae59e989b59759d5c8ded896 module="containerd/tasks"
 time="2018-05-03T13:34:55.770089518Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 4639.417246] aufs au_opts_verify:1597:dockerd[4217]: dirperm1 breaks the protection by the permission bits on the lower branch
 time="2018-05-03T13:35:28.725226613Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"
 time="2018-05-03T13:36:34.737944275Z" level=info msg="Processing signal 'terminated'"

We can see here that there were 5 containers that were created successfully. 4 of them each failed to start once (500 error code on the client) and then succeeded and one of them failed to start the first time with 500 error code and then failed to start 5 times with 403.

Here is the filtered version of the above logs with only the relevant parts:

 time="2018-05-03T12:56:41.930879110Z" level=error msg="53fd32c54195dcb961a0c5eba182c0bd9088dd6c7043511e6f5b66153fec8469 cleanup: failed to delete container from containerd: no such container"
 time="2018-05-03T12:56:41.934737735Z" level=error msg="Handler for POST /containers/53fd32c54195dcb961a0c5eba182c0bd9088dd6c7043511e6f5b66153fec8469/start returned error: failed to create endpoint practical_fermi on network bridge: failed to save bridge endpoint d4455f1 to store: failed to update bridge store for object type *bridge.bridgeEndpoint: timeout"
 time="2018-05-03T12:58:29.298547311Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"
 time="2018-05-03T12:58:29.334262030Z" level=error msg="Handler for POST /containers/f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144/start returned error: failed to update the store state of sandbox: failed to update store for object type *libnetwork.sbState: timeout"
 time="2018-05-03T13:05:24.922085385Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"
 time="2018-05-03T13:05:24.981522266Z" level=error msg="4929d8e75a56ca2e2d8abfe94f8245bd4f84a19309018b177df9db0c0125e910 cleanup: failed to delete container from containerd: no such container"
 time="2018-05-03T13:05:24.986465355Z" level=error msg="Handler for POST /containers/4929d8e75a56ca2e2d8abfe94f8245bd4f84a19309018b177df9db0c0125e910/start returned error: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:402: container init caused \\\"process_linux.go:385: running prestart hook 0 caused \\\\\\\"error running hook: exit status 1, stdout: , stderr: time=\\\\\\\\\\\\\\\"2018-05-03T12:45:20Z\\\\\\\\\\\\\\\" level=fatal msg=\\\\\\\\\\\\\\\"failed to update store for object type *libnetwork.sbState: timeout\\\\\\\\\\\\\\\"\\\\\\\\n\\\\\\\"\\\"\": unknown"
 time="2018-05-03T13:05:25.281302482Z" level=error msg="631ca9eadfccdfc43f1537451f4c978a7c20bd7b792f8c1683c3018be1cd123e cleanup: failed to delete container from containerd: no such container"
 time="2018-05-03T13:05:25.281357469Z" level=error msg="Handler for POST /containers/631ca9eadfccdfc43f1537451f4c978a7c20bd7b792f8c1683c3018be1cd123e/start returned error: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:402: container init caused \\\"process_linux.go:385: running prestart hook 0 caused \\\\\\\"error running hook: exit status 1, stdout: , stderr: time=\\\\\\\\\\\\\\\"2018-05-03T12:57:32Z\\\\\\\\\\\\\\\" level=fatal msg=\\\\\\\\\\\\\\\"failed to update store for object type *libnetwork.sbState: timeout\\\\\\\\\\\\\\\"\\\\\\\\n\\\\\\\"\\\"\": unknown"
 time="2018-05-03T13:07:27.683705899Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"
 time="2018-05-03T13:11:27.894633572Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"
 time="2018-05-03T13:19:28.174034845Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"
 time="2018-05-03T13:35:28.725226613Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"

Output of docker version:

Client:
 Version:	17.12.1-ce
 API version:	1.35
 Go version:	go1.9.4
 Git commit:	7390fc6
 Built:	Tue Feb 27 22:17:40 2018
 OS/Arch:	linux/amd64

Server:
 Engine:
  Version:	17.12.1-ce
  API version:	1.35 (minimum version 1.12)
  Go version:	go1.9.4
  Git commit:	7390fc6
  Built:	Tue Feb 27 22:16:13 2018
  OS/Arch:	linux/amd64
  Experimental:	false

Output of docker info:
It's not the exact machine on which this error occurred, but one with same configuration, i.e. difference is in number of containers and images, but those are also quite similar.

Containers: 34
 Running: 2
 Paused: 0
 Stopped: 32
Images: 6
Server Version: 17.12.1-ce
Storage Driver: aufs
 Root Dir: /mnt/docker/aufs
 Backing Filesystem: extfs
 Dirs: 119
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9b55aab90508bd389d7654c4baf173a981477d55
runc version: 9f9c96235cc97674e935002fc3d78361b696a69e
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-1052-aws
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 36
Total Memory: 58.97GiB
Name: i-0d076c493fc5e0244
ID: RDNQ:6SAC:BZ4U:REOE:KTVI:DB3Z:RDJA:ZRA2:IUE6:GX5K:BFBR:EDOW
Docker Root Dir: /mnt/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.)
AWS box

Related issues
#235 is similar, but not sure if related
@daniloradenovic daniloradenovic mentioned this issue May 4, 2018
Closed
3 tasks
@thaJeztah
Copy link
Member

ping @ctelfer @fcrisciani - was anything around this fixed in a more recent version of Docker?

@ctelfer
Copy link

ctelfer commented May 7, 2018

AFAICT, the only file that references sbState is sandbox_store.go. According to git blame this code has not been touched since July 2017 (on master, but don't think we'd change anything for a release branch that wasn't first in master).

@withsmilo withsmilo mentioned this issue Jun 11, 2019
Merged
@thaJeztah
Copy link
Member

Are you still seeing this issue on a current version of docker?

@noamgreen
Copy link

@thaJeztah HI i see this error
#8 0.312 runc run failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: failed to update store for object type *libnetwork.sbState: database not open

i am in the Docker version 27.1.2, build d01f264

Docker version 27.1.2, build d01f264
/ # docker info
Client:
Version: 27.1.2
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.16.2
Path: /usr/local/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.29.2
Path: /usr/local/libexec/docker/cli-plugins/docker-compose

Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 27.1.2
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353
runc version: v1.1.13-0-g58aa920
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.1.102-108.177.amzn2023.x86_64
Operating System: Alpine Linux v3.20 (containerized)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 20.04GiB
Name: runner-dind-api-wvkbk-runner-jkvbk
ID: 5ce9b2d4-d083-4e1a-b114-1e71aa53e066
Docker Root Dir: /var/lib/docker
Debug Mode: true
File Descriptors: 24
Goroutines: 42
System Time: 2024-08-18T07:37:47.680443305Z
EventsListeners: 0
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

@daniloradenovic
Copy link
Author

Are you still seeing this issue on a current version of docker?

Sorry @thaJeztah , I missed this comment completely. It's been a while since I haven't run the types of workflows on EC2 instances that would cause this error to happen. Since there have been many releases in the meantime and nobody else complaining about this issue, I think we should close it 🙂 .

@noamgreen If you'd like, please go ahead and open a separate issue, as I don't think the root causes of these two are related.

@daniloradenovic daniloradenovic closed this as not planned Won't fix, can't repro, duplicate, stale Aug 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/networking status/more-info-needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@thaJeztah @daniloradenovic @ctelfer @noamgreen