got everytihng running so far; mount still does not show up in host

[cornelius-keller]

The main problem seems to be that with a recent change of quobyte the way devices are scanned has changed so that the one level depth subdirectories are not scanned anymore for marker files.
So the bind mounts have to point to the device directly in order to make registry, metadata and data pods run properly. Thanks to the guys at quobyte for helping me to solve this.

Next I had a problem with the client pot not finding some shared libs. After some fiddeling around I skipped the nsenter approach and use the quobyte-client docker image for this.

This is working and the client is running, however the shared mount works only in the container.

Issues remaining:

• with the new mount points device files for metadata and data are probably not created
• cluster file system is working only inside the pod, not on the host.

[cornelius-keller]

HI @quolix @sttts,

sorry for beeing inactive on this for some time. I hit my timbox limit in at work, now I have spend half a day of my spare time to look at this again with some distance.
I finally got everything running.
Finally the problem was that I did not understand the nsenter + lib/ld-linux-x86-64.so.2 trick @sttts did. So it took me some time that the shared libraries that failed to be loaded (like libboost, libmicrohttpd ) aktually seem not to be in the /lib directory of the quay.io/sttts/quobyte-client-nsenter:1.2.10 container.

I resolved the problem by creating my own quay.io/cornelius/quobyte-client-nsenter:1.2.16 container derived from the quay.io/quobyte/quobyte-client:1.2.16 here: https://github.com/cornelius-keller/quobyte-client-nsenter .

In the long run I think it would be great if the nsenter binary could be part of the quobyte-client container, so that it is not necessary to create a custom container for that.

[sttts]

@cornelius-keller just fyi, in newer Docker releases there is mount propagation support: moby/moby#17034. The corresponding Kubernetes PR is kubernetes/kubernetes#20698. quobyte-client is definitely another use-case for that feature.

When it is merged into Kube, the is no need for this hacky nsenter code anymore.

[casusbelli]

Hi @cornelius-keller @sttts ,
Quobyte release 1.3.2+ will contain nsenter in the quobyte-client image.
1.3.2 push will most likely happen in the first half of next week.

[johscheuer]

@cornelius-keller @sttts I played around with Docker 1.12 + Ubuntu 16.04:

Since Docker 1.12 the MountFlags=shared are enabled per Default moby/moby#22806 if you use a Docker Version below 1.12 you need to remove the line with MountFlags (or set it to shared).

The following snippet allows a Mount from a Container without running it in a fully privileged mode (on CoreOS you probably don't need the apparmor:unconfied flag ;)):

$ docker run -ti \
--cap-add SYS_ADMIN \
--security-opt apparmor:unconfined \
--device /dev/fuse \
--pid=host \
-v /mnt:/mnt:shared \
quay.io/quobyte/quobyte-client:1.3.1 ./bin/mount.quobyte -f -d INFO ${REGISTRY}:7861/ /mnt

$ findmnt -m /mnt
TARGET SOURCE                       FSTYPE OPTIONS
/mnt   quobyte@<IP>:7861/ fuse   rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other

$ echo "Hello Quobyte" > /mnt/TestVolume/hello.txt
$ cat /mnt/TestVolume/hello.txt
Hello Quobyte

I just found a challenge that needs to be solved: If the container gets killed or crashes the mount is not usable anymore and needs a umount /mnt on the host system. I think this problem is also there with nsenter?

$ docker kill dadb4d1dfa78
dadb4d1dfa78

 $ ll /mnt
ls: cannot access '/mnt': Transport endpoint is not connected

$ $ docker run -ti \
--cap-add SYS_ADMIN \
--security-opt apparmor:unconfined \
--device /dev/fuse \
--pid=host \
-v /mnt:/mnt:shared \
quay.io/quobyte/quobyte-client:1.3.1 ./bin/mount.quobyte -f -d INFO ${REGISTRY}:7861/ /mnt
853d6e18d45a5e72ebcc26f30f52af28abee3cf6c018ef70f36e00a88f0f366d

docker: Error response from daemon: mkdir /mnt: file exists.

[johscheuer]

@casusbelli like @sttts said I think there is no need for nsenter in the quobyte-client image. Or is the goal of including nsenter to support any "platform" that doesn't support mount propagation?

[casusbelli]

@johscheuer, thanks for the hint!
In essence nsenter comes automatically with another update we did to the image so it doesn't make any difference at this point. We're not adding anything specific for nsenter now.

[cornelius-keller]

@johscheuer the need for nsenter comes exacly from having existing running kubernetes clusters where there is no mount propagaton (yet). The if you are on plain docker you don't need this, also for future kubernetes clusters it will become obsolete.

[casusbelli]

Hi! As this pr is somewhat old and the nsenter topic solved i will close this at the end of the week (unless somebody wants to update & push this?).