mount /var/lib/docker/image to host for image persistency on delete?

[strobelight]

I'm curious if /var/lib/docker/image directory can be mounted from the host in order that images don't disappear after a lima restart. Or, perhaps another solution for persistent images on lima restart using docker. I'm aware of the registry, but that's not used with docker images to my knowledge.

[afbjorklund]

What do you mean with "restart" ? All images should survive a simple stop + start of the instance, they are on the "diffdisk".

It would be possible to add a lima feature to cache images on the host, so that they could be loaded into new instances...

[AkihiroSuda]

/var/lib/docker/* has to be on a local filesystem such as ext4 or XFS.

[AkihiroSuda]

so no, not a simple restart.

@strobelight Could you fix the OP to avoid confusion?

[afbjorklund]

Changed the title

[strobelight]

/var/lib/docker/* has to be on a local filesystem such as ext4 or XFS.

@AkihiroSuda so the bind mount solution by @jandubois isn't a good one? So far, it seems to work using the same yaml file.

[jandubois]

The additional disk is formatted as ext4 and is a local disk.

I think @AkihiroSuda was commenting that my first suggestion of using sshfs to mount the directory from the host is not going to work.

[strobelight]

I can't seem to get this extra disk mounted in user space.

[jandubois]

I'm not quite sure what you are trying to do. In rootless mode the data is stored under $HOME/.local/docker, so the provisioning script should be something like (again, untested):

provision:
- mode: system
  script: |
    mkdir -p /home/${LIMA_CIDATA_USER}.linux/.local/docker
    mount --bind /mnt/lima-docker /home/${LIMA_CIDATA_USER}.linux/.local/docker

I would mount the full docker/ directory and not just docker/image (and have created a docker disk to match this new usage). I'm not sure if just preserving the latter will be sufficient.

The provisioning script must still run as root because the user is not allowed to mount the directory.

[jandubois]

Finally got some time to actually test this, and here is a working version that persists both the root and user images on a separate volume:

$ grep -A 18 additionalDisk ~/.lima/docker/lima.yaml
additionalDisks:
- docker

provision:
- mode: system
  script: |
    #!/bin/bash
    set -eux -o pipefail

    MNT=/mnt/lima-docker
    ROOT=/var/lib/docker
    USER=/home/${LIMA_CIDATA_USER}.linux/.local/share/docker

    mkdir -p $MNT/root $MNT/user
    mkdir -p $ROOT $USER
    mount --bind /$MNT/root $ROOT
    mount --bind /$MNT/user $USER

    chown -R $LIMA_CIDATA_USER:$LIMA_CIDATA_USER $USER

$ limactl shell docker ls -l /mnt/lima-docker
total 24
drwx------  2 root root 16384 Apr  6 01:56 lost+found
drwx--x--- 12 root root  4096 Apr  6 04:56 root
drwx--x--- 13 jan  jan   4096 Apr  6 04:55 user

Make sure you put the provision script first, before the other scripts already in lima.yaml!

---

I just realized that the chown should probably start at ~/.local if you start with a fresh VM, so something like

chown -R $LIMA_CIDATA_USER:$LIMA_CIDATA_USER /home/${LIMA_CIDATA_USER}.linux/.local

[strobelight]

I just want my images local and not ephemeral. I should be able to delete the instance, start a different one, mount the images, and not have to recreate them or pull them.

I was trying {{.UID}}; wasn't aware of LIMA_CIDATA_USER environment variable.

Thanks again!

[strobelight]

BTW, is the whole docker dir necessary to bind mount or can the image and overlay2 be sufficient? Guess that might change depending on default storage driver.

[jandubois]

is the whole docker dir necessary to bind mount

I don't know, so I thought it is better to play it safe. It doesn't hurt, as the bulk of the space used is by the images.

[afbjorklund]

I just want my images local and not ephemeral.

The ubuntu VM is not ephemeral, there is one "pet" VM and it contains everything... It is possible to transform it to be more like the alpine VM, so that it has an "OS disk" and a "data disk" (like in the modified YAML), but that also makes the default more complex.

Maybe it should ask for confirmation, before deletion.