State database doesn't save storage options

[Silvanoc]

/kind bug

Description

State DB (Bolt) doesn't save storage options, therefore they are honored on the first run, but not on following runs.

Steps to reproduce the issue:

1. Configure a storage option (e.g. ostree_repo = "/var/lib/ostree").

2. Pull an image with debug level "looking" for the options output: sudo podman --log-level debug pull alpine:3.8 2>&1 | grep "overlay: "

3. Pull another image with debug level "looking" for the options output: sudo podman --log-level debug pull alpine:3.7 2>&1 | grep "overlay: "

Describe the results you received:

On first pull:

time="..." level=debug msg="overlay: ostree_repo=/var/lib/ostree" 
time="..." level=debug msg="overlay: override_kernelcheck=true" 
time="..." level=warning msg="Not using native diff for overlay, this may cause degraded performance for building images: failed to mount overlay: invalid argument" 

References available in OStree repository (ostree refs --repo=/var/lib/ostree):

containers-storage/7bff100f35cb359a368537bb07829b055fe8e0b1cb01085a3a628ae9c187c7b8

On second pull:

<nothing at all>

References available in OStree repository (ostree refs --repo=/var/lib/ostree):

containers-storage/7bff100f35cb359a368537bb07829b055fe8e0b1cb01085a3a628ae9c187c7b8

Describe the results you expected:

On first pull:

time="..." level=debug msg="overlay: ostree_repo=/var/lib/ostree" 
time="..." level=debug msg="overlay: override_kernelcheck=true" 
time="..." level=warning msg="Not using native diff for overlay, this may cause degraded performance for building images: failed to mount overlay: invalid argument" 

References available in OStree repository (ostree refs --repo=/var/lib/ostree):

containers-storage/7bff100f35cb359a368537bb07829b055fe8e0b1cb01085a3a628ae9c187c7b8

On second pull:

time="..." level=debug msg="overlay: ostree_repo=/var/lib/ostree" 
time="..." level=debug msg="overlay: override_kernelcheck=true" 
time="..." level=warning msg="Not using native diff for overlay, this may cause degraded performance for building images: failed to mount overlay: invalid argument" 

References available in OStree repository (ostree refs --repo=/var/lib/ostree):

containers-storage/d6da3c54c8f3a43215d51cc0d43eff9759624996e40aaf11e4be5a555c038c32
containers-storage/7bff100f35cb359a368537bb07829b055fe8e0b1cb01085a3a628ae9c187c7b8

Additional information you deem important (e.g. issue happens only occasionally):

If the state DB (bolt_state.db) is removed after the first run, then the output is the expected one.

Output of podman version:

Version:       1.0.0
Go Version:    go1.11.4
Git Commit:    ""
Built:         Mon Jan 14 23:12:09 2019
OS/Arch:       linux/amd64

Output of podman info:

host:
  BuildahVersion: 1.6-dev
  Conmon:
    package: Unknown
    path: /usr/libexec/crio/conmon
    version: 'conmon version 1.13.0, commit: e8a2525c2e7f5ab057d5a2b5f1950be5643d8053-dirty'
  Distribution:
    distribution: arch
    version: unknown
  MemFree: 245493760
  MemTotal: 3923992576
  OCIRuntime:
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc5+dev
      commit: 079817cc26ec5292ac375bb9f47f373d33574949
      spec: 1.0.1-dev
  SwapFree: 8188518400
  SwapTotal: 8204779520
  arch: amd64
  cpus: 4
  hostname: ...
  kernel: 4.20.3-arch1-1-ARCH
  os: linux
  rootless: false
  uptime: 2h 24m 40.3s (Approximately 0.08 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions: null
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
  ImageStore:
    number: 2
  RunRoot: /var/run/containers/storage

Whereas the value of GraphOptions after first run is

  - overlay.ostree_repo=/var/lib/ostree
  - overlay.mountopt=nodev
  - overlay.override_kernel_check=true

Additional environment details (AWS, VirtualBox, physical, etc.):

Bare metal installation of Arch Linux. Libpod from AUR package: https://aur.archlinux.org/packages/libpod/

[mheon]

We store nothing related to images in the BoltDB state (except for references to which image our containers use). We also don't store storage configuration there, except to use for checking the storage configuration passed in by the user. I wonder if it isn't the storage configuration checks that are blowing up here, though. It could be overwriting some information used by the ostree backend.

[Silvanoc]

@mheon I don't think that it's only the OStree configuration being affected, since another option active override_kernelcheck=true is also getting missed.

If you are not saving any of those values, then it's probably the configuration check blowing up.

[Silvanoc]

@mheon I don't know if the presence of the BoltDB is changing the checks in any ways. Remember that removing the DB after each run helps podman honoring those options.

[mheon]

It seems like we're somehow blanking supplemental storage options when the DB's saved storage configuration is not empty

[mheon]

(This is strange, because I know we're not doing that for rootless Podman - the fuse-overlayfs mount program is preserved there)

[Silvanoc]

Due to an issue that I initially faced trying to use OStree with rootless containers I started playing around with these combinations as root. I could isolate the issue as root, but I would expect the same issue to appear rootless.

In fact it would probably explain by I'm having issues to get rootless containers using fuse-overlayfs although I installed it and configured mount_program as documented.

[mheon]

Hm.
@Silvanoc Can you pastebin your /etc/containers/storage.conf? I'm wondering if something might be affecting how we parse it.

(We have issues where we won't error on trying to load a storage config that is completely unreadable to us, but that doesn't explain why it works the first time and not the second)

[Silvanoc]

Let me pastebin everything.

/etc/containers/storage.conf -> https://pastebin.com/apTiF1QX

Debug output first run -> https://pastebin.com/1YUTuC4U
Debug output second run -> https://pastebin.com/8kDS6eC4

Debug output second run removing BoltDB -> https://pastebin.com/6nET34WK

BTW, the result of the second run is exactly the same if I wipe the whole /var/lib/containers and run everything from scratch removing BoltDB between the first and the second run.

[Silvanoc]

Wow! This is strange! Running the same commands rootless (pretty much same configuration, except for user paths and mount_program = "/usr/bin/fuse-overlayfs") works as expected! 😮

So the issue appears in root execution, but not in rootless execution. This might help you isolate the issue.

[mheon]

I think it has to be https://github.com/containers/libpod/blob/master/pkg/util/utils.go#L322

For rootless cases, we force a reload of the storage configuration file, which pre-populates our storage config from storage.conf - including storage options

For non-rootless cases, once the DB has been created, we pre-populate a good deal of storage.conf from it (to try and ensure path consistency - if you change a directory after containers have been created, strange and undesirable things may happen). That seems to be overriding our normal check into storage.conf which would pick up the additional storage options.

I think we need to make that storage.conf reload unconditional, not for only rootless cases

[rhatdan]

@giuseppe and I were playing in this code a while back. WDYT @giuseppe

[giuseppe]

I agree with @mheon, storage.conf should probably be reloaded for root containers as well, so that if Podman honors any change happening there

[mheon]

Taking a look at this tomorrow

[mheon]

#2268 should fix

@Silvanoc Any chance you can build a patched Podman and verify that the fix works?

[Silvanoc]

@mheon sure! I'll provide any feedback directly on the PR.