Docker/stage generates different Dockerfile depending on docker version

[eed3si9n]

steps

$ eval $(minikube docker-env)
$ sbt
> Docker/stage

expected

Docker/stage either fails or generates an identical Dockerfile with or without running eval $(minikube docker-env).

ADD --chown=daemon:daemon opt /opt

actual

The content of Dockerfile is different.

ADD opt /opt
RUN ["chown", "-R", "daemon:daemon", "."]

This causes deployment to fail on OpenShift.

notes

Ref #1044

https://github.com/sbt/sbt-native-packager/blob/v1.3.15/src/main/scala/com/typesafe/sbt/packager/docker/DockerPlugin.scala#L226-L230

    if (dockerVersion.exists(DockerSupport.chownFlag)) {
      Seq(Cmd("ADD", s"--chown=$daemonUser:$daemonGroup $files /$files"))
    } else {
      Seq(Cmd("ADD", s"$files /$files"), makeChown(daemonUser, daemonGroup, "." :: Nil))
    }

It turns out that when I run eval $(minikube docker-env) it was detecting "17.12.1-ce".

[muuki88]

Thanks for the detailed bug report ❤️

From your description it looks like you have two very different docker versions on your host and in Kubernetes. If you want to support a fixed version of docker you can set the dockerVersion setting.

dockerVersion := Some("10.0.0")

or set it to None to use no newer features. Maybe we should point that out in the docs explicitly.

I'm not sure what happens if you build docker images with different docker daemon version. Are they backwards compatible? What's the error on open shift. From my little docker experience I thought the Dockerfile is only relevant during build time.

[eed3si9n]

I think it makes sense for the build to occasionally outsource its work to external system like docker, but as a general principle the build should be repeatable.
I suggest we create a setting for dockerCompatibilityVersion and set it to whatever the version that introduced --chown. If we detect any version below it we can error out asking the user to either bump their Docker or lower the dockerCompatibilityVersion to "17.0".

What's the error on open shift. From my little docker experience I thought the Dockerfile is only relevant during build time.

On OpenShift, I get /opt/docker/bin/name-of-app: Permission denied.

Given that OpenShift runs containers using an arbitrarily assigned user ID, I actually don't think ADD --chown=daemon:daemon opt /opt would work either, but the failure let me realize this repeatability issue.

OKD-Specific Guidelines says:

Adding the following to your Dockerfile sets the directory and file permissions to allow users in the root group to access them in the built image:

RUN chgrp -R 0 /some/directory && \
    chmod -R g=u /some/directory

[muuki88]

Thanks for the clarification.

The dockerVersion setting is actually used for exactly this. While I really like the name dockerCompatibilityVersion, we don't really need it to solve this issue. If you set the dockerVersion then you get a repeatable build. What you don't get is a good error message (e.g. dockerCompatibilityVersion is higher than your actual dockerVersion). I favor less settings in this case.

Regarding the OpenShift permission issue. Should this be part of this issue? Or is this rather a feature request to remove the chown part completely?

[eed3si9n]

Regarding the OpenShift permission issue. Should this be part of this issue? Or is this rather a feature request to remove the chown part completely?

Yea, let's split the OpenShift compatibility into another issue.

I favor less settings in this case.

Do you think it's a good design that sbt-native-packager produces different docker image depending on the docker environment at the time? My opinion is that it's counter to the immutable infrastructure idea that Docker tries to implement, and the default should be safer.

If we want to do this using one setting what we can do is let it fail when it detects older Docker servers with and error message like

Docker server 17.0.0 was detected. update Docker server or set `dockerVersion := Some("17.0.0")`

[muuki88]

Do you think it's a good design that sbt-native-packager produces different docker image depending on the docker environment at the time? My opinion is that it's counter to the immutable infrastructure idea that Docker tries to implement, and the default should be safer.

I agree with you. The issue with docker is that it's changing things rather quickly (deprecating, adding, removing cli flags). While rpm and debian kept stable since I started working with sbt-native-packager, docker changed a lot.

I would argue that you can expect a certain build environment to be present to produce a certain artifact. The build pipeline is part of the immutable infrastructure. The same docker version should create the same Dockerfile.

If we want to do this using one setting what we can do is let it fail when it detects older Docker servers with and error message like

You are right. We can validate the set version to the actual detected version and error out if they don't match. The question is how do we validate?

• Check if the Dockerfiles would be equal?
• Check only the major version?
• Check all DockerSupport checks?

There is already a validateDockerVersion validation task, which runs before the actual packaging and prints warnings or errors. We could simply extend this logic. WDYT?

[muuki88]

This should be fixed in 1.3.18 as we can configure the various permissions and docker build strategies.