COPY --link incompatibility with COPY --chown + Both flags undocumented effect on existing destination path parents

[polarathene]

Is this a docs issue?

• My issue is about the documentation content or website

Type of issue

I can't find what I'm looking for

Description

Actionables:

• COPY --chown + COPY --link should document their effect on modifying the permissions of any existing directories of the destination path provided. --chmod does not behave in this manner which only adds to confusion.
  • While the COPY --chown docs mention default ownership of the copied source content is 0:0 (regardless of --link):
    • This actually differs when using the --from flag (without a --chown override) to specify a different stage or image where the COPY source content ownership is preserved. This is demonstrated here with a visual + reproduction. As opposed to COPY with git and local contexts (instead of via --from), where the original ownership is not preserved (becomes 0:0 in container).
    • It is only the parent dirs (if any) of the COPY target path that will use the fallback 0:0 755. However... if those parent dirs did already exist and --link is used, their ownership and permission will also be replaced by 0:0 755 (the implicit FROM scratch that --link introduces for it's rebase layer causes that).
  • The COPY source content permissions are preserved (regardless of --link), unless intentionally overriding with --chmod.
  • NOTE: BuildKit 0.13.2 introduced 777 (dir) + 666 (file) permissions for checkout via git context when no umask was applied?, fixed in BuildKit 0.15.2 (PR). Not applicable to the local build context.
• The COPY --chown with non-numeric values section should highlight awareness of COPY --link being a common case for the vague build error it outputs:

  Error: buildx failed with: ERROR: failed to solve: invalid user index: -1
  

  • Unless using docker driver, where it'll instead silently fail and fallback to 0.
• Related, mode= (the --chmod equivalent) for RUN --mount=type=cache ... has two known quirks to consider documenting:
  • The mode= option will persist a cache mount across builds with --no-cache (which would normally discard the cache).
    • EDIT: This quirk is due to this BuildKit line (thus also applies to uid and gid options). Line identified from here.
  • docker buildx build --builder docker ... (default for docker build ...) prior to BuildKit 0.15.2 will apply 022 umask to a configured mode= value of the cache target= (mode=773 sets 751 on the target dir). Yet does not apply to docker-container driver running BuildKit prior to 0.15.2.

---

Full report details

COPY --link does not describe it's affect on parent directories where the permissions and ownership are modified to 755 0:0.

Perhaps because this seems to be specific to the docker buildx build --builder ... driver used, at least with docker-container it occurs while it does not with the docker driver. The behaviour also differs in the context of --chmod and --chown usage with --link.

More information about this inconsistency here:

• docs: proposal to raise awareness about an unexpected behavior of COPY --link moby/buildkit#4964 (comment)
• docker-container driver: COPY --link --chown unexpectedly changes parent dir ownership buildx#1855 (comment)
• COPY command changes the directory permissions to change. moby/buildkit#3602 (comment)

The docs do not indicate how to verify --link is being used.

• This is a concern as known workarounds may unintentionally act as --link=false?
• In previous releases of Docker with BuildKit older than 0.10.0 the COPY --link feature would not raise an error and silently fallback to --link=false.

It would be good for the docs to at least clarify the --link scenario that may result in unexpected permissions/ownership changes to parent dirs of the target path. And if the --chmod=null workaround has any relevant concerns before that potentially spreads in adoption.

---

In the context of --link --chown=invalid, this is hinted at in the docs:

https://docs.docker.com/reference/dockerfile/#copy---chown---chmod

If the container root filesystem doesn't contain either /etc/passwd or /etc/group files and either user or group names are used in the --chown flag, the build will fail on the COPY operation. Using numeric IDs requires no lookup and does not depend on container root filesystem content.

However, as has been seen in various reports regarding these two features.. it's often not understood that --link is the cause when a build outputs this unhelpful error message. Some projects respond to this by removing --link, or finding a way to ensure the UID/GID values are deterministic.

That section should be amended to cite the common case with --link?

---

Potentially relevant context:

• COPY --link not compatible with chown moby/buildkit#2987 (comment)

  Also note that some cases could be solvable with an intermediate stage. You can copy to the intermediate stage with --chown and then to final stage with --link.

• dockerfile: eliminate dependency on dest directory for COPY moby/buildkit#2414

  Cases that could be solved with some additional syntax

  COPY --chown=uid and COPY --chmod=non-default-perms would not work by default. We can't just exclude the implied parents as docker would only create these parents with default perms/user. While in Dockerfile, unfortunately, the rule is that implied parents also get these chown/chmod values

• [dnm] testing if FileMode is affected by umask moby/buildkit#4936 (comment) (cache mount permissions have an implicit umask 022 (mode adjusting permissions affects cache mount persistence), has some relevance to COPY but maybe not --link?)

  the cause of this seems to be different umask behavior in moby vs buildkitd. buildkitd resets umask

• COPY command changes the directory permissions to change. moby/buildkit#3602 (comment)

  If the COPY --link step in this case does not add a directory entry for /tmp, and the "parent" layer is FROM scratch then the permissions for that directory are "undefined"
  versions before 23.0.0 of the daemon used 0777 and depended on umask, which resulted in undefined behavior (depending on the umask and storage driver used, this could either result in 0755 permissions or 0600).

• COPY command changes the directory permissions to change. moby/buildkit#3602 (comment)

  COPY --link was first implemented in BuildKit v0.10 that was added to Moby with v23.0 release. If you used --link in an older release, that didn't do any linking behavior then you get the previous semantics exactly as without any --link flag.

  Dockerfiles that opt-in to --link are compatible with previous copy semantics. But if you just take old Dockerfile and add --link to all COPY commands you can get slightly different semantics depending on what source parameters and flags are used.

---

Location

https://docs.docker.com/reference/dockerfile/#copy

Suggestion

Suggestions above under Actionables.

• Needs to be worded into proper documentation, plenty of reference context linked and insights provided to assist with that.
• EDIT: I've sense appended to this list additional caveats from other COPY flags that should be accounted for. Along with semi-related concerns with RUN.

Usually the doc issues I engage with here become stale and locked by the docker bot, never to be addressed. Thus the above is serves as an informal documentation reference if that occurs.

Some of the observed behaviour may be bugs that will get addressed. Docs could be delayed, but note that some of these concerns have been known for well over a year, if that continues it may be better to document the caveat for the benefit of users.