Hard linked files broken when copied from previous stage

[michaellmonaghan]

Actual behavior
When Kaniko copies a directory from a previous stage with COPY --from=... if the directory contains files with hard links to each other, the files copied do not have hard links. As a result, in the worst cases, I have seen the size of images built by Kaniko that contain many hard links be close to twice the size of the same image built by Docker.

Expected behavior
When Kaniko copies a directory from a previous stage with COPY --from=... if the directory contains files with hard links to each other, the files copied should have hard links to each other as well.

To Reproduce
Use COPY --from= within a dockerfile to copy a directory from a previous stage that contains hard linked files.

Additional Information

• Dockerfile

FROM ubuntu AS stage1
RUN mkdir /s1 && touch /s1/a && ln /s1/a /s1/b
RUN ls -lahi /s1

FROM ubuntu
COPY --from=stage1 /s1 /s2
RUN ls -lahi /s2

Note: ls -lahi shows the inode of the file in the first column. The inodes for hard linked files are always identical.

When built with Docker, the files are properly hard linked in stage1 and the final stage.

user@localhost:~/docker/multi-stage-hard-link-example$ docker build .
Sending build context to Docker daemon   38.4kB
Step 1/6 : FROM ubuntu AS stage1
 ---> f643c72bc252
Step 2/6 : RUN mkdir /s1 && touch /s1/a && ln /s1/a /s1/b
 ---> Running in 4aac77946f7e
Removing intermediate container 4aac77946f7e
 ---> 672ac702b6af
Step 3/6 : RUN ls -lahi /s1
 ---> Running in b3fa031ee52c
total 8.0K
525762620 drwxr-xr-x 2 root root 4.0K Sep 17 19:35 .
525762607 drwxr-xr-x 1 root root 4.0K Sep 17 19:35 ..
525762621 -rw-r--r-- 2 root root    0 Sep 17 19:35 a
525762621 -rw-r--r-- 2 root root    0 Sep 17 19:35 b
Removing intermediate container b3fa031ee52c
 ---> 6e0b0eb3aea3
Step 4/6 : FROM ubuntu
 ---> f643c72bc252
Step 5/6 : COPY --from=stage1 /s1 /s2
 ---> d860892ebd11
Step 6/6 : RUN ls -lahi /s2
 ---> Running in 1de40cf39bb4
total 8.0K
525762622 drwxr-xr-x 2 root root 4.0K Sep 17 19:35 .
525762625 drwxr-xr-x 1 root root 4.0K Sep 17 19:35 ..
525762623 -rw-r--r-- 2 root root    0 Sep 17 19:35 a
525762623 -rw-r--r-- 2 root root    0 Sep 17 19:35 b
Removing intermediate container 1de40cf39bb4
 ---> 10b8431573b6
Successfully built 10b8431573b6

When built with Kaniko, the files are properly hard linked in stage1 but not in the final stage. The inodes of a and b are 525926952 and 525926957.

user@localhost:~/docker/multi-stage-hard-link-example$ docker run --rm -v $PWD:/workspace gcr.io/kaniko-project/executor --no-push --verbosity debug
DEBU[0000] Copying file /workspace/Dockerfile to /kaniko/Dockerfile 
DEBU[0000] Skip resolving path /kaniko/Dockerfile       
DEBU[0000] Skip resolving path /workspace/              
DEBU[0000] Skip resolving path /cache                   
DEBU[0000] Skip resolving path                          
DEBU[0000] Skip resolving path                          
DEBU[0000] Skip resolving path                          
DEBU[0000] Skip resolving path                          
INFO[0000] Resolved base name ubuntu to stage1          
DEBU[0000] Built stage name to index map: map[stage1:0] 
INFO[0000] Retrieving image manifest ubuntu             
INFO[0000] Retrieving image ubuntu from registry index.docker.io 
INFO[0000] GET KEYCHAIN                                 
INFO[0001] Retrieving image manifest ubuntu             
INFO[0001] Returning cached image manifest              
DEBU[0001] Resolved /s1 to /s1                          
DEBU[0001] Resolved /s2 to /s2                          
INFO[0001] Built cross stage deps: map[0:[/s1]]         
INFO[0001] Retrieving image manifest ubuntu             
INFO[0001] Returning cached image manifest              
INFO[0001] Executing 0 build triggers                   
INFO[0001] Unpacking rootfs as cmd RUN mkdir /s1 && touch /s1/a && ln /s1/a /s1/b requires it. 
DEBU[0001] Mounted directories: [{/kaniko false} {/etc/mtab false} {/tmp/apt-key-gpghome true} {/var/run false} {/proc false} {/dev false} {/dev/pts false} {/sys false} {/sys/fs/cgroup false} {/sys/fs/cgroup/systemd false} {/sys/fs/cgroup/cpu,cpuacct false} {/sys/fs/cgroup/hugetlb false} {/sys/fs/cgroup/perf_event false} {/sys/fs/cgroup/net_cls,net_prio false} {/sys/fs/cgroup/blkio false} {/sys/fs/cgroup/rdma false} {/sys/fs/cgroup/freezer false} {/sys/fs/cgroup/pids false} {/sys/fs/cgroup/cpuset false} {/sys/fs/cgroup/devices false} {/sys/fs/cgroup/memory false} {/dev/mqueue false} {/dev/shm false} {/workspace false} {/etc/resolv.conf false} {/etc/hostname false} {/etc/hosts false} {/proc/bus false} {/proc/fs false} {/proc/irq false} {/proc/sys false} {/proc/sysrq-trigger false} {/proc/acpi false} {/proc/kcore false} {/proc/keys false} {/proc/timer_list false} {/proc/sched_debug false} {/proc/scsi false} {/sys/firmware false}] 
DEBU[0001] Not adding /dev because it is ignored        
DEBU[0001] Not adding /etc/hostname because it is ignored 
DEBU[0001] Not adding /etc/hosts because it is ignored  
DEBU[0001] Not adding /etc/resolv.conf because it is ignored 
DEBU[0001] Not adding /proc because it is ignored       
DEBU[0001] Not adding /sys because it is ignored        
DEBU[0003] Not adding /var/run because it is ignored    
INFO[0003] RUN mkdir /s1 && touch /s1/a && ln /s1/a /s1/b 
INFO[0003] Taking snapshot of full filesystem...        
DEBU[0003] symlink path /usr/share/doc/libpam-runtime/NEWS.Debian.gz, target does not exist 
DEBU[0003] symlink path /usr/share/doc/libpam-modules-bin/NEWS.Debian.gz, target does not exist 
DEBU[0003] symlink path /usr/share/doc/base-files/FAQ, target does not exist 
DEBU[0003] symlink path /usr/share/doc/procps/NEWS.Debian.gz, target does not exist 
DEBU[0003] symlink path /usr/share/doc/mount/examples/fstab, target does not exist 
DEBU[0003] symlink path /usr/share/doc/mount/mount.txt, target does not exist 
DEBU[0003] symlink path /usr/share/doc/libpam-modules/NEWS.Debian.gz, target does not exist 
DEBU[0003] symlink path /usr/share/doc/apt/NEWS.Debian.gz, target does not exist 
DEBU[0003] symlink path /usr/share/doc/apt/changelog.gz, target does not exist 
INFO[0004] cmd: /bin/sh                                 
INFO[0004] args: [-c mkdir /s1 && touch /s1/a && ln /s1/a /s1/b] 
INFO[0004] Running: [/bin/sh -c mkdir /s1 && touch /s1/a && ln /s1/a /s1/b] 
INFO[0004] Taking snapshot of full filesystem...        
DEBU[0004] /s1/b inode exists in hardlinks map, linking to /s1/a 
INFO[0004] RUN ls -lahi /s1                             
INFO[0004] cmd: /bin/sh                                 
INFO[0004] args: [-c ls -lahi /s1]                      
INFO[0004] Running: [/bin/sh -c ls -lahi /s1]           
total 8.0K
525926951 drwxr-xr-x 2 root root 4.0K Sep 17 20:12 .
525762716 drwxr-xr-x 1 root root 4.0K Sep 17 20:12 ..
525926952 -rw-r--r-- 2 root root    0 Sep 17 20:12 a
525926952 -rw-r--r-- 2 root root    0 Sep 17 20:12 b
INFO[0004] Taking snapshot of full filesystem...        
INFO[0004] No files were changed, appending empty layer to config. No layer added to image. 
DEBU[0004] mapping stage idx 0 to digest sha256:c79f5d4665ab472f161f000850c3e5e380e3344576cd8107fe479c62d91f8e06 
DEBU[0004] mapping digest sha256:c79f5d4665ab472f161f000850c3e5e380e3344576cd8107fe479c62d91f8e06 to cachekey  
INFO[0004] Saving file s1 for later use                 
INFO[0004] Deleting filesystem...                       
DEBU[0004] Not deleting /etc, as it contains a ignored path 
DEBU[0004] Not deleting /etc/hostname, as it's ignored  
DEBU[0004] Not deleting /etc/hosts, as it's ignored     
DEBU[0004] Not deleting /etc/mtab, as it's ignored      
DEBU[0004] Not deleting /etc/resolv.conf, as it's ignored 
DEBU[0004] Not deleting /tmp, as it contains a ignored path 
DEBU[0004] Not deleting /var, as it contains a ignored path 
INFO[0004] Retrieving image manifest ubuntu             
INFO[0004] Returning cached image manifest              
INFO[0004] Executing 0 build triggers                   
INFO[0004] Unpacking rootfs as cmd COPY --from=stage1 /s1 /s2 requires it. 
DEBU[0004] Mounted directories: [{/kaniko false} {/etc/mtab false} {/tmp/apt-key-gpghome true} {/var/run false} {/proc false} {/dev false} {/dev/pts false} {/sys false} {/sys/fs/cgroup false} {/sys/fs/cgroup/systemd false} {/sys/fs/cgroup/cpu,cpuacct false} {/sys/fs/cgroup/hugetlb false} {/sys/fs/cgroup/perf_event false} {/sys/fs/cgroup/net_cls,net_prio false} {/sys/fs/cgroup/blkio false} {/sys/fs/cgroup/rdma false} {/sys/fs/cgroup/freezer false} {/sys/fs/cgroup/pids false} {/sys/fs/cgroup/cpuset false} {/sys/fs/cgroup/devices false} {/sys/fs/cgroup/memory false} {/dev/mqueue false} {/dev/shm false} {/workspace false} {/etc/resolv.conf false} {/etc/hostname false} {/etc/hosts false} {/proc/bus false} {/proc/fs false} {/proc/irq false} {/proc/sys false} {/proc/sysrq-trigger false} {/proc/acpi false} {/proc/kcore false} {/proc/keys false} {/proc/timer_list false} {/proc/sched_debug false} {/proc/scsi false} {/sys/firmware false}] 
DEBU[0004] Not adding /dev because it is ignored        
DEBU[0004] Not adding /etc/hostname because it is ignored 
DEBU[0004] Not adding /etc/hosts because it is ignored  
DEBU[0004] Not adding /etc/resolv.conf because it is ignored 
DEBU[0004] Not adding /proc because it is ignored       
DEBU[0004] Not adding /sys because it is ignored        
DEBU[0005] Not adding /var/run because it is ignored    
INFO[0005] COPY --from=stage1 /s1 /s2                   
DEBU[0005] Resolved /s1 to /s1                          
DEBU[0005] Resolved /s2 to /s2                          
DEBU[0005] Getting files and contents at root /kaniko/0/s1/ for /kaniko/0/s1 
DEBU[0005] Copying file /kaniko/0/s1/a to /s2/a         
DEBU[0005] Copying file /kaniko/0/s1/b to /s2/b         
INFO[0005] Taking snapshot of files...                  
DEBU[0005] Taking snapshot of files [/s2/b /s2 / /s2/a] 
INFO[0005] RUN ls -lahi /s2                             
INFO[0005] Taking snapshot of full filesystem...        
DEBU[0006] symlink path /usr/share/doc/libpam-runtime/NEWS.Debian.gz, target does not exist 
DEBU[0006] symlink path /usr/share/doc/libpam-modules-bin/NEWS.Debian.gz, target does not exist 
DEBU[0006] symlink path /usr/share/doc/base-files/FAQ, target does not exist 
DEBU[0006] symlink path /usr/share/doc/procps/NEWS.Debian.gz, target does not exist 
DEBU[0006] symlink path /usr/share/doc/mount/examples/fstab, target does not exist 
DEBU[0006] symlink path /usr/share/doc/mount/mount.txt, target does not exist 
DEBU[0006] symlink path /usr/share/doc/libpam-modules/NEWS.Debian.gz, target does not exist 
DEBU[0006] symlink path /usr/share/doc/apt/NEWS.Debian.gz, target does not exist 
DEBU[0006] symlink path /usr/share/doc/apt/changelog.gz, target does not exist 
INFO[0006] cmd: /bin/sh                                 
INFO[0006] args: [-c ls -lahi /s2]                      
INFO[0006] Running: [/bin/sh -c ls -lahi /s2]           
total 8.0K
525926951 drwxr-xr-x 2 root root 4.0K Sep 17 20:12 .
525762716 drwxr-xr-x 1 root root 4.0K Sep 17 20:12 ..
525926952 -rw-r--r-- 1 root root    0 Sep 17 20:12 a
525926957 -rw-r--r-- 1 root root    0 Sep 17 20:12 b
INFO[0006] Taking snapshot of full filesystem...        
INFO[0006] No files were changed, appending empty layer to config. No layer added to image. 
DEBU[0006] mapping stage idx 1 to digest sha256:020d2fca0ccefc42cde5ddd6b6d2510c143ab7dc96c912d20dbb93c699d35697 
DEBU[0006] mapping digest sha256:020d2fca0ccefc42cde5ddd6b6d2510c143ab7dc96c912d20dbb93c699d35697 to cachekey  
INFO[0006] Skipping push to container registry due to --no-push flag

• Build Context
  None
• Kaniko Image (fully qualified with digest)
  gcr.io/kaniko-project/executor:latest 1c812ffa8ec1

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

Thank you for your time Kaniko team!

[pidydx]

I have experienced this problem as well going back quite a few versions. It still exists in 1.7.0

[davidgiga1993]

I'm facing the same issue. Including busybox (using COPY --from=busybox:1.32.0 /bin /bb) will increase the image size to > 400mb. Using docker build correctly preserves the symlinks resulting in a much smaller image (~5mb)

[clawoflight]

Still exists, I was able to reproduce this today.