Packages overwrite symlinked directories

[glindstedt]

I'm not sure if this is a valid issue, perhaps it's a "wont fix", but I figured it's worth opening an issue for visibility in case other people run into it and perhaps to discuss workarounds.

When using ubuntu:22.04 as a base image, a lot of the directories are symlinks, e.g. /lib -> /usr/lib, which is quite common. However the packages in the ubuntu repositories don't map into the target directories of the symlinks, but into the symlink itself. For example libc6 maps some .so libraries into /lib and /lib64.

Because of this, when installing a package using deb_index the package will overwrite the /lib symlink with a directory, breaking the connection to the libs that come with the base image, causing strange issues.

I tried using remap_paths in pkg_tar like this, but doesn't seem to do the trick:

pkg_tar(
    name = "debs",
    remap_paths = {
        "/bin": "/usr/bin",
        "/lib": "/usr/lib",
        "/lib32": "/usr/lib32",
        "/lib64": "/usr/lib64",
        "/libx32": "/usr/libx32",
        "/sbin": "/usr/sbin",
    },
...

[alexconrey]

I've had success with some modifications to existing logic:

load("@aspect_bazel_lib//lib:tar.bzl", "tar")
tar(
  name = "symlink_tree",
  mtree = [
    "./bin type=link link=usr/bin mode=0777 uid=0 gid=0",
    "./lib type=link link=usr/lib mode=0777 uid=0 gid=0",
    ....
  ],
)

And slapping the :symlink_tree tar as the final layer in an oci_image rule:

# Workaround for https://github.com/GoogleContainerTools/rules_distroless/issues/36
pkg_tar(
    name = "package_layers.tar",
    deps = select({
        "@platforms//cpu:arm64": [
            "%s/arm64" % package
            for package in PACKAGES
        ],
        "@platforms//cpu:x86_64": [
            "%s/amd64" % package
            for package in PACKAGES
        ],
    }),
)

oci_image(
    name = "image",
    architecture = select({
        "@platforms//cpu:arm64": "arm64",
        "@platforms//cpu:x86_64": "amd64",
    }),
    entrypoint = [
        "/bin/bash",
        "-lc",
    ],
    env = {
        "TZDATA": "Etc/Utc",
        "DEBIAN_FRONTEND": "noninteractive",
    },
    os = "linux",
    tars = [
        ":package_layers.tar",
        ":symlink_tree",
    ],
)

[thesayyn]

this sounds like ubuntu does something that is not standard because we don't do any remapping of the files that comes with the deb packages?

Do you how does apt handle this case (debian distribution of apt)?

[glindstedt]

I'm not sure what would be considered "standard", but it definitely seems strange that e.g. in the libc6 package they have files put in both /lib and in /usr/lib, even though they have the symlink /lib -> /usr/lib:

So I would assume that apt/dpkg always resolves symlinks in their actions, and packages have been defined with this assumption.

This also means that the symlink tree workaround that @alexconrey described wouldn't work in this case, since then the files that the package wanted to put in /lib would disappear from the image instead of residing in /usr/lib, at least as far as I understand.

I'm wondering if the only way to solve this would be to be able to define symlinks like this on the deb_index level, that would generate some pre-processing of the tar archives that moves files to where they'd need to be.

[thesayyn]

I'm wondering if the only way to solve this would be to be able to define symlinks like this on the deb_index level, that would generate some pre-processing of the tar archives that moves files to where they'd need to be.

Something along the lines of this yes, however i'd like to know how dpkg handles this.

[loosebazooka]

I think the problem here is that we are unaware of symlinks on any lower layers in a container. dpkg doesn't have to deal with this because you can tell tar to preserve symlinks --keep-directory-symlink or whatever?

[oncilla]

Symlink tree as suggested by @alexconrey did not work for us unfortunately.

I came up with this horrific workaround that at least works for our use case.
Leaving it here in case an interested reader stumbles across this issue:

    # The tars created by rules_distroless have proper directories instead of symlinks
    # which overwrite the symlinks in the base image. This will result in a broken image.
    # To counter this, we move the contents of the supposedly symlinked sources to the
    # symlink target directories, remove the source directories and create symlinks to the
    # target directories.
    #
    # See: https://github.com/GoogleContainerTools/rules_distroless/issues/53
    native.genrule(
        name = "tester_layer_deb_remapped",
        srcs = ["tester_layer_deb"],
        outs = ["tester_layer_deb_remapped.tar"],
        cmd = " ; ".join([
            "SCRATCH=$$(mktemp -d )",
            "REALOUT=$$(realpath $@)",
            "mkdir -p $$SCRATCH/bundle",
            "echo $$SCRATCH/bundle",
            "tar -xf $(location :tester_layer_deb) -C $$SCRATCH/bundle",
            "cd $$SCRATCH/bundle",
            "[ -e bin ] && rsync -av bin/ usr/bin/ && rm -rf bin && ln -s /usr/bin bin || true",
            "[ -e sbin ] && rsync -av sbin/ usr/sbin/ && rm -rf sbin && ln -s /usr/sbin sbin || true",
            "[ -e lib ] && rsync -av lib/ usr/lib/ && rm -rf lib && ln -s /usr/lib lib || true",
            "[ -e lib64 ] && rsync -av lib64/ usr/lib64/ && rm -rf lib64 && ln -s /usr/lib64 lib64 || true",
            "[ -e var/run ] && rsync -av var/run/ run/ && rm -rf var/run && ln -s /run var/run || true",
            "[ -e var/lock ] && rsync -av var/lock/ run/lock/ && rm -rf var/lock && ln -s /run/lock var/lock || true",
            "tar -cf $$REALOUT .",
            "rm -rf $$SCRATCH"
        ]),
    )

[gfrankliu]

Same issue for Debian, which has /bin symlinked to /usr/bin.

To show the problem, you can add the package "iproute2" on Debian 12 which has files /bin/ip /bin/ss. That breaks the /bin symlink, and we end up with a real /bin directory with only "ip" and "ss".

[gfrankliu]

I just tried this on a Debian 12 and the end result is the symlink bin -> usr/bin got removed, and a new bin directory is created with a single file in it (sh symlink).

[gfrankliu]

this sounds like ubuntu does something that is not standard because we don't do any remapping of the files that comes with the deb packages?

Not just Ubuntu, Debian 12 (bookworm) has it too:

/# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

/# ls -l /
total 52
lrwxrwxrwx   1 root   root     7 Jun 12 00:00 bin -> usr/bin
drwxr-xr-x   2 root   root  4096 Jan 28 21:20 boot
drwxr-xr-x   5 root   root   360 Jun 19 00:00 dev
drwxr-xr-x   1 root   root  4096 Jun 19 00:00 etc
drwxr-xr-x   2 root   root  4096 Jan 28 21:20 home
lrwxrwxrwx   1 root   root     7 Jun 12 00:00 lib -> usr/lib
lrwxrwxrwx   1 root   root     9 Jun 12 00:00 lib64 -> usr/lib64
drwxr-xr-x   2 root   root  4096 Jun 12 00:00 media
drwxr-xr-x   2 root   root  4096 Jun 12 00:00 mnt
drwxr-xr-x   2 root   root  4096 Jun 12 00:00 opt
dr-xr-xr-x 567 root   root     0 Jun 19 00:00 proc
drwx------   2 root   root  4096 Jun 12 00:00 root
drwxr-xr-x   3 root   root  4096 Jun 12 00:00 run
lrwxrwxrwx   1 root   root     8 Jun 12 00:00 sbin -> usr/sbin
drwxr-xr-x   2 root   root  4096 Jun 12 00:00 srv
dr-xr-xr-x  13 root   root     0 Jun 18 23:43 sys
drwxrwxrwt   2 root   root  4096 Jun 12 00:00 tmp
drwxr-xr-x  12 root   root  4096 Jun 12 00:00 usr
drwxr-xr-x  11 root   root  4096 Jun 12 00:00 var

You can see bin links to usr/bin, lib links to usr/lib, etc.

Do you how does apt handle this case (debian distribution of apt)?

If you use debian apt install, it won't break those directory symlinks. Only rules_distroless does.

[gfrankliu]

I found a workaround. Create

load("@aspect_bazel_lib//lib:tar.bzl", "tar")

# workaround issues with debian 12 and ubuntu 22.04
# https://github.com/GoogleContainerTools/rules_distroless/issues/53
tar(
    name = "system_directory_symlinks",
    mtree = [
        "usr/bin type=dir mode=0755 uid=0 gid=0",
        "usr/lib type=dir mode=0755 uid=0 gid=0",
        "usr/lib/x86_64-linux-gnu type=dir mode=0755 uid=0 gid=0",
        "usr/lib64 type=dir mode=0755 uid=0 gid=0",
        "usr/sbin type=dir mode=0755 uid=0 gid=0",
        "./bin type=link link=usr/bin mode=0777 uid=0 gid=0",
        "./lib type=link link=usr/lib mode=0777 uid=0 gid=0",
        "./lib64 type=link link=usr/lib64 mode=0777 uid=0 gid=0",
        "./sbin type=link link=usr/sbin mode=0777 uid=0 gid=0",
    ],
)

[loosebazooka]

I like this workaround. The library still remains pretty brittle in this respect, and I think without knowledge of the underlying filesystem, users still need to be a little careful.

[gfrankliu]

This doesn't seem to work.

If I put :system_directory_symlinks at the beginning of the "tars" list, the symlink'ed /bin is created, but later gets deleted by any deb package who wants to install into /bin. In the end, the /bin becomes a real directory with only those few binaries that new deb packages in the "tars" list explicitly install into.

If I put :system_directory_symlinks at the end of the "tars" list, it will delete the real /bin and create a symlink /bin to /usr/bin. This means we lose those binaries in /bin that that deb packages in the beginning of the "tars" list have explicitly installed into.

BTW, I did a manual test with a test tar file that has ./bin/test. I then use bsdtar -P option to exact this tar file to the skeleton symlinked directory structure. This is able to preserved the /bin symlink, and the file were ended up in /usr/bin. I am wondering if there is anything we can do to flatten multiple tar files into :system_directory_symlinks using bsdtar -P?

gfrankliu@gfrankliu /tmp/test $ ls -l
total 4
lrwxrwxrwx 1 gfrankliu primarygroup    9 Jun 21 17:11 bin -> ./usr/bin
drwxr-xr-x 3 gfrankliu primarygroup 4096 Jun 21 17:11 usr
gfrankliu@gfrankliu /tmp/test $ ls -lR 
.:
total 4
lrwxrwxrwx 1 gfrankliu primarygroup    9 Jun 21 17:11 bin -> ./usr/bin
drwxr-xr-x 3 gfrankliu primarygroup 4096 Jun 21 17:11 usr

./usr:
total 4
drwxr-xr-x 2 gfrankliu primarygroup 4096 Jun 21 17:11 bin

./usr/bin:
total 0
gfrankliu@gfrankliu /tmp/test $ tar tvf /tmp/pkg.tar
-rw-r--r-- gfrankliu/primarygroup 0 2024-06-21 17:11 ./bin/test
gfrankliu@gfrankliu /tmp/test $ bsdtar -P -xf /tmp/pkg.tar
gfrankliu@gfrankliu /tmp/test $ ls -lR
.:
total 4
lrwxrwxrwx 1 gfrankliu primarygroup    9 Jun 21 17:11 bin -> ./usr/bin
drwxr-xr-x 3 gfrankliu primarygroup 4096 Jun 21 17:11 usr

./usr:
total 4
drwxr-xr-x 2 gfrankliu primarygroup 4096 Jun 21 17:12 bin

./usr/bin:
total 0
-rw-r--r-- 1 gfrankliu primarygroup 0 Jun 21 17:11 test
gfrankliu@gfrankliu /tmp/test $ 

[gfrankliu]

I tried pkg_tar to flatten the tar files as suggested in bazel-contrib/rules_oci#23

I created 2 test tar files:

gfrankliu@gfrankliu /tmp/test $ tar tvf file1.tar
lrwxrwxrwx gfrankliu/primarygroup 0 2024-06-21 17:30 bin -> ./usr/bin
drwxr-xr-x gfrankliu/primarygroup 0 2024-06-21 17:30 usr/
drwxr-xr-x gfrankliu/primarygroup 0 2024-06-21 17:30 usr/bin/
gfrankliu@gfrankliu /tmp/test $ tar tvf file2.tar
-rw-r--r-- gfrankliu/primarygroup 0 2024-06-21 17:11 ./bin/test

If I manually untar file1 and then file2, I get correct result:

gfrankliu@gfrankliu /tmp/test $ tar xf file1.tar
gfrankliu@gfrankliu /tmp/test $ tar xf file2.tar
gfrankliu@gfrankliu /tmp/test $ ls -l
total 28
lrwxrwxrwx 1 gfrankliu primarygroup     9 Jun 21 17:30 bin -> ./usr/bin
-rw-r--r-- 1 gfrankliu primarygroup 10240 Jun 21 17:30 file1.tar
-rw-r--r-- 1 gfrankliu primarygroup 10240 Jun 21 17:31 file2.tar
drwxr-xr-x 3 gfrankliu primarygroup  4096 Jun 21 17:30 usr
gfrankliu@gfrankliu /tmp/test $ ls -l bin/
total 0
-rw-r--r-- 1 gfrankliu primarygroup 0 Jun 21 17:11 test
gfrankliu@gfrankliu /tmp/test $ 

If I use pkg_tar:

load("@rules_pkg//:pkg.bzl", "pkg_tar")

pkg_tar(
    name = "flatten",
    deps = [
        "file1.tar",
        "file2.tar",
    ],
)  

The resulting flatten.tar file looks like this:

lrwxrwxrwx 434078/89939      0 2024-06-21 17:30 bin -> ./usr/bin
drwxr-xr-x 434078/89939      0 2024-06-21 17:30 usr/
drwxr-xr-x 434078/89939      0 2024-06-21 17:30 usr/bin/
drwxr-xr-x 434078/89939      0 2024-06-21 17:11 ./bin/
-rw-r--r-- 434078/89939      0 2024-06-21 17:11 ./bin/test

If I extract the flatten.tar (tar -xvf ...), I get

gfrankliu@gfrankliu /tmp/flatten $ ls -lR
.:
total 8
drwxr-xr-x 2 gfrankliu primarygroup 4096 Jun 21 17:11 bin
drwxr-xr-x 3 gfrankliu primarygroup 4096 Jun 21 17:30 usr

./bin:
total 0
-rw-r--r-- 1 gfrankliu primarygroup 0 Jun 21 17:11 test

./usr:
total 4
drwxr-xr-x 2 gfrankliu primarygroup 4096 Jun 21 17:30 bin

./usr/bin:
total 0

As you can see, the symlink /bin got deleted, so pkg_tar doesn't seem to create the correct flatten tar file.

BTW, if I use -P option with bsdtar, bsdtar -P -xvf flatten.tar seems to keep the symlink ...

[gfrankliu]

I am able to make my above flatten pkg_tar test work by setting the new pkg_tar create_parents attribute to False.

This is because in my test file2.tar, I didn't include the parent directory:

gfrankliu@gfrankliu /tmp/test $ tar tvf file2.tar
-rw-r--r-- gfrankliu/primarygroup 0 2024-06-21 17:11 ./bin/test

If I were to include the parent directory in the file2.tar, setting create_parents to False won't be able to help :(

gfrankliu@gfrankliu /tmp/test $ tar cvf file2.tar ./bin
./bin/
./bin/test
gfrankliu@gfrankliu /tmp/test $ tar tvf file2.tar
drwxr-xr-x gfrankliu/primarygroup 0 2024-06-23 10:47 ./bin/
-rw-r--r-- gfrankliu/primarygroup 0 2024-06-23 10:47 ./bin/test

It's unfortunately that most of the tar files from deb packages includes the parent directory. eg: on Debian 12:

$ dpkg -L iproute2 | grep ^/bin
/bin
/bin/ip
/bin/ss

Checking data.tar.xz of iproute2:

drwxr-xr-x root/root         0 2023-05-22 13:19 ./bin/
-rwxr-xr-x root/root    691016 2023-05-22 13:19 ./bin/ip
-rwxr-xr-x root/root    193680 2023-05-22 13:19 ./bin/ss
drwxr-xr-x root/root         0 2023-05-22 13:19 ./usr/bin/
-rwxr-xr-x root/root     27224 2023-05-22 13:19 ./usr/bin/lnstat
-rwxr-xr-x root/root    106952 2023-05-22 13:19 ./usr/bin/nstat
-rwxr-xr-x root/root    184936 2023-05-22 13:19 ./usr/bin/rdma
-rwxr-xr-x root/root      1658 2023-05-22 13:19 ./usr/bin/routel
lrwxrwxrwx root/root         0 2023-05-22 13:19 ./sbin/ip -> /bin/ip
lrwxrwxrwx root/root         0 2023-05-22 13:19 ./usr/bin/ctstat -> lnstat
lrwxrwxrwx root/root         0 2023-05-22 13:19 ./usr/bin/rtstat -> lnstat

My guess is the parent directory ./bin/ in the tar file caused symlinked "bin->usr/bin" got removed and actual "bin" directory being created. I don't know what's the best way to fix this. A few thoughts:

• Remove those parent directories from the tar, then set create_parents to False in pkg_tar to flatten all packages.
• Create a new flatten rule and use bsdtar -P to untar all the files, and then recreate a new single tar.

[BillyBlaze]

We had the same problem when our external image was upgraded from debian bullseye to bookworm, however we had some extra difficulties since 2 packages from Debian was put into the /bin folder.

We had to combine both workarounds from this thread:

pkg_tar(
    name = "netcat-package",
    deps = select({
        "@platforms//cpu:arm64": [
            "@bookworm//netcat-openbsd/arm64",
        ],
        "@platforms//cpu:x86_64": [
            "@bookworm//netcat-openbsd/amd64",
        ],
    }),
)

# Move netcat from /bin to /usr/bin
genrule(
    name = "fix-netcat",
    srcs = [":netcat-package"],
    outs = ["netcat-usr-bin.tar"],
    cmd = " ; ".join([
        "SCRATCH=$$(mktemp -d )",
        "REALOUT=$$(realpath $@)",
        "mkdir -p $$SCRATCH/bundle",
        'for i in $(SRCS); do tar -xmf "$$i" -C $$SCRATCH/bundle ;done',
        "cd $$SCRATCH/bundle",
        "mkdir -p ./usr/bin/",
        "mv ./bin/nc.openbsd ./usr/bin/nc",
        "rm -rf ./bin/",
        "tar --sort=name --owner=root:0 --group=root:0 --mtime='UTC 2019-01-01' -cf $$REALOUT .",
        "rm -rf $$SCRATCH",
    ]),
)

# Revert symlink /bin to /usr/bin
tar(
    name = "fix-bin",
    mtree = [
        "./bin type=link link=usr/bin mode=0777 uid=0 gid=0",
    ],
)

flatten(
    name = "image-fix",
    tars = [
        ":fix-netcat",
        ":fix-bin",
    ],
    visibility = ["//visibility:public"],
)

@oncilla thanks for your suggestion, to return the favor, add --sort=name --owner=root:0 --group=root:0 --mtime='UTC 2019-01-01' to the tar command to make your build deterministic!

[gfrankliu]

Maybe we can improve the "flatten" rule to add a new attribute flag to handle the Debian directory symlinks.

For now, based on discussions in this thread, I created a custom flatten_debs rule:
cat myrules.bzl

def flatten_debs(name, srcs, **kwargs):
  """Create a tar file from a list of deb package tar files.

  The tars created by rules_distroless have proper directories instead of symlinks
  which overwrite the symlinks in the base image and result in a broken image.
  This rule will fix this.
  https://github.com/GoogleContainerTools/rules_distroless/issues/53#issuecomment-2141987898
  """
  native.genrule(
    name = name,
    srcs = srcs,
    outs = [name + ".tar"],
    cmd  = " ; ".join([
      "SCRATCH=$$(mktemp -d )",
      "REALOUT=$$(realpath $@)",
      "mkdir -p $$SCRATCH/bundle",
      'for i in $(SRCS); do tar -xmf "$$i" -C $$SCRATCH/bundle ;done',
      "cd $$SCRATCH/bundle",
      "[ -e bin ] && rsync -av bin/ usr/bin/ && rm -rf bin && ln -s usr/bin bin || true",
      "[ -e sbin ] && rsync -av sbin/ usr/sbin/ && rm -rf sbin && ln -s usr/sbin sbin || true",
      "[ -e lib ] && rsync -av lib/ usr/lib/ && rm -rf lib && ln -s usr/lib lib || true",
      "[ -e lib64 ] && rsync -av lib64/ usr/lib64/ && rm -rf lib64 && ln -s usr/lib64 lib64 || true",
      "[ -e var/run ] && rsync -av var/run/ run/ && rm -rf var/run && ln -s /run var/run || true",
      "[ -e var/lock ] && rsync -av var/lock/ run/lock/ && rm -rf var/lock && ln -s /run/lock var/lock || true",
      "tar --sort=name --owner=root:0 --group=root:0 --mtime='UTC 2019-01-01' -cf $$REALOUT .",
      "rm -rf $$SCRATCH",
    ]),
    **kwargs
  )

Then in my BUILD file, I can use this custom rule:

load("//:myrules.bzl", "flatten_debs")

# use "flatten_debs" to flatten all Debian packages, properly handle symlinks if any.
flatten_debs(
    name = "deb_packages",
    srcs = [
        "@bookworm//curl/amd64",
        "@bookworm//jq/amd64",
        "@bookworm//netcat-openbsd/amd64",
        "@bookworm//openssl/amd64",
    ],
)

# use the flattened tar in oci_image:
oci_image(
    name = "image",
    tars = [
        ":deb_packages",
    ],
    ...
)

[BillyBlaze]

I was informed by an Apple user that the workaround doesn't work on a MacBook.
So I have tweaked my workaround with the tar toolchain from aspect_bazel_lib and removed realpath

genrule(
    name = "fix-netcat",
    srcs = [":netcat-package"],
    outs = ["netcat-bin.tar"],
    cmd = " ; ".join([
        "SCRATCH=$$(mktemp -d )",
        'REALOUT="$$PWD/$@"',
        "TAR=$$PWD/$(execpath @bsd_tar_toolchains//:resolved_toolchain)",
        "mkdir -p $$SCRATCH/bundle",
        # Extract all TAR files
        'for i in $(SRCS); do $$TAR -x -f "$$i" -C $$SCRATCH/bundle ;done',
        "cd $$SCRATCH/bundle",
        # Move netcat to /usr/bin and remove bin folder
        "mkdir -p ./usr/bin/",
        "mv ./bin/nc.openbsd ./usr/bin/nc",
        "rm -rf ./bin/",
        # Make build detereminstic
        "find ./usr/bin/ -exec touch -m -d '2000-01-01T00:00:00Z' \\{\\} +",
        "find ./usr/ -exec touch -m -d '2000-01-01T00:00:00Z' \\{\\} +",
        # Tar all extracted files back into a tar file
        "find ./etc -print0 | sort -z | $$TAR --null -nrf $$REALOUT -T -",
        "find ./lib -print0 | sort -z | $$TAR --null -nrf $$REALOUT -T -",
        "if [ -d ./lib64 ]; then find ./lib64 -print0 | sort -z | $$TAR --null -nrf $$REALOUT -T - ; fi",
        "find ./usr -print0 | sort -z | $$TAR --null -nrf $$REALOUT -T -",
        # Cleanup
        "rm -rf $$SCRATCH",
    ]),
    toolchains = [
        "@bsd_tar_toolchains//:resolved_toolchain",
    ],
)

[lazcamus]

I'm using a "clobber the symlinks in the ubuntu base image" recipe on the ubuntu base:

tar(
    name = "fix_ubuntu_base",
    mtree = [
        "bin/ uid=0 gid=0 mode=0755 type=dir",
        "lib/ uid=0 gid=0 mode=0755 type=dir",
        "sbin/ uid=0 gid=0 mode=0755 type=dir",
        "var/run/ uid=0 gid=0 mode=0755 type=dir",
        "var/lock/ uid=0 gid=0 mode=0755 type=dir",
        # /bin/bash and /bin/sh must exist
        "bin/bash uid=0 gid=0 type=link link=/usr/bin/bash",
        "bin/sh uid=0 gid=0 type=link link=/bin/bash",
    ],
)

tar(
    name = "fix_ubuntu_base_arm64",
    mtree = [
        "lib/aarch64-linux-gnu uid=0 gid=0 type=link link=/usr/lib/aarch64-linux-gnu",
        "lib/ld-linux-aarch64.so.1 uid=0 gid=0 type=link link=aarch64-linux-gnu/ld-linux-aarch64.so.1",
    ],
)

tar(
    name = "fix_ubuntu_base_amd64",
    mtree = [
        "lib32/ uid=0 gid=0 mode=0755 type=dir",
        "lib64/ uid=0 gid=0 mode=0755 type=dir",
        "libx32/ uid=0 gid=0 mode=0755 type=dir",
        "lib/x86_64-linux-gnu uid=0 gid=0 type=link link=/usr/lib/x86_64-linux-gnu",
        "lib64/ld-linux-x86-64.so.2 uid=0 gid=0 type=link link=/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2",
    ],
)

oci_image(
    name = "fixed_ubuntu",
    base = "@ubuntu//:ubuntu",
    tars = [":fix_ubuntu_base"] + select(
        {
            "@platforms//cpu:arm64": [
                ":fix_ubuntu_base_arm64",
            ],
            "@platforms//cpu:x86_64": [
                ":fix_ubuntu_base_amd64",
            ],
        },
    ),
)

oci_tarball(
    name = "fixed_ubuntu_tar",
    image = ":fixed_ubuntu",
    repo_tags = ["fixed_ubuntu"],
)

I've thought about re-assembling the ubuntu container's rootfs, but the clobbering layers work OK for now.

Not relevant to this particular issue, but while I have a like minded audience: using an ubuntu base image with rules_distroless packages yields duplicated files when you add debs that have deps that are already in the base image. It'd be nice to be able to inject a list of excludes into the deb dependency resolution (maybe hard?), or build a more general tar layer file de-duper.

[thesayyn]

inject a list of excludes

I found myself needing this as well, if anybody is willing to file an issue and send a reasonable PR, i'd be happy review and try to get it landed.

As for the symlink case, i believe we could implement a mode where we ensure that no entry in the debian packages override these symlinks by rewriting some paths, like in the workarounds above.

[thesayyn]

I'd be happy to review a PR if anyone is willing to send one.