Add macOS support #4059

slonopotamus · 2023-07-24T20:05:02Z

This PR adds experimental support for running native macOS containers using rund.

This PR depends on

pkg/chrootarchive: fix Darwin build moby#46094 (trivial 5-line change)
[Reverted in #8838] Add support for bind-mounts on Darwin (a.k.a. "make native snapshotter work") containerd/containerd#8789. This one is problematic. containerd doesn't have enough abstraction layer in mount.All and its developers declined PR that puts in implementation based on bindfs. Both containerd and buildkit call mount.All a lot. So it needs to work properly.
Add option to specify containerd runtime #4141
Fix usages of mountinfo.PrefixFilter containerd/containerd#9054

Also, see rund macOS containerd shim.

crazy-max

This PR almost fixes compilation of buildkitd on macOS.

I'm not sure what you try to "fix". Afaik there is no isolation primitives / bind mounts in the MacOS kernel. I guess we would need to shell out to some kind of mount helper.

If we can start with a proposal to discuss on a proper abstraction (pluggable) interface that would be great.

I also can't first fix moby and then fix buildkit.

This is possible:

slonopotamus · 2023-07-24T21:51:32Z

Afaik there is no isolation primitives / bind mounts in the MacOS kernel.

That's a completely different story and I'd like to avoid expanding the scope of current PR to cover those aspects. It is just a small initial step that only makes the thing compile and nothing else.

This is possible

I'll take a look, thanks.

slonopotamus · 2023-07-25T10:22:04Z

I'll add some background context here.

I'm currently working on a containerd shim for macOS: https://github.com/macOScontainers/rund.

While it doesn't offer the usual container level of isolation from the host due to kernel limitations, there is a highly requested containerd/containerd#5525. Even with the limited functionality of rootfs delivery + chroot, such thing would be very useful for CI needs.

I'm not sure it will ever become an official containerd shim, but actually that doesn't matter much.

So, the point here is that eventually I'd like to be able to get the whole containerd+buildkit+moby combo working and current PR is just a small step in that direction.

slonopotamus · 2023-07-25T10:41:54Z

It seems like changes in this PR somewhat overlap with #2376 and would definitely cause conflicts. If #2376 is going to be merged any time soon, let it go first. I'll resolve conflicts on my side after that.

slonopotamus · 2023-07-25T11:51:44Z

This is possible

Sorry, I'm still not understanding.

I have two separate repos, with buildkit and with moby.

If I try to build buildkit with changes from current PR, it fails in moby code:

% go build -o buildkitd ./cmd/buildkitd
# github.com/docker/docker/pkg/chrootarchive
vendor/github.com/docker/docker/pkg/chrootarchive/archive_unix.go:30:8: undefined: goInChroot
vendor/github.com/docker/docker/pkg/chrootarchive/archive_unix.go:52:8: undefined: goInChroot
vendor/github.com/docker/docker/pkg/chrootarchive/diff_unix.go:44:8: undefined: goInChroot

If I try to build moby, it fails in buildkit code:

% ./hack/make.sh

Removing bundles/

---> Making bundle: binary-daemon (in bundles/binary-daemon)
Building static bundles/binary-daemon/dockerd (darwin/arm64)...
+ tee /Users/marat/Documents/moby/go.mod
module github.com/docker/docker

go 1.19
+ trap 'rm -f "${ROOTDIR}/go.mod"' EXIT
+ GO111MODULE=on
+ go build -mod=vendor -modfile=vendor.mod -o bundles/binary-daemon/dockerd -tags 'netgo osusergo static_build  libdm_dlsym_deferred_remove' -ldflags '-w -X "github.com/docker/docker/dockerversion.Version=dev" -X "github.com/docker/docker/dockerversion.GitCommit=96b473a0bd" -X "github.com/docker/docker/dockerversion.BuildTime=" -X "github.com/docker/docker/dockerversion.PlatformName=" -X "github.com/docker/docker/dockerversion.ProductName=" -X "github.com/docker/docker/dockerversion.DefaultProductLicense="  -extldflags -static ' github.com/docker/docker/cmd/dockerd
package github.com/docker/docker/cmd/dockerd
        imports github.com/docker/docker/api/server/backend/build
        imports github.com/docker/docker/builder/builder-next
        imports github.com/docker/docker/builder/builder-next/adapters/containerimage
        imports github.com/moby/buildkit/cache
        imports github.com/moby/buildkit/snapshot
        imports github.com/containerd/containerd/snapshots/overlay/overlayutils: build constraints exclude all Go files in /Users/marat/Documents/moby/vendor/github.com/containerd/containerd/snapshots/overlay/overlayutils
package github.com/docker/docker/cmd/dockerd
        imports github.com/docker/docker/api/server/backend/build
        imports github.com/docker/docker/builder/builder-next
        imports github.com/docker/docker/libnetwork
        imports github.com/docker/docker/libnetwork/iptables: build constraints exclude all Go files in /Users/marat/Documents/moby/libnetwork/iptables
package github.com/docker/docker/cmd/dockerd
        imports github.com/docker/docker/api/server/backend/build
        imports github.com/docker/docker/builder/builder-next
        imports github.com/moby/buildkit/executor/oci
        imports github.com/moby/buildkit/util/entitlements/security: build constraints exclude all Go files in /Users/marat/Documents/moby/vendor/github.com/moby/buildkit/util/entitlements/security
package github.com/docker/docker/cmd/dockerd
        imports github.com/docker/docker/api/server/backend/build
        imports github.com/docker/docker/builder/builder-next
        imports github.com/docker/docker/builder/builder-next/adapters/containerimage
        imports github.com/moby/buildkit/cache
        imports github.com/moby/buildkit/snapshot
        imports github.com/moby/buildkit/util/overlay: build constraints exclude all Go files in /Users/marat/Documents/moby/vendor/github.com/moby/buildkit/util/overlay
+ rm -f /Users/marat/Documents/moby/go.mod

So, I don't see any way to fix them one by one. I suppose the procedure has to be

Fix issues in one of the repos (but it won't be buildable)
Update dependency in second repo to fixed state
Update dependency in first repo to fixed state

@akhramov could you please enlighten me, how you worked around this cycle in context of your FreeBSD changes? If I understand things properly, when moby/moby#45724 was merged, moby actually cannot be built on FreeBSD, right?

akhramov · 2023-07-25T16:48:38Z

@slonopotamus the trick is that you don't need to build moby in its entirety, but rather a few well-isolated packages. In your case you would need to fix github.com/docker/docker/pkg/chrootarchive which does not depend on buildkit :)

slonopotamus · 2023-07-25T18:28:06Z

Okay, makes sense. Thanks.

Before this commit, `doPack`, `doUnpack` and `doUnpackLayer` were not implemented for Darwin, causing build failure. This change allows all non-Linux Unixes to use FreeBSD reexec-based pack/unpack implementation See also: moby/buildkit#4059 See also: 8b84373 Signed-off-by: Marat Radchenko <marat@slonopotamus.org>

slonopotamus · 2023-07-27T20:28:30Z

See moby/moby#46094 for fixes on Moby side.

slonopotamus · 2023-08-02T16:09:05Z

Given that #2376 was merged, I'll rebase soon and add some more code unification across Darwin/FreeBSD.

slonopotamus · 2023-08-03T17:53:32Z

Okay, I managed to put bits together and even build a simple macOS native Dockerfile:

Functionality of this PR has some dependencies, see top message.

Before this commit, `doPack`, `doUnpack` and `doUnpackLayer` were not implemented for Darwin, causing build failure. This change allows all non-Linux Unixes to use FreeBSD reexec-based pack/unpack implementation See also: moby/buildkit#4059 See also: 8b84373 Signed-off-by: Marat Radchenko <marat@slonopotamus.org>

slonopotamus · 2023-08-04T18:30:39Z

I'd like to get some feedback on buildkit changes. Vendoring update is not going to happen as it is present in current PR for obvious reasons, so I'll revert it before the actual merge.

worker/containerd/containerd.go

go.mod

Before this commit, `doPack`, `doUnpack` and `doUnpackLayer` were not implemented for Darwin, causing build failure. This change allows all non-Linux Unixes to use FreeBSD reexec-based pack/unpack implementation See also: moby/buildkit#4059 See also: 8b84373 Signed-off-by: Marat Radchenko <marat@slonopotamus.org>

slonopotamus · 2023-09-27T20:05:44Z

I've restructured my branch a bit to make it clearer what changes are buildkit-related.

First commit just updates containerd to a version that includes containerd/containerd#8789. That is out of discussion currently, every party agreed that there needs to be a more universal solution.

Second commit is what I'd really like to land in buildkit repo. I'll rebase this branch a lot, so the link will become outdated, but look for "Add macOS support" commit message. As of now, it is this: 57cd14f

I suggest we only review and merge this single commit and nothing else in current PR.

UPD: Okay, it won't work this way because we want CI to pass. I'll possibly create a separate PR from a branch that only contains that single commit.

slonopotamus · 2023-09-27T20:17:38Z

Reorganised things again so this PR now only includes Buildkit-specific changes. CI is currently failing, will investigate tomorrow.

UPD: Fixed, ready for review. CI that builds these changes + updated containerd for macOS can be seen in https://github.com/macOScontainers/buildkit

Before this commit, `doPack`, `doUnpack` and `doUnpackLayer` were not implemented for Darwin, causing build failure. This change allows all non-Linux Unixes to use FreeBSD reexec-based pack/unpack implementation See also: moby/buildkit#4059 See also: 8b84373 Signed-off-by: Marat Radchenko <marat@slonopotamus.org>

util/appdefaults/appdefaults_unix_nondarwin.go

cmd/buildkitd/main_containerd_worker.go

Before this commit, `doPack`, `doUnpack` and `doUnpackLayer` were not implemented for Darwin, causing build failure. This change allows all non-Linux Unixes to use FreeBSD reexec-based pack/unpack implementation See also: moby/buildkit#4059 See also: 8b84373 Signed-off-by: Marat Radchenko <marat@slonopotamus.org>

Signed-off-by: Marat Radchenko <marat@slonopotamus.org>

slonopotamus · 2024-08-29T15:12:30Z

I've split these changes into easier-to-review #5271 (already merged) and #5276.

slonopotamus force-pushed the macos branch from 2caf1b1 to a23b528 Compare July 24, 2023 20:06

crazy-max requested changes Jul 24, 2023

View reviewed changes

slonopotamus mentioned this pull request Jul 27, 2023

pkg/chrootarchive: fix Darwin build moby/moby#46094

Merged

slonopotamus marked this pull request as draft July 28, 2023 10:22

slonopotamus force-pushed the macos branch from a23b528 to c8f4dfa Compare August 3, 2023 17:46

slonopotamus changed the title ~~Build fixes for macOS~~ Add experimental macOS support Aug 3, 2023

slonopotamus force-pushed the macos branch from c8f4dfa to bd225ea Compare August 3, 2023 20:15

slonopotamus force-pushed the macos branch from bd225ea to 6aad7ee Compare August 3, 2023 21:16

slonopotamus marked this pull request as ready for review August 4, 2023 18:29

AkihiroSuda reviewed Aug 7, 2023

View reviewed changes

worker/containerd/containerd.go Outdated Show resolved Hide resolved

AkihiroSuda reviewed Aug 7, 2023

View reviewed changes

go.mod Outdated Show resolved Hide resolved

slonopotamus marked this pull request as draft August 12, 2023 23:05

AkihiroSuda mentioned this pull request Aug 22, 2023

[Reverted in #8838] Add support for bind-mounts on Darwin (a.k.a. "make native snapshotter work") containerd/containerd#8789

Merged

slonopotamus force-pushed the macos branch from 6aad7ee to 1495514 Compare September 10, 2023 13:45

slonopotamus force-pushed the macos branch from 1495514 to 4ad38a6 Compare September 24, 2023 20:01

slonopotamus force-pushed the macos branch 4 times, most recently from 2d7f2cc to 05b4434 Compare September 27, 2023 20:01

slonopotamus force-pushed the macos branch from 05b4434 to 825598b Compare September 27, 2023 20:09

slonopotamus changed the title ~~Add experimental macOS support~~ Add macOS support Sep 27, 2023

AkihiroSuda reviewed Sep 27, 2023

View reviewed changes

util/appdefaults/appdefaults_unix_nondarwin.go Outdated Show resolved Hide resolved

slonopotamus commented Sep 28, 2023

View reviewed changes

cmd/buildkitd/main_containerd_worker.go Outdated Show resolved Hide resolved

slonopotamus force-pushed the macos branch 3 times, most recently from 86a44d1 to d65b10c Compare September 28, 2023 07:59

slonopotamus marked this pull request as ready for review September 28, 2023 08:04

slonopotamus force-pushed the macos branch from d65b10c to 2739a7f Compare September 28, 2023 08:15

slonopotamus mentioned this pull request Sep 28, 2023

Add macOS support moby/moby#46558

Closed

slonopotamus marked this pull request as draft December 10, 2023 14:03

Add macOS support

f13ca3e

Signed-off-by: Marat Radchenko <marat@slonopotamus.org>

slonopotamus force-pushed the macos branch from 2739a7f to f13ca3e Compare December 23, 2023 14:02

slonopotamus mentioned this pull request Aug 29, 2024

Add stub implementations to make buildkitd buildable for Darwin #5276

Merged

slonopotamus closed this Aug 29, 2024

slonopotamus deleted the macos branch August 29, 2024 15:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add macOS support #4059

Add macOS support #4059

slonopotamus commented Jul 24, 2023 •

edited

Loading

crazy-max left a comment

slonopotamus commented Jul 24, 2023 •

edited

Loading

slonopotamus commented Jul 25, 2023

slonopotamus commented Jul 25, 2023

slonopotamus commented Jul 25, 2023 •

edited

Loading

akhramov commented Jul 25, 2023

slonopotamus commented Jul 25, 2023

slonopotamus commented Jul 27, 2023

slonopotamus commented Aug 2, 2023

slonopotamus commented Aug 3, 2023 •

edited

Loading

slonopotamus commented Aug 4, 2023

slonopotamus commented Sep 27, 2023 •

edited

Loading

slonopotamus commented Sep 27, 2023 •

edited

Loading

slonopotamus commented Aug 29, 2024 •

edited

Loading

Add macOS support #4059

Add macOS support #4059

Conversation

slonopotamus commented Jul 24, 2023 • edited Loading

crazy-max left a comment

Choose a reason for hiding this comment

slonopotamus commented Jul 24, 2023 • edited Loading

slonopotamus commented Jul 25, 2023

slonopotamus commented Jul 25, 2023

slonopotamus commented Jul 25, 2023 • edited Loading

akhramov commented Jul 25, 2023

slonopotamus commented Jul 25, 2023

slonopotamus commented Jul 27, 2023

slonopotamus commented Aug 2, 2023

slonopotamus commented Aug 3, 2023 • edited Loading

slonopotamus commented Aug 4, 2023

slonopotamus commented Sep 27, 2023 • edited Loading

slonopotamus commented Sep 27, 2023 • edited Loading

slonopotamus commented Aug 29, 2024 • edited Loading

slonopotamus commented Jul 24, 2023 •

edited

Loading

slonopotamus commented Jul 24, 2023 •

edited

Loading

slonopotamus commented Jul 25, 2023 •

edited

Loading

slonopotamus commented Aug 3, 2023 •

edited

Loading

slonopotamus commented Sep 27, 2023 •

edited

Loading

slonopotamus commented Sep 27, 2023 •

edited

Loading

slonopotamus commented Aug 29, 2024 •

edited

Loading