Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-arch builds #200

Open
cedricve opened this issue Sep 21, 2022 · 11 comments
Open

Multi-arch builds #200

cedricve opened this issue Sep 21, 2022 · 11 comments

Comments

@cedricve
Copy link

We are running a multi arch build in a github workflow using docker buildx build, and are wondering how to manage the balena deploy so all architecture images are merged in a single release.

https://github.com/kerberos-io/agent/blob/master/.github/workflows/docker.yml#L63-L67

@pipex
Copy link
Contributor

pipex commented Sep 26, 2022

Hi @cedricve, thanks for your question. I was also looking into this myself. Unfortunately the limitation here is that the balena CLI, which this action uses under the hood, does not yet support pushing multiple images for the same release to our registry, and it doesn't push the multi-arch manifest either. Multi-arch support is on our roadmap but we don't have an ETA for that feature yet or the implementation details for it.

That being said, I am looking into adding a useDeploy feature, that could use image artifacts from a previous step in order to do a balena deploy instead of a balena push

The action would look in its local image store for one or more <projectName>_<serviceName>:<architecture> or <projectName>_<serviceName>:latest images (loaded by a previous step of the same job), and then deploy the images matching the target fleet architecture.

What do you think about that as an interface?

@cedricve
Copy link
Author

cedricve commented Sep 27, 2022

Thanks @pipex that would already make more sense. Where I think a lot of confusion starts is the difference between the "direct" deployment on Balena Hub and the forking of a fleet.

  1. When browsing to the Balena Hub, opening the Kerberos Agent project, and pressing the Get Started button, it will list all the available supported architectures. However under the hood the fleet was pointed to the arm7 architecture, so only that image is available in the balena registry. If you select an ARM64 device for example Raspberry Pi 4 from the list, it will not boot. Which is not expected and can been seen by the users as malfunctioning of the project, or even worse Balena.

Screenshot 2022-09-27 at 08 21 55

Screenshot 2022-09-27 at 08 22 07

  1. On the other hand when you "fork" the fleet to your own fleet. You can select all (even non-supported) architectures from the list. This helps to get the other "available" architectures which are supported, as after creation it will make a new release using the right base image. However, here you can select a non-supported architecture from the list, and then it will not work.

Screenshot 2022-09-27 at 08 26 20

Bottom line is that it's really not straight forward to a users what is working and what is not.

@pipex
Copy link
Contributor

pipex commented Sep 27, 2022

Hi again @cedricve, thanks for clarifiying, I think I was jumping the gun on what you were asking so I apologize.

So as I mentioned before, the balena registry is single arch for now, this means, if you have a docker-compose.yml pointing to a multi-arch image, for instance

version: "2.1"

services:
    hello:
        image: hello-world

when you do a balena push myfleet via the command line (or using this action), the builder gets the architecture from the target fleet (myfleet) and pushes just the image for the correct architecture to the balena registry. So if the target fleet is rapspberrypi3, then the builder will download the arm/v7 image and push just that to the registry.

The arm/v7 image can be used by any compatible device type, including arm/v8 devices (e.g. the pi4), since arm/v8 is backwards compatible, which explains the range of possible device types you see on the dashboard.

A simple test can be done to confirm this, on a raspberry pi 4 device running balenaOS for arm64, I can do the following

balena pull --platform linux/arm/v7 alpine
balena run --rm -ti alpine sh
WARNING: The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
# despite the above warning, the command below works
$ apk add --update nodejs 

I tested the same with the kerberos image

balena pull --platform linux/arm/v7 kerberos/agent
balena run --rm -ti kerberos/agent sh
WARNING: The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
# The command below works
$ ./main run opensource 80
{"level":"info","msg":"Successfully Opened config.json from opensource","time":"2022-09-27T15:36:06+02:00"}
{"level":"info","msg":"Main: updated unique key for agent to: S0UH034CAAHDXYIWZU2DUFGYXZAHYX","time":"2022-09-27T15:36:06+02:00"}

Which makes me think there might be something else going on when provisioning a rpi4 to the kerberos fleet. I would suggest trying to replicate the issue with your own fleet, that is, creating a raspberrypi3 fleet, pushing a release from the kerberos agent and then trying to provision a raspberry pi 4. Having access to the device via the dashboard will give you more visibility on what is going on.

I would also encourage you to look in the forums for answers, as our community is very active and we do active support there as well.

@cedricve
Copy link
Author

Thanks for explaining, I didn't know that v8 was backwards compatible. However that wouldn't solve the amd64 boards right, as the target board on your fleet is determining the specific architecture build.

@pipex
Copy link
Contributor

pipex commented Sep 27, 2022

If your fleet is raspberry pi3 (armv7), you should be able to add raspberry pi 4 (armv8) devices to it, the other way around is not possible: a raspberry pi3 (arvm7) will fail if deployed into an armv8 fleet.

@cedricve
Copy link
Author

cedricve commented Oct 11, 2022 via email

@klutchell
Copy link
Collaborator

I'm hoping to have more to share on multi-arch apps soon!

@cedricve
Copy link
Author

exciting !

@avihaibd
Copy link

Hi @pipex

Any updates on deploying for multi-architecture?
I am looking for implementing the following flow:

  1. Build each component in the docker-compose as multiplatform docker image
  2. Push to private docker registry which is not accessible to balena servers.
  3. Use images in the docker compose, build locally for multi-arch and deploy to balena.

Currently balena deploy enforces images to match platform of the fleet default machine type, (regardless of supported types in balena.yml), while devices of other machine types can still join the fleet.

Also, regarding

The action would look in its local image store for one or more : or :latest images (loaded by a previous step of the same job), and then deploy the images matching the target fleet architecture.

What do you think about that as an interface?

Consider allowing non-latest images, for example: <architecture>/<projectName>_<serviceName>:<tag>.
Also note that docker also allows building for multiple platforms in a single command, in which case the docker manifest will just use the digests of all pushed platforms.

Thanks

@klutchell
Copy link
Collaborator

Hey @avihaibd, thanks for your interest in this feature! Here is the related roadmap item that you can follow for updates!

https://roadmap.balena.io/posts/28/mixed-architecture-fleets-and-apps

In short, there are a lot of backend changes required to our API and database to support mixed device fleets, and multi-arch applications, but it is a priority project that we are excited to progress!

@robsonvn
Copy link

Meanwhile, this is what I am doing

FROM arm64v8/nginx:1.25 as nginx-aarch64
FROM arm32v7/nginx:1.25 as nginx-armv7hf
FROM nginx:1.25 as nginx-amd64

FROM nginx-%%BALENA_ARCH%%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants