Make docker release images multi-platform (arm + amd)

[mistercrunch]

As it came up in #26783, it'd be nice to have our official release docker images be multi-platform (support both amd and arm). To be clear, we currently support both platforms, but not under the same tag, meaning currently we have say lean and lean-arm, but in thoery, lean could contain both arm and amd for convenience. From my understanding that would also make downstream constructs like docker-compose and helm more universal/simpler

Copy pasting the most-relevant comment from there here:

Two approaches:

• I think there's a way to merge the manifests post build, but that seemed tricky/annoying
• pass two platform arguments to docker build (which doubles the build time) and we only want on-release

I'd go with option 2, and on-release only, but it'd require doing a bit of work around the build script. Overall the process would be:

• add support for multi-platform argument here
• remove platform from the matrix here
• pass the two platforms in the call here
• re-trigger the job for old releases (easy)

[arundh93]

@mistercrunch One suggestion for helm installation. I think we should update the docs to mention that we might have to override https://github.com/apache/superset/blob/master/helm/superset/values.yaml#L189 for helm installation on a ARM platform.

This is needed because users try to install the superset helm chart will face issue since the dockerise tag in the image default is not for ARM platform.

Let me know if I can help in updating the docs for the same or if you have any other thoughts

[mistercrunch]

Ok feel free to update the docs to reflect that. I don't use helm / docker personally so I'm not the best person to document or maintain that. The main reason I worked on docker over the past few weeks was because it was slowing down CI (was taking 50 minutes to run CI for each commit in a PR).

[mistercrunch]

Also I think that if we implement multi-platform images as proposed in this issue then helm should just work without messing with values

[arundh93]

Sure let's wait for the multi platform images to go in first then.

[mistercrunch]

Yup, let's wait, maybe someone will pick this up.

Leaves me wondering who uses helm on arm, like do people run k8s on their Mac? Run superset on arm-powered EC2 boxes? Why?

[mistercrunch]

Here, this should help #27055

[mhassanzahid]

Hello,

Just a question regarding multi-arch images, we are using superset version 4.0.2 (which is multi-arch) as a base image and installing 4 dependencies for using the "ENABLE FORECAST" feature i.e.

• lunarcalendar
• tqdm
• pystan
• prophet

The problem we are facing is, httpstan part of pystan doesn't have an arm64 binary (atleast according to what i see on pypi, i can be wrong) and our docker build is failing because of this for arm64 images, is this resolvable with any alternatives, or is it not possible to build for arm64 architecture with "ENABLE FORECAST" enabled?

[mistercrunch]

Seems like more of a pystan problem than a Superset one. Are you using arm64 as a production deploy? For development (if you're actively working on this feature) and as a workaround you could fire up an EC2 amd box of some kind.

[mhassanzahid]

Yes, we're using arm64 nodes as production deploy. Is there an alternative to pystan then in this case, or any ways we could get around this issue?

[mistercrunch]

I'd recommend opening an issue here https://github.com/stan-dev/pystan

Also you may want to use amd64 architecture in production to avoid this whole family of issues. Many of us are on OSX for local dev, but I'd assume 95%+ of the community uses amd for production

[mistercrunch]

Closing this discussion as the main goal has been largely achieved. We can re-open specific issues/discussions that are more targeted from this point on.