[ci] Build wheels for Linux ARM64

[krfricke]

Why are these changes needed?

We currently only build linux wheels for x86_64. This means that users e.g. on Mac M1 machines can't run Ray in docker, as this would require linux/aarch64 (arm64) architecture wheels.

This PR updates our CI to:

• Install bazel for arm64 architecture
• Build and upload wheels for arm64 architecture

Related issue number

Closes #13780
Closes #21786
Closes #28103

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[wuisawesome]

Nice! I tried running the docker image on my M1 machine and it works. A few follow up questions about what we call "complete" here.

1. Do we need any docs for this?
2. Related, do we need to take any additional action to make sure the dockerhub images get pushed during the release process?
3. It doesn't seem like we're building/publishing any wheels. Should we be doing that and publishing them to s3/pypi?

[krfricke]

I've also confirmed that the wheels work on graviton instances and can run example scripts.

I've now also added the post-wheels tests and debug wheels for ARM64.

We can also pick this onto Ray 2.2.0 branch to build wheels for the latest release.

[pcmoritz]

How are debug wheels distinguished from non-debug ones when uploading?

[krfricke]

They have a +dbg suffix! Example: ray-3.0.0.dev0+dbg-cp38-cp38-linux_aarch64.whl dist/ray_cpp-3.0.0.dev0+dbg-cp38-cp38-linux_aarch64.whl

[pcmoritz]

Very excited to see this, thanks for doing it Kai ❤️

[krfricke]

Looks like everything is passing. The debug wheels take ~3 hours and time out sometimes (our limit is 3h at the moment). I've disabled them for now - when we actually need them, we can look into enabling this and speeding it up (e.g. adding a bigger runner, or avoiding some recompilation).

I'll merge once this passed

[richardliaw]

hmm why is docker build failing

[krfricke]

The docker failures should be unrelated (they use different docker files and we've seen this being flaky before).