Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: document development process #280

Merged
merged 5 commits into from
Jun 27, 2023
Merged

docs: document development process #280

Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9ed489e
docs: document development process
kuisathaverat Mar 31, 2023
05ee3aa
Apply suggestions from code review
kuisathaverat Apr 10, 2023
1af574f
Merge branch 'main' into docs/docker_images_docs
v1v Apr 19, 2023
7af4049
Merge branch 'main' into docs/docker_images_docs
kuisathaverat May 5, 2023
2582d01
Merge branch 'main' into docs/docker_images_docs
kuisathaverat Jun 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
291 changes: 254 additions & 37 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,45 @@
[![Build Status](https://beats-ci.elastic.co/buildStatus/icon?job=golang-crossbuild%2Fgolang-crossbuild-mbp%2Fmain)](https://beats-ci.elastic.co/job/golang-crossbuild/job/golang-crossbuild-mbp/job/main/)

# golang-crossbuild

golang-crossbuild is a set of Docker images containing the requisite
cross-compilers for cross compiling Go applications. The cross-compilers are
needed when the application uses [cgo](https://golang.org/cmd/cgo/).
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CGO is not recommended when you use cross-compilers. See cgo is not Go


The base image used is Debian 9 (stretch) unless otherwise specified.

The `armel`, `mips`, `ppc` and `s390x` platforms are only supported from `Debian 11` onwards. The last known version for `Debian 10` was based on Golang version `1.18.4`/`1.17.12`.

`mips32` is not available in `Debian 11`.

## Docker Repo

`docker.elastic.co/beats-dev/golang-crossbuild:[TAG]`
# golang-crossbuild Docker images

This repository contains Dockerfiles for cross building Go binaries for various platforms.
The aim is to provide a simple way to build Go binaries for multiple platforms without having to install and configure cross compilers on your host machine.
To do that the project provides a set of Docker images that can be used to build Go binaries for the following platforms:

* linux/amd64
* linux/arm
* linux/armel
* linux/armhf
* linux/arm64
* linux/mips
* linux/mipsle
* linux/mips64
* linux/ppc64
* linux/ppc64le
* linux/s390x
* windows/amd64
* darwin/amd64
* darwin/arm64

The Docker images are based on Debian and the cross compilers are installed using the crossbuild-essential package.
Each architecture has its own folder with the files needed to build the Docker image for that architecture.
Each architecture has its own Dockerfile to build a Docker image for that architecture.
Each architecture Dockerfile installs the crossbuild-essential package for that architecture and the libraries needed to build our binaries.
These Dockerfiles are generated files from `Dockerfile.tmpl` template file that is in the architectures folder.
This template is processed using a Makefile that is in the architectures folder.
Each architecture folder has a `roofs` folder that contains the files that will be copied to the Docker image in the root folder.
In `rootfs` we have the `compolers.yml` file that contains the list of compilers that will be installed in the Docker image.
Each Docker image has a basic compilation test that is executed when the image is built. This test uses `rootfs/helloworld.c` file to compile a simple C program and verify the architecture of the result binary.
The compiler used to build the binaries is LLVM.
Some of the Docker images are build for the amd64 and arm64 architectures, this allow to run the Docker images in linux/amd64, linux/arm64, darwin/amd64, and darwin/arm64. This is done using the `.ci/scripts/buildx.sh` command.

The Docker images are tagged using the following format:

* `docker.elastic.co/beats-dev/golang-crossbuild:<go-version>-<arch>-<debian-version>`

For the latest version of the images based on the latest Debian and Go versions, the following tag is also used:

* `docker.elastic.co/beats-dev/golang-crossbuild:<go-version>-<arch>`

## Build tags

Expand Down Expand Up @@ -73,24 +98,111 @@ Until Golang version 1.15
**Debian9** uses `glibc 2.24`.
**Debian10** uses `glibc 2.28`.

## Usage Example
## Makefiles

```sh
docker run -it --rm \
-v $GOPATH/src/github.com/user/go-project:/go/src/github.com/user/go-project \
-w /go/src/github.com/user/go-project \
-e CGO_ENABLED=1 \
docker.elastic.co/beats-dev/golang-crossbuild:1.16.7-armhf \
--build-cmd "make build" \
-p "linux/armv7"
There are several Makefiles in the profect across the different folders.
The Makefile in the root folder is used to build the Docker images for the different architectures,
it triggers the build of all Docker images for all architectures and Debian versions supported.

The file `go/Makefile.common` is the default Makefile used to build the Docker images for the different architectures.
There is additional Makefile for each Debian version that is used to build the Docker images for that Debian version.

* `go/Makefile.debian7`
* `go/Makefile.debian8`
* `go/Makefile.debian9`
* `go/Makefile.debian10`
* `go/Makefile.debian11`

No all architectures are supported in all Debian versions, so the Makefile for each Debian version will only build the Docker images for the architectures that are supported in that Debian version.

On the Makefiles there are some variables to define the name of the Docker image, the version of the Docker image, the Debian version, and the suffix to use in the tag of the Docker image.

```make
NAME := golang-crossbuild
VERSION := 1.20.2
DEBIAN_VERSION ?= 9
SUFFIX := -$(shell basename $(CURDIR))
TAG_EXTENSION ?=

export DEBIAN_VERSION TAG_EXTENSION

DOCKER_CMD := docker build
```

This will execute your projects `make build` target. While executing the build
command the following variables with be added to the environment: GOOS, GOARCH,
GOARM, PLATFORM_ID, CC, and CXX.
In this example the name of the Docker image is `golang-crossbuild`, the go version is `1.20.2` that is uses as part of the tag, the Debian version is `9`, the suffix is `-debian9`, and the tag extension is empty. Also the `DOCKER_CMD` variable is used to define the command to use to build the Docker image, in this case `docker build`.

The tag is build using the following format:

```make
TAG := $(REPOSITORY)/$(NAME):$(VERSION)$(SUFFIX)$(TAG_EXTENSION)
```

The common variables to all the Makefiles are defined at `Makefile.common` file.

To more information about the supported architextures and the correlation with comercial names,
you can check the [Debian Supported Architectures](https://wiki.debian.org/SupportedArchitectures) page.

To make and push the Docker images for all the architectures and default Debian version, you can run the following command:

```shell
make build push
```

To make and push the Docker images for all the architectures and Debian 10, you can run the following command:

```shell
make -C go -f Makefile.debian10 build push
```

Finally, to build a single Docker image for `arm` architecture and Debian 10, you can run the following command:

```shell
make -C go -f Makefile.debian10 build push IMAGES=arm
```

## Multiarch Docker images

Some of the Docker images are built for the amd64 and arm64 architectures, this allow to run the Docker images in linux/amd64, linux/arm64, darwin/amd64, and darwin/arm64. This is done using the `.ci/scripts/buildx.sh` command.
To choose if build a Docker image for amd64 and arm64 or only for amd64, The Makefiles check fot the value o `BUILDX` and `DOCKER_MULTIARCH` variables. In the target Makefile the `DOCKER_COMMAND` is replaced with the `.ci/scripts/buildx.sh` command.

```make
ifeq ($(DOCKER_MULTIARCH),1)
DOCKER_CMD := $(SELF_DIR)/../.ci/scripts/buildx.sh
endif
```

## Docker images dependencies

The Docker images depends on each other, so there is an order to build them.
The are two Docker images that are parent of all the other images, the `fpm` and the `go/llvm-apple`.
Anytime a new Debian version is released, the `fpm` and `go/llvm-apple` image needs to be updated to use the new Debian version.
The following diagram shows the dependencies between the Docker images.

```mermaid
stateDiagram-v2
[*] --> Debian
Debian --> fpm
Debian --> base
Debian --> base_arm
Debian --> llvm_apple
llvm_apple --> darwing_arm64
base --> arm
base --> armel
base --> armhf
base --> main
base --> darwin
base --> darwing_arm64
base --> mips
base --> mips32
base --> npcap
base --> ppc
base --> s390x
```

## Releasing images for a new Go version

With every new version of `go` we made a new branch with the name of the previous version to allow continue building the Docker images for the previous version of `go`. So if we are in go `1.19` and go `1.20` is released, we create a new branch `1.19`, the we update the `main` branch to install go `1.20`. Due to the changes in the Debian packages repositories, there is no guaranties that the Docker images for the previous version of `go` will continue to work after some time.

1. Update the Docker tag in
[Makefile.common](https://github.com/elastic/golang-crossbuild/blob/main/go1.10/Makefile.common#L5) and/or
[Makefile.common](https://github.com/elastic/golang-crossbuild/blob/main/go1.11/Makefile.common#L5) and/or
Expand All @@ -103,14 +215,6 @@ GOARM, PLATFORM_ID, CC, and CXX.
1. Create a Pull Request with the description `'Update to Go 1.x.y'`.
1. When merging the PR then the automation will release those docker images.

### Manual steps

> This is not required unless the CI service is down.

1. Build the images from the project's root with `make`.
1. Get a logon token for the container registry by visiting <https://docker-auth.elastic.co>.
1. Publish the images with `make push`.

## Packaging MacOS SDK

The osxcross repository used to cross compile for MacOSX has [instructions for packaging the SDK](https://github.com/tpoechtrager/osxcross#packaging-the-sdk).
Expand All @@ -122,11 +226,123 @@ The instructions for packaging the SDK on a Linux instance are:
1. Download [Xcode from Apple](Download Xcode: https://developer.apple.com/download/more]).
1. Run `./tools/gen_sdk_package_pbzx.sh <xcode>.xip`.

## Usage Example

```shell
docker run -it --rm \
-v $GOPATH/src/github.com/user/go-project:/go/src/github.com/user/go-project \
-w /go/src/github.com/user/go-project \
-e CGO_ENABLED=1 \
docker.elastic.co/beats-dev/golang-crossbuild:1.16.7-armhf \
--build-cmd "make build" \
-p "linux/armv7"
```

This will execute your projects `make build` target. While executing the build
command the following variables with be added to the environment: GOOS, GOARCH,
GOARM, PLATFORM_ID, CC, and CXX.

## fpm Docker image

This Docker image install the [fpm](https://github.com/jordansissel/fpm) tool that is used to build packages.

## go/llvm-apple Docker image

The LLVM compiler present in Debian does not support arm64e architecture, so we need to build our own LLVM compiler to support this architecture.
The llvm-apple Docker image is based on [Apple LLVM fork](https://github.com/apple/llvm-project) and [osxcross](https://codeload.github.com/tpoechtrager/osxcross). The image build the LLVM compiler and configure the image to be able to cross compile for MacOSX.

LLVM need a SDK for macOS, the MacOSX-SDK used in the Docker image must be genarated from a MacOSX machine.
For the instructions to package the MacOS SDK see the [Packaging MacOS SDK](#packaging-macos-sdk) section.

## go/base Docker image

This Docker image is the base image for all the other Docker images, it contains the base packages fro cross compilation.
It is build for amd64 and arm64 architectures for Debian 9+.
In the folder you can find the `sources.list` file that contains the list of repositories to use to install the packages,
this file is different for each Debian version. In some cases, this file must point to `http://archive.debian.org/debian` instead of `http://deb.debian.org/debian` to be able to install the packages, this happens when the Debian version reach the end of life.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's pretty neat!! ❤️

I recently fixed some issues with Debian 8 and Debian 9


The base image is the one that install the `go` compiler, and the build tools for the rest of Docker images.
There is a scrip `install-go.sh` that download, check the SHA256, and install the `go` compiler, during the build of the Docker image.
When a new version of go i released, the `install-go.sh` script must be updated to install the new version.

## go/base-arm Docker image

The `base-arm` image is the base image to crossbuild `linux/arm64` binaries on `arm` hosts, it is build for Debian 7+.
This image is user to crosscompile in `linux/arm`.
It was replaced by the `base` image for Debian 9+ when we started to build multiarchitecture Docker images.
It could be removed in the future, it allow to `crosscompile` arm binaries in `linux/arm64` machines, that it is not needed because the native architecture is the same than the target architecture.
It was added to golang-crossbuild due limitations of the build system.

## go/main Docker image

The `main` image is the base image for the `amd64` architecture, it is build for Debian 7+.
It is used to cross compile for `linux/amd`, `linux/amd64`, `win/amd`, and `win/amd64`.
This Docker immage add two libraries to the `base` image, `libpcap` and `WpdPack` to be able to capture network packages on diferent OS.
Thes two libraries are precompiled and stores at https://storage.googleapis.com/obs-ci-cache.

## go/darwin Docker image

The `darwin` image is the base image for the MacOSX cross compilation, it is build for Debian 8+.
It can compiles for `darwin/amd` (Debian 10+), `darwin/amd64`, `darwin/arm64`, `darwin/arm64e`, and universal binaries.
This Docker image is based on the `base` image.
It uses [osxcross](https://codeload.github.com/tpoechtrager/osxcross) to configure the crosscompile for MacOSX.
This image require a MacOSX SDK to be installed in the Docker image,
for the instructions to package the MacOS SDK see the [Packaging MacOS SDK](#packaging-macos-sdk) section.

## go/darwin-arm64 Docker image

The `darwin-arm64` image is the base image for the MacOSX cross compilation, it is build for Debian 10+.
It can compiles for `darwin/amd64`, `darwin/arm64`, and `darwin/arm64e`, and universal binaries.
This Docker image is based on the `base` image.
It uses the `llvm-apple` image to cross compile for MacOSX.
The `darwin-arm64` can replace the `darwin` image in the future, it is faster to build and it does not need to build [osxcross](https://codeload.github.com/tpoechtrager/osxcross) and uses the official LLVM fork from Apple so it support enhacement for `darwin` architectures.

## go/arm Docker image

The `arm` image is the base image for the `arm64` architecture, it is build for Debian 9+.
It is used to cross compile for `linux/arm64`. This Docker image is based on the `base` image.

## go/armel Docker image

The `armel` image is the base image for the `armel` architecture, it is build for Debian 9+.
It is used to cross compile for `linux/armel`. This Docker image is based on the `base` image.

## go/armhf Docker image

The `armhf` image is the base image for the `armhf` architecture, it is build for Debian 9+.
It is used to cross compile for `linux/armhf`. This Docker image is based on the `base` image.

## go/mips Docker image

The `mips` image is the base image for the `mips` architecture, it is build for Debian 11+.
It is used to cross compile for `linux/mips`. This Docker image is based on the `base` image.

## go/mips32 Docker image

The `mips32` image is the base image for the `mips32` architecture, it is build for Debian 11+.
It is used to cross compile for `linux/mips32`. This Docker image is based on the `base` image.

## go/ppc Docker image

The `ppc` image is the base image for the `ppc` architecture, it is build for Debian 11+.
It is used to cross compile for `linux/ppc`. This Docker image is based on the `base` image.

## go/s390x Docker image

The `s390x` image is the base image for the `s390x` architecture, it is build for Debian 11+.
It is used to cross compile for `linux/s390x`. This Docker image is based on the `base` image.

## go/npcap Docker image

The `npcap` image is a placeholder for the `npcap` library, see [npcap](./NPCAP.md) for more information.

## Troubleshooting

### bzip2 issues

If the `gen_sdk_package_pbza.sh` script gives an error that reads:

```
```shell
Error while extracting archive:(Metadata): bzip2 support not compiled in. (Success)
```

Expand All @@ -139,7 +355,8 @@ Then re-run `./tools/gen_sdk_package_pbzx.sh <xcode>.xip`.
Go to the tmp dir created in the build dir: `cd osxcross/build/tmp_<X>`.

Then run:
```

```shell
../../target/SDK/tools/bin/pbzx -n Content | cpio -i
cd ../..
XCODEDIR=osxcross/build/tmp_<X> ./tools/gen_sdk_package.sh
Expand Down