To make bpftrace more portable, it has long supported an alpine-based musl build, which statically compiled bpftrace resulting in no runtime linking required.
The drawback to this approach is that LLVM libraries, even when statically compiled, depends on symbols from libdl, and works best and most predictably when dynamically linked to libc.
To embed everything except for libc, building LLVM and Clang from source is supported. This allows for linking to arbitrary libc targets dynamically, which may provide the best of both worlds between a purely static and a purely dynamically-linked bpftrace executable.
For this reason, there is CMake support in the bpftrace project to build LLVM and Clang from source, as these are the heaviest dependencies of bpftrace. Other library dependencies can be obtained by most package managers reliably.
Upstream packages provided by package maintainers can't be depended on to present all of the necessary built artifacts to statically link against LLVM and Clang, despite the project LLVM catering to this with its own CMake build process.
To make an semi-static executable that includes everything except libc, the
CMake option STATIC_LIBC:BOOL=OFF can be added. This should allow for a
bpftrace executable that can run on any system with a compatible libc.
To assist in this, new configuration flags allow to download, configure, build and embed static libraries for all dependencies of bpftrace, rather than relying on system libraries.
To build the necessary static LLVM or Clang libraries, -DEMBED_LLVM:BOOL=ON
and -DEMBED_CLANG=ON can be used respectively. Clang can link to system LLVM
as well, but may need to be patched.
It is possible to link bpftrace's embedded libraries with system libraries, so
linking to a distribution specific LLVM is possible. Many distributions do not
include libclang.a, using -DEMBED_LIBCLANG_ONLY=ON will build only
libclang.a, allowing for linking to system LLVM and Clang libraries on Debian,
as opposed to Vanilla LLVM sources.
To build from scratch on a modern development laptop with an 8-way parallel build, bpftrace and its dependencies can be built in about 30 minutes.
Subsequent builds will not have this overhead, once the embedded dependencies are built and cached, and will proceed at the comparable speeds as dynamically linked builds.
Internal CI environments with any sort of caching mechanism should be able to just cache any external project's build directories, and the overhead of building on CI shouldn't be any different than other CI jobs.
On the alpine platform, libclang.a is generated by default and a static build is already achieved using this distribution. This libclang dependency is only achievable with custom builds, or by using alpine.
To get around this problem, the embed_clang.cmake file provides the necessary
static libraries for bpftrace to link against, by downloading LLVM from a github
tagged release and compiling it directly using custom cmake flags, based on
those provided by alpine already. This basically pulls the fairly simple
alpine-build "in-tree", using cmake flags based on it, and specifically
itemizing the libraries that will be link-time dependencies of bpftrace.
In turn, these clang libraries depend heavily on LLVM libraries, so this necessitates having access to LLVM static libraries as well.
It is also possible to build only libclang from (mostly) vanilla sources, and
use system LLVM and Clang libraries. This is what the EMBED_LIBCLANG_ONLY
flag provides.
It is possible to avoid embedding LLVM by applying distribution specific patches to embedded clang, so that it will be patch-compatible with the LLVM libraries shipped with the target system.
This can save time by avoiding building LLVM altogether, but as distributions LLVM libraries may have more dependencies and other bloat than the embedded LLVM, this can result in a larger executable. On Ubuntu, the size of the bpftrace executable is increased from 36MB to 75MB (as of Jan 13 2020).
This is still a desirable time-saver where space doesn't matter, such as when doing Debug builds - it can avoid building 15GB of LLVM libraries.
The LLVM builds maintained by distributions have patches on top of the tagged releases. It may be favorable from time-to-time to pull in these patches.
By downloading patches from an external source and writing a custom series
file, quilt can be used to apply arbitrary patches to embedded sources.
The embed_helpers.cmake file has the necessary helper functions to download
patches, add them to a patch series, and set the patch command to be used with
the ExternalProject.
These are currently used to patch the embedded libclang build to link to system libraries on Debian.
Luckily this is actually pulling patches directly from LLVM's own build repo, which is linked to by https://apt.llvm.org/. You can see the branch associated with the LLVM version. This shows the patch series used to build LLVM 9, etc for Debian.
In the case of LLVM 8, to get libclang.a to build only one patch, is needed, not the whole series. The worst possible case in adding support for a new LLVM version is to copy the series file, and set "-p2" or "-p1" depending on if it is LLVM itself or a subproject, and what patch level it was written at.
The workflow on a new LLVM release support linking to system libs would be to:
- Try it with the existing patch series and maybe it just works
- Examine the build failure, and grep for patches affecting these files from those in the patch series on the LLVM 9 / 10 etc branch
- Add patches to the series file until there is a minimal one that builds. Most of the time the patches change minor cosmetic things, no major functionality. In this case, the patch makes it from KFreebsd to kFreebsd which breaks a header, but it is easy to fix
In order to build successfully in an environment like Travis CI, it is necessary that build jobs complete in less than 50 minutes. Unfortunately, it takes about 50 minutes to build just LLVM dependencies alone on Travis.
In order to build a new version of LLVM, or LLVM from a cold cache, the build
must be done incrementally. This is done with controlled timeouts, to make the
build complete within the 50 minute window. This allows for progress to be
saved, and another CI job to pick up where it left off. This is done by setting
the CI_TIMEOUT environment variable, currently to 40 minutes (to allow time
for static bcc to build and cache artifacts to be uploaded).
It may take several successive builds to rebuild the LLVM and Clang embedded library dependencies, when a new embedded LLVM target is added or the cache is cleared. Once the cache is warmed, the build times should be comparable to other builds.
The debug build is too large to complete successfully on Travis CI. The debug artifacts for LLVM and Clang build process are around 35GB, which simply takes too long to save and restore on travis, resulting in builds being killed due to travis' 10 minute no-output timeout. This makes the incremental approach used to warm the cache for the Release binary impracticable.
Debug builds ultimately produce a bpftrace binary that is 1.2GB, and not practical for redistribution. When run locally or on a less restricted CI environment, embedded Debug build should still complete and pass all tests, but it may take 1-2 hours for this to complete from a cold cache.
As a way to ensure that debug builds can succeed, the system LLVM may be linked
to by setting EMBED_LLVM=OFF.
This is the default, and currently only tested and supported.
bpftrace is built against the glibc from Ubuntu bionic (18.04), which provides 2.27.
The original static build of bpftrace built against musl statically, but there is no reason why it could not link against musl dynamically.
It should be possible to link to bionic libc for Android, allowing for bpftrace to run on Android systems with Kernels that support it.
Theoretically no reason this shouldn't work to dynamically link to, may be helpful for supporting bpftrace on embedded environments.