Skip to content

Latest commit

 

History

History
306 lines (204 loc) · 11.8 KB

technical_overview.md

File metadata and controls

306 lines (204 loc) · 11.8 KB
Raw

1 Introduction

Isar can be shortly described as a set of bitbake recipes that implement main build system logic. To simplify overall Isar understanding, this document is split into two main parts:

  • Isar logical components

  • Isar internal processes

2 Isar Logical Components

In this chapter the most important Isar components are considered in details. In this text component doesn't especially mean some self-contained single entity of something, it's just an attempt to split Isar internals by various criteria.

2.1 Bitbake and Recipes

All the processes in Isar are started by bitbake, so it manages general build executing process. Recipes in Isar can be split in two categories:

  • System recipes and classes: they are responsible for setting up Debian-like infrastructure, manages Debian tools execution for package building and installation, generation root file system images

  • User recipes: custom user applications, that can be built from sources

There are two types of dependencies in Isar:

  • Dependency between bitbake recipes

  • Dependencies in Debian filesystem

NOTE: Isar doesn't manage dependencies in Debian file systems. User can only list specific Debian dependencies in recipe, so they eventually will be passed to apt-get. Dependency installation is managed by Debian tools.

2.2 Stamps

Each task managed by bitbake uses stamp to notify that it has been completed. Due to Isar supports various Debian distributions and parallel builds for multiple machines and architectures, the stamps for tasks use special suffixes that include:

  • Debian distro name

  • Architecture

  • Machine

Typical example, when Isar builds the following configurations:

  • Debian Bullseye, amd64

  • Debian Bullseye, arm64

  • Debian Bookworm, arm

In this case there will be 3 different build subtrees. So the standard hello-isar demo application will be processed 3 times for each environment. Three different sets of stamps will be used to distinguish build progress between different bitbake configurations.

2.3 Schroot rootfs

Also called sbuild chroot rootfs, because of the usage by sbuild tool. One of the key aspect of Debian philosophy claims the fact, that everything in Debian should be built within Debian environment. To follow this rule, Isar introduces the new component - schroot rootfs. Schroot rootfs is typical Debian filesystem that is created using standard Debian tools: debootstrap/mmdebstrap, apt-get. The source of packages can be either official Debian repositories or custom repositories created by user. It could be one of the following type:

  • sbuild-chroot-host with the host architecture for cross-compilation.

  • sbuild-chroot-target with the target architecture for target native compilation.

Schroot rootfs lifecycle can be described as following:

  • Schroot rootfs starts out with a minimal base system that was generated by the isar-bootstrap-{host,target}/isar-mmdebstrap-{host,target} recipe that uses debootstrap/mmdebstrap respectively and apt-get.

  • Common development packages needed for packages building are installed on top of this rootfs.

2.4 Target Root Filesystem

Target filesystem is quite similar to schroot rootfs. The only difference is that it doesn't have development packages installed and always has target architecture.

Target filesystem lifecycle can be described as following:

  • Target filesystem starts, like the schroot rootfs, with the same minimal base system generated by the isar-bootstrap recipe.

  • According to the list of custom packages in bitbake recipes, the initial filesystem will be populated by successfully built packages from locally created Isar repository.

3 Isar Internal Processes

3.1 General Overview

Whole Isar build process can be split into the following steps:

  • Generation of a minimal base system by isar-bootstrap/isar-mmdebstrap using debootstrap/mmdebstrap and apt for each configuration requested by the user (Debian distro, machine and architecture).

  • Generation of initial schroot rootfs for each configuration.

  • Generation of initial target filesystems for each configuration.

  • Building custom packages.

  • Populate target filesystems.

  • Generate bootable images.

All these steps are described in details below.

3.2 Minimal base system

A minimal base system is generated by the isar-bootstrap recipe found here: meta/recipes-core/isar-bootstrap/isar-bootstrap-[host,target].bb.

This recipe implements several tasks which are executed in this order:

  1. do_generate_keyring: Generates a new gpg keyring from the gpg keys specified in the APTKEYFILES variable. This keyring is later used by debootstrap/mmdebstrap and apt to verify the apt repositories.

  2. do_apt_config_prepare: Generates aggregated apt preferences and source files based on the DISTRO_APT_PREFERENCES and DISTRO_APT_SOURCES variables. That are later deployed to the minimal base system to be used by apt itself.

  3. do_bootstrap: Bootstraps a minimal debian base system using debootstrap/mmdebstrap.

  4. do_apt_config_install: Installs the apt configuration generated by do_apt_config_prepare to the base system as well as an apt base configuration that ensures no recommended or suggested package are installed automatically. This setting keeps the system minimal for all further steps.

  5. do_apt_update: Performs an initial apt-get update and apt-get dist-upgrade on the minimal system. This is necessary because debootstrap itself only allows using one repository. This task ensures that more up to date packages, for instance from security repos are installed if they exist.

  6. do_deploy: Creates a symlink to the directory containing the generated minimal base system, that can be used by the other recipes, that depend on isar-bootstrap.

3.3 Initial Schroot Rootfs Generation

As mentioned above, initial schroot rootfs is generated using the minimal base system generated by isar-bootstrap. The bitbake recipe which is responsible for schroot rootfs can be found here: meta/recipes-devtools/sbuild-chroot/sbuild-chroot-{host,target}.bb

Section 3.8 describes when host or target architecture is used, please refer to it for more details.

This recipe implements rootfs preparation for the next schroot usage and performs the following:

  1. Calls rootfs_prepare helper function that copies the minimal base system to the schroot rootfs working directory.

  2. Sets up the isar-apt repository which will later contain all isar-build packages. Also configure apt to retry and not install recommended packages.

  3. Mounts dev, proc, sys and also isar-apt, base-apt into schroot rootfs.

  4. Calls apt-get update, imports package download cache and installs the packages specified as an argument to this rootfs. Those are, in schroot rootfs case, the basic package building toolchain.

The single stamp is created for each user schroot rootfs configuration.

3.4 Initial Target Filesystem Generation

Initial target filesystem generation process is very similar to schroot rootfs creating, the difference is only in initial packages list.

Target image recipes are the part of Isar core. There is a sample of typical Isar image that can be customized according to the user requirements: meta-isar/recipes-core/images/isar-image-base.bb.

3.5 Building Custom Packages

Isar provides the possibility to build Debian packages from sources. This feature works with Debian-like source packages, i.e. the source code tree should contain debian folder. This build process is implemented in meta/classes/dpkg.bbclass. Moreover this process is common for both: native and cross compilation modes.

Another way of creating Debian packages is implemented in meta/classes/dpkg-raw.bbclass. This class can be used for customizations that are not based on source code that should become a package.

Both consist of the following steps:

  1. Task do_fetch: fetch files from external links

  2. Task do_unpack: unpack those files to ${WORKDIR}

  3. Task do_install only for dpkg-raw: copy all you want in your debian package to ${D}, install hooks in ${D}/DEBIAN

  4. Task do_prepare: Perform any preparation steps to the unpacked/patched sources before the build. This task calls the dpkg_prepare shell function with the schroot rootfs mounts in place (dpkg_do_mounts).

    4.1. the dpkg_prepare function of dpkg.bbclass runs /isar/deps.sh in the chroot rootfs. That performs the following:

     1. Go to `/home/build/${PN}`

 2. Get list of dependencies from debian/control and install them

    4.2. the dpkg_prepare function of dpkg-raw.bbclass translate the recipe meta-data into a debian/control file suitable for packaging with dpkg-deb

  5. Task do_build: Mount folder (dpkg_do_mounts) with unpacked files to schroot rootfs, execute the actual build function dpkg_runbuild, and finally unmount again (dpkg_undo_mounts).

    5.1. the dpkg_runbuild function of dpkg.bbclass runs build.sh in the schroot rootfs. That performs the following:

     1. Go to `/home/build/${PN}`

 2. Run dpkg-buildpackage

    5.2. the dpkg_runbuild function of dpkg-raw.bbclass basically runs dpkg-deb to construct a Debian package from a folder of files, without compiling anything

  6. Task do_deploy_deb: install successfully built packages ${WORKDIR}/*.deb to deploy directory ${DEPLOY_DIR_DEB}

3.6 Populate Target Filesystem

Each target image can be extended by custom packages listed in IMAGE_INSTALL variable. Task do_populate performs the following:

  1. Parse IMAGE_INSTALL variable.

  2. Find respective packages in ${DEPLOY_DIR_DEB}.

  3. Copy them to deb folder in dedicated target filesystem.

  4. Execute dpkg command in chroot for all the copied packages.

3.7 Generate an Image

This process contains the following steps:

  1. Shell function IMAGE_CMD:*: creates filesystem or disk images

    1.1. IMAGE_CMD:ext4: target filesystem is packed to extfs image. 1.2. IMAGE_CMD:wic: a bootable disk image gets created for the platform

3.8 Cross-compilation

Isar provides the possibility to cross-compile all or individual packages. If ISAR_CROSS_COMPILE variable is set to 0, the native toolchain will be used (e.g., when building for arm64 on amd64, the arm64 gcc binaries will be executed in an emulated environment). If set to 1, the cross-toolchain will be used (the amd64 cross-gcc binaries will be executed). The default is 0. The variable can be set globally for the whole build (e.g., in local.conf or multiconfig files) or overridden in individual recipes. Cross-compilation support must be implemented in source packages. Debian's standard cross-toolchains are used for cross-building.

3.9 Additional features

3.9.1 Template files

A basic templating system implemented on top envsubst(1) is available. It allows limited access to bitbake variables for file generation.

This system is implemented in the template.bbclass and defines the do_transform_template task, the TEMPLATE_FILES and TEMPLATE_VARS variables. Tasks that use need to use files generated via templates need to be executed after the do_transform_template task.

The TEMPLATE_FILES variable contains a space separated list of template files and the TEMPLATE_VARS variable a space separated list of bitbake variable names. The do_transform_template task takes these and generates files using envsubst(1). The output files are placed in the same directory as the template files. The name of the output files are either the name of the template file with .tmpl removed from the file name end or, if the template file did not end with .tmpl, the name of the template file with .out added to the end.

Only template files from the WORKDIR are accepted. Either specify relative paths based on the recipes WORKDIR or absolute paths containing the WORKDIR in the TEMPLATE_FILES variable.