Add image building workflow using disk image builder

Signed-off-by: Sunnatillo <sunnat.samadov@est.tech>

jenkins/image_building/README.md

@@ -0,0 +1,61 @@
+# Diskimage Builder
+
+As mentioned in the [documentation](https://docs.openstack.org/diskimage-builder/latest/index.html),
+Diskimage Builder is a tool for automatically building customized operating
+system images for use in clouds and other environments.
+
+We utilize Diskimage Builder for building metal3-dev images.
+
+## Elements
+
+In Diskimage Builder (DIB), an "element" is a modular and reusable component
+that defines a specific aspect of a disk image. Elements are like building
+blocks used in the image creation process, allowing users to customize and
+extend image functionality.
+
+Each element handles a specific task, such as installing packages or modifying
+files. Users choose which elements to include, offering flexibility. When
+creating a custom image with Diskimage Builder, users select elements, and
+these are combined to form the final disk image. Examples of elements include
+"base," "apache," or "cloud-init," each focusing on a specific part of the
+image's configuration.
+
+## Custom Elements
+
+For metal3-dev image building purposes, we create three custom elements:
+dev-base, ubuntu-dev, and centos-dev elements. The dev-base element is for
+installing common packages and configurations for both Ubuntu and CentOS. The
+ubuntu-dev and centos-dev elements are for installing packages and configuring
+the respective operating system images. More information on developing custom
+elements can be found [here](https://docs.openstack.org/diskimage-builder/latest/developer/developing_elements.html).
+
+## Building an Image with Diskimage Builder
+
+We use the following command to build an image:
+
+```bash
+disk-image-create --no-tmpfs -a amd64 ubuntu-dev ubuntu -o "${CI_IMG_NAME}"
+block-device-efi
+```
+
+* **--no-tmpfs**: This flag specifies that the temporary file system (tmpfs)
+should not be used during the image creation process. Tmpfs is a file system
+that resides in memory, and using this flag indicates that temporary files
+should be written directly to disk instead of in-memory.
+
+* **-a amd64***: This option specifies the architecture of the image. In this
+case, it is set to amd64, indicating a 64-bit x86 architecture.
+
+* **ubuntu-dev**, **ubuntu**: These are the elements or components used in
+building the image. The image is based on the "ubuntu-dev" element, a
+development environment for Ubuntu. Additionally, the "ubuntu" element is
+specified, likely including the base configuration for an Ubuntu-based image.
+
+* **-o** **"${CI_IMG_NAME}"**: This option specifies the output file or
+image name. The value is provided through the variable ${CI_IMG_NAME}.
+
+* **block-device-efi**: This is an additional element specified for image
+creation. It likely includes configurations or tasks related to block devices
+and EFI (Extensible Firmware Interface), commonly used in modern systems for booting.
+
+More information on building and image via Diskimage Builder can be found [here](https://docs.openstack.org/diskimage-builder/latest/user_guide/building_an_image.html).

jenkins/image_building/build-ci-image.sh

@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+
+set -eux
+
+export IMAGE_OS="${IMAGE_OS}"
+
+current_dir="$(dirname "$(readlink -f "${0}")")"
+
+# Disable needrestart interactive mode
+sudo sed -i "s/^#\$nrconf{restart} = 'i';/\$nrconf{restart} = 'a';/" /etc/needrestart/needrestart.conf   > /dev/null
+
+sudo apt-get update
+
+# Install packages
+sudo apt-get install python3-pip qemu qemu-kvm -y
+sudo pip3 install diskimage-builder python-openstackclient
+
+# shellcheck disable=SC1091
+. "$current_dir"/dib-and-image-vars.sh
+
+# Create an image
+disk-image-create --no-tmpfs -a amd64 "${IMAGE_OS}"-ci "${IMAGE_OS}" -o "${CI_IMG_NAME}" block-device-efi
+
+# shellcheck disable=SC1091
+. "$current_dir"/openstack-vars.sh
+
+# Push image to openstack
+openstack image create "${FINAL_CI_IMG_NAME}" --file "${CI_IMG_NAME}".qcow2 --disk-format=qcow2

jenkins/image_building/dib-and-image-vars.sh

@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+
+set -eux
+
+current_dir="$(dirname "$(readlink -f "${0}")")"
+
+export ELEMENTS_PATH="${current_dir}/dib_elements"
+export DIB_DEV_USER_USERNAME="metal3ci"
+export DIB_DEV_USER_PWDLESS_SUDO="yes"
+export DIB_DEV_USER_AUTHORIZED_KEYS="${current_dir}/id_ed25519_metal3ci.pub"
+export DIB_RELEASE=9
+
+if [[ "${IMAGE_OS}" == "ubuntu" ]]; then
+  export DIB_RELEASE=jammy
+else
+  export DIB_RELEASE=9
+fi 
+
+# Set image names
+commit_short="$(git rev-parse --short HEAD)"
+img_date="$(date --utc +"%Y%m%dT%H%MZ")"
+
+export FINAL_CI_IMG_NAME="metal3-ci-${IMAGE_OS}"
+export CI_IMG_NAME="${FINAL_CI_IMG_NAME}-${img_date}-${commit_short}"

jenkins/image_building/dib_elements/centos-ci/README.md

@@ -0,0 +1,13 @@
+# centos-ci element
+
+## Overview
+
+**centos-ci** element installs packages and makes configuration changes
+configuration specifically for centos-ci images. This element's script run in
+post-install.d step. It consists of two shell scripts: ***01_install*** and
+***02_configure***.
+
+## Depends
+
+* [centos](https://docs.openstack.org/diskimage-builder/latest/elements/centos/README.html)
+* ci-base

jenkins/image_building/dib_elements/centos-ci/element-deps

@@ -0,0 +1,3 @@
+centos
+ci-base
+openssh-server

jenkins/image_building/dib_elements/centos-ci/post-install.d/01_install

@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+set -eux
+
+sudo dnf distro-sync -y
+
+# Install EPEL repo (later required by atop, python3-bcrypt and python3-passlib)
+sudo dnf install epel-release -y
+
+# Install podman
+sudo dnf install podman -y
+
+# Without this minikube cannot start properly kvm and fails.
+# As a simple workaround, this will create an empty file which can
+# disable the new firmware, more details here [1], look for firmware description.
+# [1] <https://libvirt.org/formatdomain.html#operating-system-booting>
+# upstream commit fixing the behavior to not print error messages for unknown features
+# will be included in RHEL-AV-8.5.0 by next rebase to libvirt 7.4.0.
+sudo mkdir -p /etc/qemu/firmware
+sudo touch /etc/qemu/firmware/50-edk2-ovmf-cc.json

jenkins/image_building/dib_elements/centos-ci/post-install.d/02_configure

@@ -0,0 +1,33 @@
+#!/usr/bin/env bash
+
+# Configure network (set nameservers and disable peer DNS).
+set -eux
+
+sudo sed -i "0,/.*PermitRootLogin.*/s//PermitRootLogin yes/" /etc/ssh/sshd_config
+
+# RESET CLOUD INIT
+# Following will remove any cloud init's previous run
+# data and force cloud-init to again on next boot.
+
+sudo rm -rf /var/lib/cloud/*
+
+# SETUP MONITORING
+## Install atop and sysstat
+sudo dnf install sysstat atop --enablerepo=epel -y 
+
+## Collect all metrics every minute
+sudo sed -i 's/^LOGINTERVAL=600.*/LOGINTERVAL=60/' /etc/sysconfig/atop
+sudo mkdir -v /etc/systemd/system/sysstat-collect.timer.d/
+sudo bash -c "sed -e 's|every 10 minutes|every 1 minute|g' -e '/^OnCalendar=/ s|/10$|/1|' /usr/lib/systemd/system/sysstat-collect.timer > /etc/systemd/system/sysstat-collect.timer.d/override.conf"
+sudo sed -i 's|^SADC_OPTIONS=.*|SADC_OPTIONS=" -S XALL"|' /etc/sysconfig/sysstat
+
+## Reduce metrics retention to 3 days
+sudo sed -i 's/^LOGGENERATIONS=.*/LOGGENERATIONS=3/' /etc/sysconfig/atop
+sudo sed -i 's|^HISTORY=.*|HISTORY=3|' /etc/sysconfig/sysstat
+
+## Standardize sysstat log directory
+sudo mkdir -p /var/log/sysstat
+sudo sed -i 's|^SA_DIR=.*|SA_DIR="/var/log/sysstat"|' /etc/sysconfig/sysstat
+
+## Enable services
+sudo systemctl enable atop.service crond.service sysstat.service

jenkins/image_building/dib_elements/ci-base/README.md

@@ -0,0 +1,18 @@
+# ci-base element
+
+## Overview
+
+This element takes care of installing common packages both for ubuntu and
+centos ci images. **ci-base** element utilizes package-installs to declarative
+method of installing packages for image build.
+
+## Depends
+
+ci-base element depends following elements.
+
+* [base](https://docs.openstack.org/diskimage-builder/latest/elements/base/README.html)
+* [vm](https://docs.openstack.org/diskimage-builder/latest/elements/vm/README.html)
+* [devuser](https://docs.openstack.org/diskimage-builder/latest/elements/devuser/README.html)
+* [openssh-server](https://docs.openstack.org/diskimage-builder/latest/elements/openssh-server/README.html)
+* [pkg-map](https://docs.openstack.org/diskimage-builder/latest/elements/pkg-map/README.html)
+* [package-installs](https://docs.openstack.org/diskimage-builder/latest/elements/package-installs/README.html)

jenkins/image_building/dib_elements/ci-base/element-deps

@@ -0,0 +1,6 @@
+base
+vm
+devuser
+openssh-server
+pkg-map
+package-installs

jenkins/image_building/dib_elements/ci-base/package-installs.yaml

@@ -0,0 +1,15 @@
+bash-completion:
+curl:
+dnsmasq:
+git:
+jq:
+libguestfs-tools:
+make:
+openjdk-11-jre:
+ovmf:
+python3:
+python3-pip:
+qemu-kvm:
+tree:
+vim:
+wget:

jenkins/image_building/dib_elements/ci-base/pkg-map

@@ -0,0 +1,43 @@
+{
+    "release": {
+      "ubuntu": {
+        "22.04": {
+          "ntp": "chrony"
+        }
+      }
+    },
+    "family": {
+        "redhat": {
+          "bash-completion":"bash-completion",
+          "curl": "curl",
+          "dnsmasq":"dnsmasq",
+          "git": "git",
+          "libguestfs-tools":"libguestfs-tools",
+          "make": "make",
+          "openjdk-11-jre": "java-11-openjdk",
+          "ovmf":"edk2-ovmf",
+          "python3": "python3",
+          "python3-pip":"python3-pip",
+          "qemu-kvm":"qemu-kvm",
+          "tree": "tree",
+          "vim": "vim-enhanced",
+          "wget": "wget"
+        },
+        "debian":{
+          "bash-completion":"bash-completion",
+          "curl": "curl",
+          "dnsmasq":"dnsmasq",
+          "git": "git",
+          "libguestfs-tools":"libguestfs-tools",
+          "make": "make",
+          "openjdk-11-jre": "openjdk-11-jre",
+          "ovmf":"ovmf",
+          "python3": "python3",
+          "python3-pip":"python3-pip",
+          "qemu-kvm":"qemu-kvm",
+          "tree": "tree",
+          "vim": "vim",
+          "wget": "wget"
+        }
+    }
+}

jenkins/image_building/dib_elements/ci-base/post-root.d/image_prepull

@@ -0,0 +1,68 @@
+#!/usr/bin/env bash
+
+set -euxo pipefail
+
+# IMAGE_NAME conflicts with IMAGE_NAME variable in metal3-dev-env, unset it
+unset IMAGE_NAME
+
+# Install container runtime if OS is ubuntu. Centos comes with podman installed
+
+if [[ "${IMAGE_OS}" == "ubuntu" ]]; then
+    export CONTAINER_RUNTIME="docker"
+    # Install Docker
+    sudo mkdir -m 0755 -p /etc/apt/keyrings
+    echo 'y' | curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
+    sudo echo \
+    "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
+    $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list
+
+    sudo apt-get update
+    sudo apt-get install docker-ce docker-ce-cli containerd.io jq -y
+    sudo groupadd docker || true
+    sudo usermod -aG docker metal3ci || true
+    sudo systemctl enable docker
+    sudo systemctl restart docker 
+else 
+    # Install images for downloading container images for centos
+    sudo apt-get install podman -y
+fi
+
+M3_DENV_ORG="${M3_DENV_ORG:-Nordix}"
+M3_DENV_REPO="${M3_DENV_REPO:-metal3-dev-env}"
+M3_DENV_URL="${M3_DENV_URL:-https://github.com/${M3_DENV_ORG}/${M3_DENV_REPO}.git}"
+M3_DENV_BRANCH="${M3_DENV_BRANCH:-add-image-prepull-make-target-sunnat}"
+M3_DENV_ROOT="${M3_DENV_ROOT:-/tmp}"
+M3_DENV_PATH="${M3_DENV_PATH:-${M3_DENV_ROOT}/${M3_DENV_REPO}}"
+export FORCE_REPO_UPDATE="${FORCE_REPO_UPDATE:-true}"
+export CONTAINER_RUNTIME="${CONTAINER_RUNTIME:-podman}"
+export IMAGE_OS="${IMAGE_OS:-Ubuntu}"
+export EPHEMERAL_CLUSTER="${EPHEMERAL_CLUSTER:-kind}"
+
+# TODO(Sunnatillo): When we copy container images metal3 dev image does not spin up
+# on openstack.
+
+# Install metal3 requirements
+# mkdir -p "${M3_DENV_ROOT}"
+# if [[ -d "${M3_DENV_PATH}" ]] && [[ "${FORCE_REPO_UPDATE}" == "true" ]]; then
+#   sudo rm -rf "${M3_DENV_PATH}"
+# fi
+# if [ ! -d "${M3_DENV_PATH}" ] ; then
+#   pushd "${M3_DENV_ROOT}"
+#   git clone "${M3_DENV_URL}"
+#   popd
+# fi
+# pushd "${M3_DENV_PATH}"
+# git checkout "${M3_DENV_BRANCH}"
+# git pull -r || true
+# make prepull_images
+# popd
+
+# if [[ "${CONTAINER_RUNTIME}" == "docker" ]]; then
+#   sudo mkdir -p "${TMP_BUILD_DIR}/mnt/var/lib/docker/"
+#   sudo cp -a /var/lib/docker/. "${TMP_BUILD_DIR}/mnt/var/lib/docker/"
+# else
+#   sudo mkdir -p "${TMP_BUILD_DIR}/mnt/var/lib/containers/storage/"
+#   sudo cp -a /var/lib/containers/storage/. "${TMP_BUILD_DIR}/mnt/var/lib/containers/storage/"
+# fi
+
+# sudo cp -a /opt/metal3-dev-env/. "${TMP_BUILD_DIR}/mnt/opt/metal3-dev-env/"

jenkins/image_building/dib_elements/ubuntu-ci/README.md

@@ -0,0 +1,13 @@
+# ubuntu-ci element
+
+## Overview
+
+**ubuntu-dev** element installs packages and makes configuration changes
+configuration specifically for centos-ci images. This element's script run in
+post-install.d step. It consists of two shell scripts: ***01_install*** and
+***02_configure***.
+
+## Depends
+
+* [ubuntu](https://docs.openstack.org/diskimage-builder/latest/elements/ubuntu/README.html)
+* ci-base

jenkins/image_building/dib_elements/ubuntu-ci/element-deps

@@ -0,0 +1,2 @@
+ubuntu
+ci-base