Add image building workflow using disk image builder

Signed-off-by: Sunnatillo <sunnat.samadov@est.tech>

jenkins/image_building/README.md

@@ -0,0 +1,61 @@
+# Diskimage Builder
+
+As mentioned in the [documentation](https://docs.openstack.org/diskimage-builder/latest/index.html),
+Diskimage Builder is a tool for automatically building customized operating
+system images for use in clouds and other environments.
+
+We utilize Diskimage Builder for building metal3-dev images.
+
+## Elements
+
+In Diskimage Builder (DIB), an "element" is a modular and reusable component
+that defines a specific aspect of a disk image. Elements are like building
+blocks used in the image creation process, allowing users to customize and
+extend image functionality.
+
+Each element handles a specific task, such as installing packages or modifying
+files. Users choose which elements to include, offering flexibility. When
+creating a custom image with Diskimage Builder, users select elements, and
+these are combined to form the final disk image. Examples of elements include
+"base," "apache," or "cloud-init," each focusing on a specific part of the
+image's configuration.
+
+## Custom Elements
+
+For metal3-dev image building purposes, we create three custom elements:
+ci-base, ubuntu-ci, and centos-ci elements. The ci-base element is for
+installing common packages and configurations for both Ubuntu and CentOS. The
+ubuntu-ci and centos-ci elements are for installing packages and configuring
+the respective operating system images. More information on developing custom
+elements can be found [here](https://docs.openstack.org/diskimage-builder/latest/developer/developing_elements.html).
+
+## Building an Image with Diskimage Builder
+
+We use the following command to build an image:
+
+```bash
+disk-image-create --no-tmpfs -a amd64 ubuntu-ci ubuntu -o "${CI_IMG_NAME}"
+block-device-efi
+```
+
+* **--no-tmpfs**: This flag specifies that the temporary file system (tmpfs)
+should not be used during the image creation process. Tmpfs is a file system
+that resides in memory, and using this flag indicates that temporary files
+should be written directly to disk instead of in-memory.
+
+* **-a amd64***: This option specifies the architecture of the image. In this
+case, it is set to amd64, indicating a 64-bit x86 architecture.
+
+* **ubuntu-ci**, **ubuntu**: These are the elements or components used in
+building the image. The image is based on the "ubuntu-ci" element, a
+development environment for Ubuntu. Additionally, the "ubuntu" element is
+specified, likely including the base configuration for an Ubuntu-based image.
+
+* **-o** **"${CI_IMG_NAME}"**: This option specifies the output file or
+image name. The value is provided through the variable ${CI_IMG_NAME}.
+
+* **block-device-efi**: This is an additional element specified for image
+creation. It likely includes configurations or tasks related to block devices
+and EFI (Extensible Firmware Interface), commonly used in modern systems for booting.
+
+More information on building and image via Diskimage Builder can be found [here](https://docs.openstack.org/diskimage-builder/latest/user_guide/building_an_image.html).

jenkins/image_building/build-ci-image.sh

@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+
+set -eux
+
+export IMAGE_OS="${IMAGE_OS}"
+
+current_dir="$(dirname "$(readlink -f "${0}")")"
+
+# Disable needrestart interactive mode
+sudo sed -i "s/^#\$nrconf{restart} = 'i';/\$nrconf{restart} = 'a';/" /etc/needrestart/needrestart.conf   > /dev/null
+
+sudo apt-get update
+
+# Install packages
+sudo apt-get install python3-pip qemu qemu-kvm -y
+sudo pip3 install diskimage-builder python-openstackclient
+
+export ELEMENTS_PATH="${current_dir}/dib_elements"
+export DIB_DEV_USER_USERNAME="metal3ci"
+export DIB_DEV_USER_PWDLESS_SUDO="yes"
+export DIB_DEV_USER_AUTHORIZED_KEYS="${current_dir}/id_ed25519_metal3ci.pub"
+
+if [[ "${IMAGE_OS}" == "ubuntu" ]]; then
+  export DIB_RELEASE=jammy
+else
+  export DIB_RELEASE=9
+fi 
+
+# Set image names
+commit_short="$(git rev-parse --short HEAD)"
+img_date="$(date --utc +"%Y%m%dT%H%MZ")"
+
+final_ci_img_name="metal3-ci-${IMAGE_OS}"
+ci_img_name="${final_ci_img_name}-${img_date}-${commit_short}"
+
+# Create an image
+disk-image-create --no-tmpfs -a amd64 "${IMAGE_OS}"-ci "${IMAGE_OS}" -o "${ci_img_name}" block-device-efi
+
+# Push image to openstack
+openstack image create "${final_ci_img_name}" --file "${ci_img_name}".qcow2 --disk-format=qcow2

jenkins/image_building/dib_elements/centos-ci/README.md

@@ -0,0 +1,13 @@
+# centos-ci element
+
+## Overview
+
+**centos-ci** element installs packages and makes configuration changes
+configuration specifically for centos-ci images. This element's script run in
+post-install.d step. It consists of two shell scripts: ***01_install*** and
+***02_configure***.
+
+## Depends
+
+* [centos](https://docs.openstack.org/diskimage-builder/latest/elements/centos/README.html)
+* ci-base

jenkins/image_building/dib_elements/centos-ci/element-deps

@@ -0,0 +1,2 @@
+centos
+ci-base

jenkins/image_building/dib_elements/centos-ci/install.d/install

@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+set -eux
+
+sudo dnf distro-sync -y
+
+# Install EPEL repo (later required by atop, python3-bcrypt and python3-passlib)
+sudo dnf install epel-release -y
+
+# Install podman
+sudo dnf install podman -y
+
+# Without this minikube cannot start properly kvm and fails.
+# As a simple workaround, this will create an empty file which can
+# disable the new firmware, more details here [1], look for firmware description.
+# [1] <https://libvirt.org/formatdomain.html#operating-system-booting>
+# upstream commit fixing the behavior to not print error messages for unknown features
+# will be included in RHEL-AV-8.5.0 by next rebase to libvirt 7.4.0.
+sudo mkdir -p /etc/qemu/firmware
+sudo touch /etc/qemu/firmware/50-edk2-ovmf-cc.json

jenkins/image_building/dib_elements/centos-ci/post-install.d/configure

@@ -0,0 +1,30 @@
+#!/usr/bin/env bash
+
+# Configure network (set nameservers and disable peer DNS).
+set -eux
+
+sudo sed -i "0,/.*PermitRootLogin.*/s//PermitRootLogin yes/" /etc/ssh/sshd_config
+
+# SETUP MONITORING
+## Install atop and sysstat
+sudo dnf install sysstat atop --enablerepo=epel -y 
+
+## Collect all metrics every minute
+sudo sed -i 's/^LOGINTERVAL=600.*/LOGINTERVAL=60/' /etc/sysconfig/atop
+sudo mkdir -v /etc/systemd/system/sysstat-collect.timer.d/
+sudo bash -c "sed -e 's|every 10 minutes|every 1 minute|g' -e '/^OnCalendar=/ s|/10$|/1|' /usr/lib/systemd/system/sysstat-collect.timer > /etc/systemd/system/sysstat-collect.timer.d/override.conf"
+sudo sed -i 's|^SADC_OPTIONS=.*|SADC_OPTIONS=" -S XALL"|' /etc/sysconfig/sysstat
+
+## Reduce metrics retention to 3 days
+sudo sed -i 's/^LOGGENERATIONS=.*/LOGGENERATIONS=3/' /etc/sysconfig/atop
+sudo sed -i 's|^HISTORY=.*|HISTORY=3|' /etc/sysconfig/sysstat
+
+## Standardize sysstat log directory
+sudo mkdir -p /var/log/sysstat
+sudo sed -i 's|^SA_DIR=.*|SA_DIR="/var/log/sysstat"|' /etc/sysconfig/sysstat
+
+## Enable services
+sudo systemctl enable atop.service crond.service sysstat.service
+
+# Change default to shell to bash
+sudo usermod --shell /bin/bash metal3ci

jenkins/image_building/dib_elements/ci-base/README.md

@@ -0,0 +1,18 @@
+# ci-base element
+
+## Overview
+
+This element takes care of installing common packages both for ubuntu and
+centos ci images. **ci-base** element utilizes package-installs to declarative
+method of installing packages for image build.
+
+## Depends
+
+ci-base element depends following elements.
+
+* [base](https://docs.openstack.org/diskimage-builder/latest/elements/base/README.html)
+* [vm](https://docs.openstack.org/diskimage-builder/latest/elements/vm/README.html)
+* [devuser](https://docs.openstack.org/diskimage-builder/latest/elements/devuser/README.html)
+* [openssh-server](https://docs.openstack.org/diskimage-builder/latest/elements/openssh-server/README.html)
+* [pkg-map](https://docs.openstack.org/diskimage-builder/latest/elements/pkg-map/README.html)
+* [package-installs](https://docs.openstack.org/diskimage-builder/latest/elements/package-installs/README.html)

jenkins/image_building/dib_elements/ci-base/element-deps

@@ -0,0 +1,6 @@
+base
+vm
+devuser
+openssh-server
+pkg-map
+package-installs

jenkins/image_building/dib_elements/ci-base/package-installs.yaml

@@ -0,0 +1,15 @@
+bash-completion:
+curl:
+dnsmasq:
+git:
+jq:
+libguestfs-tools:
+make:
+openjdk-11-jre:
+ovmf:
+python3:
+python3-pip:
+qemu-kvm:
+tree:
+vim:
+wget:

jenkins/image_building/dib_elements/ci-base/pkg-map

@@ -0,0 +1,43 @@
+{
+    "release": {
+      "ubuntu": {
+        "22.04": {
+          "ntp": "chrony"
+        }
+      }
+    },
+    "family": {
+        "redhat": {
+          "bash-completion":"bash-completion",
+          "curl": "curl",
+          "dnsmasq":"dnsmasq",
+          "git": "git",
+          "libguestfs-tools":"libguestfs-tools",
+          "make": "make",
+          "openjdk-11-jre": "java-11-openjdk",
+          "ovmf":"edk2-ovmf",
+          "python3": "python3",
+          "python3-pip":"python3-pip",
+          "qemu-kvm":"qemu-kvm",
+          "tree": "tree",
+          "vim": "vim-enhanced",
+          "wget": "wget"
+        },
+        "debian":{
+          "bash-completion":"bash-completion",
+          "curl": "curl",
+          "dnsmasq":"dnsmasq",
+          "git": "git",
+          "libguestfs-tools":"libguestfs-tools",
+          "make": "make",
+          "openjdk-11-jre": "openjdk-11-jre",
+          "ovmf":"ovmf",
+          "python3": "python3",
+          "python3-pip":"python3-pip",
+          "qemu-kvm":"qemu-kvm",
+          "tree": "tree",
+          "vim": "vim",
+          "wget": "wget"
+        }
+    }
+}

jenkins/image_building/dib_elements/ubuntu-ci/README.md

@@ -0,0 +1,13 @@
+# ubuntu-ci element
+
+## Overview
+
+**ubuntu-ci** element installs packages and makes configuration changes
+configuration specifically for centos-ci images. This element's script run in
+post-install.d step. It consists of two shell scripts: ***01_install*** and
+***02_configure***.
+
+## Depends
+
+* [ubuntu](https://docs.openstack.org/diskimage-builder/latest/elements/ubuntu/README.html)
+* ci-base

jenkins/image_building/dib_elements/ubuntu-ci/element-deps

@@ -0,0 +1,2 @@
+ubuntu
+ci-base

jenkins/image_building/dib_elements/ubuntu-ci/install.d/install

@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+
+set -euxo pipefail
+
+sudo apt-get update
+sudo apt-get install -y \
+  coreutils \
+  apt-transport-https \
+  ca-certificates \
+  gnupg-agent \
+  software-properties-common \
+  openssl \
+  python-is-python3 \
+  chrony \
+  qemu
+
+# Configure
+sudo chronyc -a 'burst 4/4' && sudo chronyc -a makestep
+sudo systemctl enable chrony
+sudo systemctl start chrony
+
+# Enable nested virtualization
+sudo bash -c 'cat << EOF > /etc/modprobe.d/qemu-system-x86.conf
+options kvm-intel nested=y enable_apicv=n
+EOF'
+echo "Reboot required"
+
+# Install Docker
+sudo mkdir -m 0755 -p /etc/apt/keyrings
+curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
+sudo echo \
+  "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
+  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list
+
+sudo apt-get update
+sudo apt-get install docker-ce docker-ce-cli containerd.io jq -y
+sudo groupadd docker || true
+sudo usermod -aG docker metal3ci || true
+sudo systemctl enable docker
+sudo systemctl restart docker