Phase II: Assembly - Automate the generation of final packages #76

AlexRuiz7 · 2023-12-11T13:28:56Z

Description

Building packages for wazuh-indexer takes 2 steps:

Build: compiles and packages the wazuh-indexer project.
Assembly: installs plugins and adds configuration files for production.

This issue focuses on the Assembly phase, following the requirements defined in its parent issue.

We need to generate final packages for:

For each of them, these tasks are needed:

Download min package
Decompress
Install plugins
Install configuration files
Compress

Tasks

Functional requirements

The packages can be assembled locally (using bash).
The packages can be assembled locally (using Act).
The packages can be assembled automatically using GH Actions.

The text was updated successfully, but these errors were encountered:

AlexRuiz7 · 2023-12-15T16:06:39Z

RPM

The assemble.sh script will use the output from the build.sh script and use it as a base to bundle together a final package containing the plugins, the production configuration and the service files.

The script will:

Extract the rpm package using rpm2cpio and cpio tools.

By default, rpm2cpio and cpio tools expect the package to be in wazuh-indexer/artifacts/tmp/rpm. The script takes care of creating the required folder structure, copying also the min package and the SPEC file.

Current folder loadout at this stage:
```
/rpm/$ARCH
    /etc
    /usr
    /var
    wazuh-indexer-min-*.rpm
    wazuh-indexer.rpm.spec
```
usr, etc and var folders contain wazuh-indexer files, extracted from wazuh-indexer-min-*.rpm.
wazuh-indexer.rpm.spec is copied over from wazuh-indexer/distribution/packages/src/rpm/wazuh-indexer.rpm.spec. The wazuh-indexer-performance-analyzer.service file is also copied from the same folder. It is a dependency of the SPEC file.
Install the plugins using the opensearch-plugin CLI tool.
Set up configuration files.

Included in min-package. Default files are overwritten.
Bundle an RPM file with rpmbuild and the SPEC file wazuh-indexer.rpm.spec.
- rpmbuild is part of the rpm OS package.
  
  rpmbuild is invoked from wazuh-indexer/artifacts/tmp/rpm. It creates the {BUILD,RPMS,SOURCES,SRPMS,SPECS,TMP} folders and applies the rules in the SPEC file. If successful, rpmbuild will generate the package in the RPMS/ folder. The script will copy it to wazuh-indexer/artifacts/dist and clean: remove the tmp\ folder and its contents.
Current folder loadout at this stage:
```
/rpm/$ARCH
    /{BUILD,RPMS,SOURCES,SRPMS,SPECS,TMP}
    /etc
    /usr
    /var
    wazuh-indexer-min-*.rpm
    wazuh-indexer.rpm.spec
```

Running in Docker

Pre-requisites:

Current directory: wazuh-indexer/
Existing rpm package in wazuh-indexer/artifacts/dist/rpm, as a result of the Build stage.

MIN_PKG_PATH="./artifacts"
docker run --rm \
    -v ./scripts/:/home/wazuh-indexer/scripts \
    -v $MIN_PKG_PATH:/home/wazuh-indexer/artifacts \
    -v ./distribution/packages/src:/home/wazuh-indexer/distribution/packages/src \
    -w /home/wazuh-indexer \
    -it ubuntu:jammy /bin/bash

apt-get update
apt-get install -y rpm2cpio rpm cpio
bash scripts/assemble.sh -v 2.11.0 -p linux -a x64 -d rpm

Issues:

_sharedstatedir is by default set to /usr/com. Fixed with %define _sharedstatedir /var/lib
File not found: /app/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/lib/systemd/system/opensearch-performance-analyzer.service. File is missing, hosted in opensearch-build repo. Fixed by copying over the file, and placing it into the correct location.

x86-64

RPM package created successfully. Reviewing the content, these files present problems:

🟢 /etc/wazuh-indexer/jvm.options: not equal to jvm.prod.options. Fixed in 2931b57.
🟢 /etc/wazuh-indexer/opensearch.yml: not equal to opensearch.prod.yml. Fixed in 2931b57.
🟡 /var/lib/wazuh-indexer: new files performance_analyzer_enabled.conf and rca_enabled.conf. Files will be kept. Removing then requires additional work as the SPEC file depends on them.
🟢 /var/run/wazuh-indexer: new empty folder. Folder will be kept. Nothing to do.
🟡 /usr/share/wazuh-indexer/data: new file. Broken symbolic link.
🟡 /usr/share/wazuh-indexer/logs: new file. Broken symbolic link.
🟢 /usr/share/wazuh-indexer/README.md: new file.
🔴 /usr/share/wazuh-indexer/performance-analyzer-rca: empty file, but shouldn't.

File permissions are fine:

find /etc -user wazuh-indexer -not -regex ".*wazuh\-indexer.*" -name "*"
find /usr -user wazuh-indexer -not -regex ".*wazuh\-indexer.*" -name "*"
find /var -user wazuh-indexer -not -regex ".*wazuh\-indexer.*" -name "*"
find /bin -user wazuh-indexer -not -regex ".*wazuh\-indexer.*" -name "*"
find /etc -group wazuh-indexer -not -regex ".*wazuh\-indexer.*" -name "*"
find /usr -group wazuh-indexer -not -regex ".*wazuh\-indexer.*" -name "*"
find /var -group wazuh-indexer -not -regex ".*wazuh\-indexer.*" -name "*"
find /bin -group wazuh-indexer -not -regex ".*wazuh\-indexer.*" -name "*"

rpm_x64.log

Warnings:

warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/jstat
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/javap
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/jshell
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/jmap
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/jfr
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/jinfo
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/jdeprscan
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/jmod
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/jcmd
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/jstatd
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/jdb
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/keytool
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/jimage
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/javac
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/java
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/jar
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/jlink
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/jps
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/jhsdb
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/jconsole
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/serialver
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/jrunscript
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/javadoc
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/jdeps
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/jarsigner
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/jstack
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/jpackage
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/bin/rmiregistry
warning: Missing build-id in /home/wazuh-indexer/artifacts/tmp/rpm/BUILDROOT/wazuh-indexer-4.9.0-1.x86_64/usr/share/wazuh-indexer/jdk/lib/jspawnhelper
warning: File listed twice: /usr/share/wazuh-indexer/bin/indexer-init.sh
warning: File listed twice: /usr/share/wazuh-indexer/bin/indexer-ism-init.sh
warning: File listed twice: /usr/share/wazuh-indexer/bin/indexer-security-init.sh
warning: absolute symlink: /usr/share/wazuh-indexer/data -> /var/lib/wazuh-indexer
warning: absolute symlink: /usr/share/wazuh-indexer/logs -> /var/log/wazuh-indexer

arm64

FAILED: requires an ARM runner

+ echo 'Install plugins'
+ for plugin in "${plugins[@]}"
+ plugin_from_maven=org.opensearch.plugin:performance-analyzer:2.11.0.0
+ OPENSEARCH_PATH_CONF=./etc/wazuh-indexer
+ ./usr/share/wazuh-indexer/bin/opensearch-plugin install --batch --verbose org.opensearch.plugin:performance-analyzer:2.11.0.0
./usr/share/wazuh-indexer/bin/opensearch-env: line 99: /home/wazuh-indexer/artifacts/tmp/arm64-rpm/usr/share/wazuh-indexer/jdk/bin/java: cannot execute binary file: Exec format error

AlexRuiz7 · 2023-12-21T10:18:31Z

On hold due to wazuh/wazuh-packages#2697

I'll be testing the new package for wazuh-indexer.

AlexRuiz7 · 2023-12-26T16:14:55Z

#85 has been merged, but the work continues (see list on top).

AlexRuiz7 · 2024-01-08T18:57:10Z

Issue closed partially. All problems found have been transformed into issues.

The generation of wazuh-indexer packages using GitHub workflows as been completed but for the ARM64 architecture, as we need custom runners to build them.

Assemble ARM packages #111

AlexRuiz7 mentioned this issue Dec 11, 2023

Wazuh indexer fork update #54

Closed

7 tasks

AlexRuiz7 self-assigned this Dec 11, 2023

AlexRuiz7 added level/task Task issue type/enhancement Enhancement issue labels Dec 11, 2023

AlexRuiz7 mentioned this issue Dec 13, 2023

Add tools to assemble packages #78

Closed

7 tasks

AlexRuiz7 mentioned this issue Dec 22, 2023

Add workflow to assemble packages #85

Merged

7 tasks

AlexRuiz7 linked a pull request Dec 22, 2023 that will close this issue

Add workflow to assemble packages #85

Merged

7 tasks

AlexRuiz7 closed this as completed in #85 Dec 26, 2023

AlexRuiz7 reopened this Dec 26, 2023

AlexRuiz7 added level/epic Epic issue and removed level/task Task issue labels Dec 26, 2023

AlexRuiz7 mentioned this issue Dec 26, 2023

Assemble RPM packages #93

Closed

AlexRuiz7 removed a link to a pull request Dec 26, 2023

Add workflow to assemble packages #85

Merged

7 tasks

AlexRuiz7 mentioned this issue Dec 27, 2023

Add tools to assemble DEB packages #96

Merged

7 tasks

This was referenced Jan 4, 2024

Move performance-analyzer-rca to its final location #89

Closed

Validate & test packages - Iter. 1 #99

Closed

AlexRuiz7 closed this as completed Jan 8, 2024

AlexRuiz7 mentioned this issue Jan 24, 2024

Validate & test packages - Iter. 2 #142

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Phase II: Assembly - Automate the generation of final packages #76

Phase II: Assembly - Automate the generation of final packages #76

AlexRuiz7 commented Dec 11, 2023 •

edited

Loading

AlexRuiz7 commented Dec 15, 2023 •

edited

Loading

AlexRuiz7 commented Dec 21, 2023

AlexRuiz7 commented Dec 26, 2023 •

edited

Loading

AlexRuiz7 commented Jan 8, 2024

Phase II: Assembly - Automate the generation of final packages #76

Phase II: Assembly - Automate the generation of final packages #76

Comments

AlexRuiz7 commented Dec 11, 2023 • edited Loading

Description

Tasks

Functional requirements

AlexRuiz7 commented Dec 15, 2023 • edited Loading

RPM

Running in Docker

x86-64

arm64

AlexRuiz7 commented Dec 21, 2023

AlexRuiz7 commented Dec 26, 2023 • edited Loading

AlexRuiz7 commented Jan 8, 2024

AlexRuiz7 commented Dec 11, 2023 •

edited

Loading

AlexRuiz7 commented Dec 15, 2023 •

edited

Loading

AlexRuiz7 commented Dec 26, 2023 •

edited

Loading