Build time Diagrams

1. Maintaining the LLVM Snapshots for Fedora

Konrad Kleine <kkleine@redhat.com>

This document is supposed to give you an idea of what goes into maintaining the LLVM snapshots for Fedora. Don’t feel like you have read this document from top to bottom but instead consider it a reference. There is an Frequently Asked Questions (FAQ) section at the botton that probably contains many answers. Feel free to add your own there.

This is home for the generation of daily

• Fedora LLVM snapshot RPMs

  • See fedora-copr-build workflow

1.1. Getting the latest snapshot

We need a bit of post-configuration after enabling the copr repository for this project.

$ dnf -y install jq envsubst
$ dnf -y install --skip-broken 'dnf-command(copr)' 'dnf5-command(copr)'
$ dnf -y copr enable @fedora-llvm-team/llvm-snapshots
$ repo_file=$(dnf repoinfo --json *llvm-snapshots* | jq -r ".[0].repo_file_path")
$ distname=$(rpm --eval "%{?fedora:fedora}%{?rhel:rhel}") envsubst '$distname' < $repo_file > /tmp/new_repo_file
$ cat /tmp/new_repo_file > $repo_file

Then install clang or some of the other packages.

2. Tips on contributing

This project uses pre-commit to validate that certain files (e.g. fmf or python) are not broken. Please install pre-commit using pip install pre-commit as described here. Then navigate to your clone of this project and install the git hook scripts using cd ~/llvm-snapshots && pre-commit install. This will run pre-commit on every git commit.

3. Process overview

We build LLVM snapshots on a daily basis on Copr and create a github issue in this very repository for every day to keep track of the build and testing status. The maintainer changes on a monthly basis and will be assigned to the issue.

For a while now we’ve switched from standalone builds to the building everything from a single dist git repo with the working title "big-merge" on the Fedora Package Source. Before we used to have one spec file repository for all the llvm subprojects (clang, lld, lldb, etc.).

Our snapshot builds are done under our group account on Fedora Copr. There build for all supported Fedora version and some RHEL versions.

We carefully create a new copr project for each day. These projects are called incubator projects. Only if all packages for all operating systems and architectures in an incubator project were successfully built without errors, we will promote it to be the next "official" snapshot here.

For each chroot (except i386) we will also run testing-farm tests and report them in the daily issue. Some of these tests run on Red Hat internal hardware and are only reachable within the VPN. We try our best to utilize github labels to reflect the current state of builds and tests for each chroot.

Note	For now we "promote" a daily snapshot to the next "official" snapshot without having run testing-farm tests. This might change in the future.

That is the reason why sometimes it can take days until a new version of LLVM will be published here. If you’re interested in the version for a particular day, feel free to open https://copr.fedorainfracloud.org/coprs/g/fedora-llvm-team/llvm-snapshots-big-merge-YYYYMMDD/ (replace YYYYMMDD with the date you desire). Notice, that we cannot keep the invdividual incubator projects around forever.

4. Find out about the health of snapshots

Go to the Fedora Copr project and look for anything red in the monitor for today (see How do I find the monitor?).

5. How to fix an error in a package build for a specific OS/arch combination?

Suppose you want to work on the llvm package with nothing checkout locally, yet.

1. Export your Fedora Account System username so you can copy and paste the commands below:

   $ export FAS_USER=<YOUR_USERMAME>

   Note	This is optional and only for copy pasting the below commands. Use the username to login in here: https://id.fedoraproject.org/login.

2. Clone the original source code’s big-merge branch and navigate to that directory.

   $ fedpkg clone llvm -b big-merge ~/dev/llvm-rpms/llvm
   
   $ cd ~/dev/llvm-rpms/llvm

3. Go to https://src.fedoraproject.org/rpms/llvm and click the "fork" button in the top-right corner to create fork just for you. Then add it as a remote:

   $ git remote add $FAS_USER \
       ssh://$FAS_USER@pkgs.fedoraproject.org/forks/$FAS_USER/rpms/llvm.git

   You should have two git remote now: origin and one that is named after your FAS login.

4. Make your changes to llvm and commit them locally.

   $ git add -p
   $ git commit -m "My changes"

5. Push your local changes to a branch of your liking (e.g. mybranch) in your fork

   $ git push $FAS_USER HEAD:mybranch

6. Create a pull-requst for your changes so they can go into the big-merge branch by executing this and heading over to your browser:

   $ xdg-open https://src.fedoraproject.org/fork/$FAS_USER/rpms/compiler-rt/diff/big-merge..mybranch

7. Wait for your changes to be approved and merged. Maybe ping someone from the team.

   Then your changes will automatically be taken into account for the next daily build.

8. If you want to kick-off a build yourself, you can run:

   $ copr build-package \
       --name llvm \
       -r fedora-rawhide-x86_64 \
       --timeout $((30*3600)) \
       @fedora-llvm-team/llvm-snapshots-big-merge-`date +%Y%m%d`

   This will build the llvm package with your changes as soon as they landed in the big-merge branch in today’s Copr project.

5.1. Syncing a package with rawhide

Unlike with many other projects, we actually do want to keep the complete git history of downstream patches and changes being made to a .spec file.

Important

That is why we almost always prefer git merge --no-ff --log --summary over git rebase.

6. Frequently Asked Questions

6.1. How can I clone llvm?

I cloned llvm with the appropriate tool (e.g. fedpkg, centpkg and rhpkg).

$ fedpkg clone llvm -b rawhide ~/dev/llvm-rpms/llvm-rawhide #(1)
$ fedpkg clone llvm -b big-merge ~/dev/llvm-rpms/llvm-big-merge #(2)
$ centpkg clone llvm -b c9s ~/dev/llvm-rpms/llvm-centos #(3)
$ rhpkg clone llvm -b rhel-9-main ~/dev/llvm-rpms/clang-rhel #(4)

1. This is for the regular fedora work on rawhide.

2. This is for the work on the LLVM snapshots.

3. This is for the work on CentOS stream.

4. This is for the internal work on RHEL9.

6.2. How to sync with rawhide?

It is not entirely clear yet but we’re aiming towards doing the snapshot work in the rawhide branch of fedora.

These are the benefits that we see: * No special handling for contributors. They can just use rawhide. * When making changes to the spec files for snapshots we can be sure that we’re not breaking rawhide because it is tested by Fedora CI. * No complicated git merging with big-merge or upstream-snapshot branches that we’ve had in the past. * Reverts are possible * Smoother release process? * Just increment the Version: tag used in the non-snapshot part and consume the previously tested changes from the snapshots.

Of course there are some downsides as well: * Possibly a lot of commit activity can happen in the rawhide branch that doesn’t change anything related to rawhide but only snapshots. * A bit more if/else branches in the spec files. * Duplicated patch files with slightly different content as they apply to each supported LLVM version.

We would need %if %{maj_ver} >= 20 conditions or alike for different LLVM versions (see this PR for example).

Once we’ve found a good way to do the patch handling and versioning inside the rawhide spec file repository of llvm, we’re going to update this very FAQ.

6.3. How do I find the monitor?

You can find the snapshot monitor for LLVM Fedora builds on Copr here:

https://copr.fedorainfracloud.org/coprs/g/fedora-llvm-team/llvm-snapshots/monitor/

The above link brings you to the latest "forked" build. It will only contain successful builds.

To find out where this build came from, take a look at the title of the project. There it should say something like:

( forked from @fedora-llvm-team/llvm-snapshots-big-merge-20230221 )

Go to the project from which @fedora-llvm-team/llvm-snapshots was forked to find failing builds.

6.4. How do I run a local snapshot build?

$ git clone https://src.fedoraproject.org/rpms/llvm.git
$ cd llvm
$ make snapshot-rpm

You might need to install missing dependencies. The build process itself probably takes quite some time.

You’re going to find results/YYYYMMDD/snapshot-rpm.log with logging everything from this makefile target.