Ceph docker builds are broken and ceph should be bumped

[butonic]

The ceph docker build is currently broken:

#7 74.54 Failed:
#7 74.54   container-selinux-2:2.229.0-2.module_el8+847+7863d4e6.noarch                  Error: Transaction failed
#7 74.61 
#7 74.61 
#7 ERROR: process "/bin/sh -c dnf update --exclude=ceph-iscsi -y && dnf install -y   git   gcc   make   libcephfs-devel   librbd-devel   librados-devel" did not complete successfully: exit code: 1
------
 > [2/9] RUN dnf update --exclude=ceph-iscsi -y && dnf install -y   git   gcc   make   libcephfs-devel   librbd-devel   librados-devel:
52.85 Transaction test succeeded.
52.85 Running transaction
 
61.93   Running scriptlet: chrony-4.5-1.el8.x86_64                             40/157 
61.94 System has not been booted with systemd as init system (PID 1). Can't operate.
61.94 Failed to connect to bus: Host is down
61.94 
Error: Transaction failed
74.61 
74.61 
------
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
Dockerfile.revad-ceph:21
--------------------
  20 |     
  21 | >>> RUN dnf update --exclude=ceph-iscsi -y && dnf install -y \
  22 | >>>   git \
  23 | >>>   gcc \
  24 | >>>   make \
  25 | >>>   libcephfs-devel \
  26 | >>>   librbd-devel \
  27 | >>>   librados-devel
  28 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c dnf update --exclude=ceph-iscsi -y && dnf install -y   git   gcc   make   libcephfs-devel   librbd-devel   librados-devel" did not complete successfully: exit code: 1
Error: buildx failed with: ERROR: failed to solve: process "/bin/sh -c dnf update --exclude=ceph-iscsi -y && dnf install -y   git   gcc   make   libcephfs-devel   librbd-devel   librados-devel" did not complete successfully: exit code: 1

I assume because upstream released a new v16 this morning: https://quay.io/repository/ceph/ceph?tab=tags&tag=latest

We should try to bump ceph to version v18 and fix the build. The culprit seems to be:

61.93   Running scriptlet: chrony-4.5-1.el8.x86_64                             40/157 
61.94 System has not been booted with systemd as init system (PID 1). Can't operate.
61.94 Failed to connect to bus: Host is down

[butonic]

@glpatcern I think you might run into the same issue in master?

[glpatcern]

Thanks for the heads up @butonic, indeed we build against Ceph quincy aka v17 for a few months as we had to run on EL8 and previous releases had dependency issues. I understand I should even consider reef aka v18 then...

[butonic]

AFAICT reva master docker builds against v16:

reva/docker/Dockerfile.revad-ceph

Line 19 in 1b1fe7d

FROM quay.io/ceph/ceph:v16

or are there any other docker pipelines?

[glpatcern]

Ah right, the docker image needs to be updated, will do. We have another pipeline without docker, just to compile a ceph-enabled revad for EL8. Thanks!

[butonic]

chrony cannot be started ... and we don't really need to update it

# skip chrony as it cannot be started the container: 'System has not been booted with systemd as init system'
RUN dnf update --exclude=ceph-iscsi,chrony -y && dnf install -y \

hm It seems some repo urls are broken:

--> Running in bf0ee1a2d19e
CentOS Stream 8 - AppStream                     5.1 MB/s |  27 MB     00:05    
CentOS Stream 8 - BaseOS                        8.7 MB/s |  10 MB     00:01    
CentOS Stream 8 - Extras                         35 kB/s |  18 kB     00:00    
CentOS Stream 8 - Extras common packages         31 kB/s | 6.9 kB     00:00    
Copr repo for python-scikit-learn owned by tcha  31 kB/s |  20 kB     00:00    
Copr repo for python3-asyncssh owned by tchaiko  15 kB/s | 3.5 kB     00:00    
Ceph packages for x86_64                        188  B/s | 162  B     00:00    
Errors during downloading metadata for repository 'Ceph':
  - Status code: 404 for http://download.ceph.com/rpm-reef/el8/x86_64/repodata/repomd.xml (IP: 158.69.68.124)
Error: Failed to download metadata for repo 'Ceph': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

this lets me continue

# replace repo url with one that allows downloading the repo metadata
# if http://download.ceph.com/rpm-reef/el8/x86_64/repodata/repomd.xml works again this can be dropped
RUN sed -i 's/download.ceph.com/de.ceph.com/' /etc/yum.repos.d/ceph.repo

now selinux config is not there ...

Running transaction
  Preparing        :                                                        1/1 
  Running scriptlet: nfs-ganesha-selinux-5.7-2.el8s.noarch                  1/1 
  Running scriptlet: nfs-ganesha-selinux-5.7-2.el8s.noarch                 1/12 
/var/tmp/rpm-tmp.zNBaGe: line 2: /etc/selinux/config: No such file or directory
error: %prein(nfs-ganesha-selinux-5.7-2.el8s.noarch) scriptlet failed, exit status 1

Error in PREIN scriptlet in rpm package nfs-ganesha-selinux
  Running scriptlet: nfs-ganesha-5.7-2.el8s.x86_64                         2/12 

lets fake it:

RUN mkdir -p /etc/selinux/config

see #4557

[butonic]

fixed with the above steps ... until a repo stops working again ...