Cannot download docker image from Gitlab Container Registry

[position-imaging-josh]

Before you report an issue...

• Have you searched for a duplicate report?
• Have you replicated the problem on the latest release?
• Have you checked the latest documentation?
• Are you using an up-to-date system?

Only issues that can be replicated on the latest release, or development branch, of SingularityCE will be investigated and fixed. The open source project does not maintain long-term stable branches or fix bugs in prior versions. If you require LTS support then please see the Sylabs website.

Version of Singularity

4.1.1 and 4.2.2

Describe the bug

After authenticating with gitlab using the following command:

singularity registry login --username $CI_REGISTRY_USER --password $CI_REGISTRY_PASSWORD docker://$CI_REGISTRY

Singularity cannot download the image, reporting the following error:

singularity --debug build --force --authfile ./docker-config.json image.sif newimage.def
DEBUG   [U=0,P=438]        persistentPreRun()            Singularity version: 4.2.2
DEBUG   [U=0,P=438]        persistentPreRun()            Parsing configuration file /usr/local/etc/singularity/singularity.conf
DEBUG   [U=0,P=438]        maybeReExec()                 Checking whether to re-exec
DEBUG   [U=0,P=438]        handleConfDir()               /root/.singularity already exists. Not creating.
DEBUG   [U=0,P=438]        handleRemoteConf()            Ensuring file permission of 0600 on /root/.singularity/remote.yaml
DEBUG   [U=0,P=438]        Init()                        Image format detection
DEBUG   [U=0,P=438]        Init()                        Check for sandbox image format
DEBUG   [U=0,P=438]        Init()                        sandbox format initializer returned: not a directory image
DEBUG   [U=0,P=438]        Init()                        Check for sif image format
DEBUG   [U=0,P=438]        Init()                        sif format initializer returned: SIF magic not found
DEBUG   [U=0,P=438]        Init()                        Check for ocisif image format
DEBUG   [U=0,P=438]        Init()                        ocisif format initializer returned: SIF magic not found
DEBUG   [U=0,P=438]        Init()                        Check for squashfs image format
DEBUG   [U=0,P=438]        Init()                        squashfs format initializer returned: while parsing squashfs super block: not a valid squashfs image
DEBUG   [U=0,P=438]        Init()                        Check for ext3 image format
DEBUG   [U=0,P=438]        Init()                        ext3 format initializer returned: file is not a valid ext3 image
DEBUG   [U=0,P=438]        getCacheParentDir()           environment variable SINGULARITY_CACHEDIR not set, using default image cache
DEBUG   [U=0,P=438]        initCacheDir()                Creating cache directory: /root/.singularity/cache
DEBUG   [U=0,P=438]        initCacheDir()                Creating cache directory: /root/.singularity/cache/library
DEBUG   [U=0,P=438]        initCacheDir()                Creating cache directory: /root/.singularity/cache/oci-tmp
DEBUG   [U=0,P=438]        initCacheDir()                Creating cache directory: /root/.singularity/cache/shub
DEBUG   [U=0,P=438]        initCacheDir()                Creating cache directory: /root/.singularity/cache/oras
DEBUG   [U=0,P=438]        initCacheDir()                Creating cache directory: /root/.singularity/cache/net
DEBUG   [U=0,P=438]        initCacheDir()                Creating cache directory: /root/.singularity/cache/oci-sif
DEBUG   [U=0,P=438]        initCacheDir()                Creating cache directory: /root/.singularity/cache/blob
DEBUG   [U=0,P=438]        initLayout()                  Creating cache OCI layout: /root/.singularity/cache/blob
DEBUG   [U=0,P=438]        Init()                        Image format detection
DEBUG   [U=0,P=438]        Init()                        Check for sandbox image format
DEBUG   [U=0,P=438]        Init()                        sandbox format initializer returned: not a directory image
DEBUG   [U=0,P=438]        Init()                        Check for sif image format
DEBUG   [U=0,P=438]        Init()                        sif format initializer returned: SIF magic not found
DEBUG   [U=0,P=438]        Init()                        Check for ocisif image format
DEBUG   [U=0,P=438]        Init()                        ocisif format initializer returned: SIF magic not found
DEBUG   [U=0,P=438]        Init()                        Check for squashfs image format
DEBUG   [U=0,P=438]        Init()                        squashfs format initializer returned: while parsing squashfs super block: not a valid squashfs image
DEBUG   [U=0,P=438]        Init()                        Check for ext3 image format
DEBUG   [U=0,P=438]        Init()                        ext3 format initializer returned: file is not a valid ext3 image
DEBUG   [U=0,P=438]        newBundle()                   Created temporary directory "/tmp/bundle-temp-17565198" for the bundle
DEBUG   [U=0,P=438]        newBundle()                   Created directory "/tmp/build-temp-824336296/rootfs" for the bundle
INFO    [U=0,P=438]        Full()                        Starting build...
INFO    [U=0,P=438]        Get()                         Fetching OCI image...
DEBUG   [U=0,P=438]        cleanUp()                     Cleaning up "/tmp/build-temp-824336296/rootfs" and "/tmp/bundle-temp-17565198"
FATAL   [U=0,P=438]        runBuildLocal()               While performing build: conveyor failed to get: GET https://gitlab.com/jwt/auth?scope=repository%3Api-spr%2Fkiwi%2Fncs%2Fbuildenvironmentncs%2Fenginerelease3-arm64%3Apull&service=container_registry: DENIED: access forbidden

To Reproduce

1. Setup Docker image on Gitlab
2. Setup singularity to use it

Expected behavior

It would be able to pull docker images from Gitlab

OS / Linux Distribution

Its running in a virtualized AMR64 environment in Docker on a privileged Ubuntu 24.04 image, on a privileged LXC container running Ubuntu 24.04, on Proxmox 8.3.3 which is Debian

Installation Method

Originally I tried it from the Ubuntu 24.04 repository but it had 4.1.1. I compiled the lated 4.2.2 just to check but saw the same issue.

Additional context

I tried to make sure the image should be available from gitlab using their access settings. I am opening a ticket with them as well to see if it something on their end.

Thank you for your time.

[dtrudg]

You are using singularity registry login without an --authfile arg....

singularity registry login --username $CI_REGISTRY_USER --password $CI_REGISTRY_PASSWORD docker://$CI_REGISTRY

But then pointing to an --authfile for the build?

singularity --debug build --force --authfile ./docker-config.json image.sif newimage.def

The first would store the credentials in the default location, but the second is going to try and use credentials at the provided location. Can you resolve this inconsistency - maybe it's just a transcription mistake in the bug report?

Also... if you use docker per the GitLab instructions, everything works okay? I.E.

echo "$CI_REGISTRY_PASSWORD" | docker login $CI_REGISTRY -u $CI_REGISTRY_USER --password-stdin

docker pull <image used by singularity build>

Finally - in the unusual case that the $CI_REGISTRY_PASSWORD contains a special character affecting the interpretation of the rest of the command, you can try piping to STDIN like the GitLab instructions for Docker do...

echo "$CI_REGISTRY_PASSWORD" | singularity registry login -u $CI_REGISTRY_USER --password-stdin docker://$CI_REGISTRY

[position-imaging-josh]

My apologies, I run the singularity command both with and without the --authfile` arg and I seemed to grabbed the one without it. I did that because I was getting other errors and it did not seem to be using the default location. Using the authfile seemed to help so I have been using that mostly and the rest is artifacts of debugging.

...
singularity registry login --username $CI_REGISTRY_USER --password $CI_REGISTRY_PASSWORD docker://$CI_REGISTRY
singularity registry login --authfile ./docker-config.json --username $CI_REGISTRY_USER --password $CI_REGISTRY_PASSWORD docker://$CI_REGISTRY
...

I dont think special characters are causing an issue since it seems to be authenticating just fine:

Logging into 'registry.gitlab.com' with 'gitlab-ci-token'
INFO:    Token stored in /root/.singularity/docker-config.json
INFO:    Token stored in ./docker-config.json
URI                           SECURE?
docker://registry.gitlab.com  ✓

I will try the STDIN method however.

I have been able to use this method to authenticate with Singularity and docker in the past, I believe I'm using it for some automation right now but would need to look into that. I dont have the docker client installed on this setup but I will see if I can test that.

The only other thing I can think of is that this is this is running on arm64. I was having trouble with Gitlab earlier and they requested the headers being sent and received, is there any way to get that information from singularity?

[position-imaging-josh]

Just tested docker, it is able to pull the image inside the same job just fine while singularity gets "DENIED: access forbidden"

[dtrudg]

Okay - given we don't have access to free CI minutes from GitLab, and would have to setup a self-hosted runner to look into this further it isn't likely to be investigated immediately.

You can try to export GODEBUG=http2debug=2 before running the singularity command to trace HTTP2 requests - but there isn't really a convenient tracing method for HTTP1.

[position-imaging-josh]

The GODEBUG var helped, I was able to get some of the traffic and now im looking at what is going on. Just to note, if I run the same test without LXC, on a docker platform of amd64 it seems to work OK but arm64 fails.

[position-imaging-josh]

I was able to confirm with the Gitlab support team that it seems the request is making it to their side without the proper authentication information. Im not sure why there would be a difference between how it works on amd64 and arm64.