Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Docker pull fails to pull from restricted path if anonymous access allowed on the other path #2928

Open
vooon opened this issue Jan 30, 2025 · 24 comments
Labels
bug Something isn't working rm-external Roadmap item submitted by non-maintainers

Comments

@vooon
Copy link
Contributor

vooon commented Jan 30, 2025

zot version

2.1.3-rc1 (master)

Describe the bug

So, there a some base images, that i want any clients to be able to pull, but rest of the images - only for logged users.
Without login I'm able to pull repo.tld/base/foo:bar, but not repo.tld/service/baz:latest, that's expected.
Once i've done podman login repo.tld and docker login repo.tld, i can podman pull repo.tld/service/baz:latest, but not the docker one!

# docker --debug -l debug pull repo.tld/service/baz:latest
time="2025-01-30T16:00:03Z" level=debug msg="otel error" error="1 errors occurred detecting resource:\n\t* conflicting Schema URL: https://opentelemetry.io/schemas/1.21.0 and https://opentelemetry.io/schemas/1.26.0"
Error response from daemon: unauthorized: authentication required

From the Zot log i see that it does:

  1. GET /v2/, without auth, 200 OK
  2. HEAD /v2/service/baz/manifests/latest, 401
  3. end

If there no anonymous access allowed flow:

  1. GET /v2/, - 401
  2. HEAD /v2/service/baz/manifests/latest, with auth - 200 OK
  3. loading layers ...

To reproduce

  1. Configuration
http:
  address: 0.0.0.0
  port: 443
  tls:
    cert: /etc/zot/tls.crt
    key: /etc/zot/tls.key
  auth:
    accessControl:
      repositories:
        "**": {defaultPolicy: [read]}
        "base/**": {anonymousPolicy: [read]}
  1. Client tool used: docker/27.5.1 go/go1.22.11 git-commit/4c9b3b0

Expected behavior

Docker pulls protected images when there also anonymous access allowed for some other path.

Screenshots

No response

Additional context

No response

@vooon vooon added the bug Something isn't working label Jan 30, 2025
@andaaron
Copy link
Contributor

You need a default policy for "base/**" or else the logged in users won't have access.
Access for anonymous users users doesn't affect logged in users. So if anonymous users have access it doesn't mean logged in users would, it's a separate setting.

@rchincha rchincha added the rm-external Roadmap item submitted by non-maintainers label Jan 30, 2025
@vooon
Copy link
Contributor Author

vooon commented Jan 30, 2025

Oh, okay, will check tomorrow.

@vooon
Copy link
Contributor Author

vooon commented Jan 31, 2025

@andaaron does not work. The same issue as reported:

$ docker logout zot.dev.tld
$ docker pull zot.dev.tld/base/almalinux9-base:latest
latest: Pulling from base/almalinux9-base
493137e25e96: Pull complete 
6dcd0140aae1: Pull complete 
Digest: sha256:07925748790cce7014172cf66dedd49676c0f00791cd7c35605677684b4aeceb
Status: Downloaded newer image for zot.dev.tld/base/almalinux9-base:latest
zot.dev.tld/base/almalinux9-base:latest
$ docker pull zot.dev.tld/dev/almalinux9-base:latest
Error response from daemon: unauthorized: authentication required
zsh: exit 1     docker pull zot.dev.tld/dev/almalinux9-base:latest
$ docker login zot.dev.tld
Username: testuser
Password: 
WARNING! Your password will be stored unencrypted in /home/vovan/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
$ docker pull zot.dev.tld/dev/almalinux9-base:latest
Error response from daemon: unauthorized: authentication required
zsh: exit 1     docker pull zot.dev.tld/dev/almalinux9-base:latest

@vooon
Copy link
Contributor Author

vooon commented Jan 31, 2025

And podman works as expected, even without defaultPolicy on base/**:

$ podman logout zot.dev.tld
$ podman pull zot.dev.tld/base/almalinux9-base:latest
Trying to pull zot.dev.tld/base/almalinux9-base:latest...
Getting image source signatures
Copying blob 6dcd0140aae1 done   | 
Copying blob 493137e25e96 skipped: already exists  
Copying config 36116342e8 done   | 
Writing manifest to image destination
36116342e8204687b6a24161e183bed3932ec3aece147ca46396bbc673d058ea
$ podman pull zot.dev.tld/dev/almalinux9-base:latest
Trying to pull zot.dev.tld/dev/almalinux9-base:latest...
Error: initializing source docker://zot.dev.tld/dev/almalinux9-base:latest: reading manifest latest in zot.dev.tld/dev/almalinux9-base: authentication required
zsh: exit 125   podman pull zot.dev.tld/dev/almalinux9-base:latest
$ podman login zot.dev.tld                        
Username: testuser
Password: 
Login Succeeded!
$ podman pull zot.dev.tld/dev/almalinux9-base:latest
Trying to pull zot.dev.tld/dev/almalinux9-base:latest...
Getting image source signatures
Copying blob 6dcd0140aae1 skipped: already exists  
Copying blob 493137e25e96 skipped: already exists  
Copying config 36116342e8 done   | 
Writing manifest to image destination
36116342e8204687b6a24161e183bed3932ec3aece147ca46396bbc673d058ea
$ podman pull zot.dev.tld/base/almalinux9-base:latest
Trying to pull zot.dev.tld/base/almalinux9-base:latest...
Getting image source signatures
Copying blob 6dcd0140aae1 skipped: already exists  
Copying blob 493137e25e96 skipped: already exists  
Copying config 36116342e8 done   | 
Writing manifest to image destination
36116342e8204687b6a24161e183bed3932ec3aece147ca46396bbc673d058ea

@vooon
Copy link
Contributor Author

vooon commented Jan 31, 2025

@andaaron and, as i reported, if i delete all anonymousPolicy, then docker works:

$ docker pull zot.dev.tld/dev/almalinux9-base:latest
latest: Pulling from dev/almalinux9-base
Digest: sha256:07925748790cce7014172cf66dedd49676c0f00791cd7c35605677684b4aeceb
Status: Downloaded newer image for zot.dev.tld/dev/almalinux9-base:latest
zot.dev.tld/dev/almalinux9-base:latest

@vooon
Copy link
Contributor Author

vooon commented Jan 31, 2025

Well, that bug is a complete show stopper to replace Harbor.

@andaaron
Copy link
Contributor

andaaron commented Jan 31, 2025

If podman works and docker client doesn't, what calls are they making differently? Maybe docker is calling an additional API? Should check zot logs for the calls being made.

@vooon
Copy link
Contributor Author

vooon commented Jan 31, 2025

@andaaron the main difference what happens after GET /v2/.

Podman (when /v2/ - 200) - successful:

{"level":"info","module":"http","component":"session","clientIP":"146.19.213.176:40864","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Docker-Distribution-Api-Version":["registry/2.0"],"User-Agent":["containers/5.33.0 (github.com/containers/image)"]},"goroutine":26901,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-01-31T10:00:22.237349578Z","message":"HTTP API"}
{"level":"info","module":"http","username":"vermakov","component":"session","clientIP":"146.19.213.176:40864","method":"GET","path":"/v2/dev/almalinux9-base/manifests/latest","statusCode":200,"latency":"1s","bodySize":634,"headers":{"Accept":["application/vnd.oci.image.manifest.v1+json","application/vnd.docker.distribution.manifest.v2+json","application/vnd.docker.distribution.manifest.v1+prettyjws","application/vnd.docker.distribution.manifest.v1+json","application/vnd.docker.distribution.manifest.list.v2+json","application/vnd.oci.image.index.v1+json"],"Accept-Encoding":["gzip"],"Authorization":["******"],"Docker-Distribution-Api-Version":["registry/2.0"],"User-Agent":["containers/5.33.0 (github.com/containers/image)"]},"goroutine":26901,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-01-31T10:00:24.104744709Z","message":"HTTP API"}
{"level":"info","module":"http","username":"vermakov","component":"session","clientIP":"146.19.213.176:40864","method":"GET","path":"/v2/dev/almalinux9-base/manifests/sha256:35f7d9f7e78eacdfef848259cd01b0a70bd89a7286769615dd9db9437c9cd3e1","statusCode":200,"latency":"2s","bodySize":1235,"headers":{"Accept":["application/vnd.oci.image.manifest.v1+json","application/vnd.docker.distribution.manifest.v2+json","application/vnd.docker.distribution.manifest.v1+prettyjws","application/vnd.docker.distribution.manifest.v1+json","application/vnd.docker.distribution.manifest.list.v2+json","application/vnd.oci.image.index.v1+json"],"Accept-Encoding":["gzip"],"Authorization":["******"],"Docker-Distribution-Api-Version":["registry/2.0"],"User-Agent":["containers/5.33.0 (github.com/containers/image)"]},"goroutine":26901,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-01-31T10:00:26.844312477Z","message":"HTTP API"}
{"level":"info","module":"http","username":"vermakov","component":"session","clientIP":"146.19.213.176:40864","method":"GET","path":"/v2/dev/almalinux9-base/blobs/sha256:36116342e8204687b6a24161e183bed3932ec3aece147ca46396bbc673d058ea","statusCode":200,"latency":"0s","bodySize":4878,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Docker-Distribution-Api-Version":["registry/2.0"],"User-Agent":["containers/5.33.0 (github.com/containers/image)"]},"goroutine":26901,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-01-31T10:00:27.578403646Z","message":"HTTP API"}

Podman (when /v2/ - 401) - successful

{"level":"info","module":"http","component":"session","clientIP":"146.19.213.176:60716","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept-Encoding":["gzip"],"Docker-Distribution-Api-Version":["registry/2.0"],"User-Agent":["containers/5.33.0 (github.com/containers/image)"]},"goroutine":1383,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-01-31T10:05:55.259281617Z","message":"HTTP API"}
{"level":"info","module":"http","username":"vermakov","component":"session","clientIP":"146.19.213.176:60724","method":"GET","path":"/v2/dev/almalinux9-base/manifests/latest","statusCode":200,"latency":"1s","bodySize":634,"headers":{"Accept":["application/vnd.oci.image.manifest.v1+json","application/vnd.docker.distribution.manifest.v2+json","application/vnd.docker.distribution.manifest.v1+prettyjws","application/vnd.docker.distribution.manifest.v1+json","application/vnd.docker.distribution.manifest.list.v2+json","application/vnd.oci.image.index.v1+json"],"Accept-Encoding":["gzip"],"Authorization":["******"],"Docker-Distribution-Api-Version":["registry/2.0"],"User-Agent":["containers/5.33.0 (github.com/containers/image)"]},"goroutine":1460,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-01-31T10:05:56.60890512Z","message":"HTTP API"}
{"level":"info","module":"http","username":"vermakov","component":"session","clientIP":"146.19.213.176:60724","method":"GET","path":"/v2/dev/almalinux9-base/manifests/sha256:35f7d9f7e78eacdfef848259cd01b0a70bd89a7286769615dd9db9437c9cd3e1","statusCode":200,"latency":"0s","bodySize":1235,"headers":{"Accept":["application/vnd.oci.image.manifest.v1+json","application/vnd.docker.distribution.manifest.v2+json","application/vnd.docker.distribution.manifest.v1+prettyjws","application/vnd.docker.distribution.manifest.v1+json","application/vnd.docker.distribution.manifest.list.v2+json","application/vnd.oci.image.index.v1+json"],"Accept-Encoding":["gzip"],"Authorization":["******"],"Docker-Distribution-Api-Version":["registry/2.0"],"User-Agent":["containers/5.33.0 (github.com/containers/image)"]},"goroutine":1460,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-01-31T10:05:57.004880871Z","message":"HTTP API"}
{"level":"info","module":"http","username":"vermakov","component":"session","clientIP":"146.19.213.176:60724","method":"GET","path":"/v2/dev/almalinux9-base/blobs/sha256:36116342e8204687b6a24161e183bed3932ec3aece147ca46396bbc673d058ea","statusCode":200,"latency":"0s","bodySize":4878,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Docker-Distribution-Api-Version":["registry/2.0"],"User-Agent":["containers/5.33.0 (github.com/containers/image)"]},"goroutine":1460,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-01-31T10:05:57.218356105Z","message":"HTTP API"}

Docker (when /v2/ - 200) - failed:

{"level":"info","module":"http","component":"session","clientIP":"146.19.213.176:35260","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Connection":["close"],"User-Agent":["docker/26.1.3 go/go1.22.2 git-commit/26.1.3-0ubuntu1~24.04.1 kernel/6.8.0-51-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.3 \\(linux\\))"]},"goroutine":26937,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-01-31T10:02:09.763175192Z","message":"HTTP API"}
{"level":"info","module":"http","component":"session","clientIP":"146.19.213.176:35276","method":"HEAD","path":"/v2/dev/almalinux9-base/manifests/latest","statusCode":401,"latency":"0s","bodySize":266,"headers":{"Accept":["application/vnd.docker.distribution.manifest.list.v2+json","application/vnd.oci.image.index.v1+json","application/vnd.oci.image.manifest.v1+json","application/vnd.docker.distribution.manifest.v1+prettyjws","application/json","application/vnd.docker.distribution.manifest.v2+json"],"Connection":["close"],"User-Agent":["docker/26.1.3 go/go1.22.2 git-commit/26.1.3-0ubuntu1~24.04.1 kernel/6.8.0-51-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.3 \\(linux\\))"]},"goroutine":26963,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-01-31T10:02:10.052108389Z","message":"HTTP API"}
{"level":"info","module":"http","component":"session","clientIP":"146.19.213.176:35280","method":"GET","path":"/v2/dev/almalinux9-base/manifests/latest","statusCode":401,"latency":"0s","bodySize":266,"headers":{"Accept":["application/vnd.docker.distribution.manifest.v1+prettyjws","application/json","application/vnd.docker.distribution.manifest.v2+json","application/vnd.docker.distribution.manifest.list.v2+json","application/vnd.oci.image.index.v1+json","application/vnd.oci.image.manifest.v1+json"],"Accept-Encoding":["gzip"],"Connection":["close"],"User-Agent":["docker/26.1.3 go/go1.22.2 git-commit/26.1.3-0ubuntu1~24.04.1 kernel/6.8.0-51-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.3 \\(linux\\))"]},"goroutine":26997,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-01-31T10:02:10.324827765Z","message":"HTTP API"}

Docker (when /v2/ - 401) - successful:

{"level":"info","module":"http","component":"session","clientIP":"146.19.213.176:50270","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept-Encoding":["gzip"],"Connection":["close"],"User-Agent":["docker/26.1.3 go/go1.22.2 git-commit/26.1.3-0ubuntu1~24.04.1 kernel/6.8.0-51-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.3 \\(linux\\))"]},"goroutine":6435,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-01-31T10:06:24.274428411Z","message":"HTTP API"}
{"level":"info","module":"http","username":"vermakov","component":"session","clientIP":"146.19.213.176:50282","method":"HEAD","path":"/v2/dev/almalinux9-base/manifests/latest","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["application/vnd.docker.distribution.manifest.v1+prettyjws","application/json","application/vnd.docker.distribution.manifest.v2+json","application/vnd.docker.distribution.manifest.list.v2+json","application/vnd.oci.image.index.v1+json","application/vnd.oci.image.manifest.v1+json"],"Authorization":["******"],"Connection":["close"],"User-Agent":["docker/26.1.3 go/go1.22.2 git-commit/26.1.3-0ubuntu1~24.04.1 kernel/6.8.0-51-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.3 \\(linux\\))"]},"goroutine":6425,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-01-31T10:06:25.371829485Z","message":"HTTP API"}
``

@vooon
Copy link
Contributor Author

vooon commented Jan 31, 2025

So podman simply always send credentials after call to /v2/, docker - only if this call returns 401.
Not sure what Harbor's distribution sends back, but probably some extra header, like for /v2/_catalog?

@andaaron
Copy link
Contributor

I don't think we should return 401 on /v2 if anonymous access is allowed on other repos, as we don't know what repos the client would try to access next.
Is the docker client not checking the WWW-Authenticate header we are sending back in the response to the call on /v2?

@andaaron
Copy link
Contributor

@eusebiu-constantin-petu-dbk you worked more on this side, where was the expected behavior of /v2 discussed/documented in the community?

@eusebiu-constantin-petu-dbk
Copy link
Collaborator

https://github.com/opencontainers/wg-auth/blob/main/docs/implementations/moby.md

They decide to setup basic auth on further requests based on /v2/ route and not if credentials are set, unlike all other tools.

@andaaron
Copy link
Contributor

andaaron commented Feb 1, 2025

I think we could check and see if the client is docker and add logic to return 401 for /v2 based on that, even if anonymous auth is available. The question is if that would break the behavior for other docker clients who are accessing repos which do not require authentication. It would require every docker client to authenticate regardless of the repo they access.

How do other registries solve this?

@andaaron
Copy link
Contributor

andaaron commented Feb 1, 2025

We can't guess what will be the requested repo based on the initial /v2 call.

@vooon
Copy link
Contributor Author

vooon commented Feb 1, 2025 via email

@andaaron
Copy link
Contributor

andaaron commented Feb 1, 2025

Distribution https://github.com/distribution/distribution/blob/main/docs/content/spec/api.md.tmpl#L298 mentions "Depending on access control setup, the client may still have to authenticate against different resources, even if this check succeeds.", so theoretically the clients should be able to handle authentication on other endpoints if needed.

@andaaron
Copy link
Contributor

andaaron commented Feb 1, 2025

I reproduced locally.

diff --git a/pkg/api/authn.go b/pkg/api/authn.go
index 5f14f6c1..25a2092b 100644
--- a/pkg/api/authn.go
+++ b/pkg/api/authn.go
@@ -356,6 +356,7 @@ func (amw *AuthnMiddleware) tryAuthnHandlers(ctlr *Controller) mux.MiddlewareFun

                        isMgmtRequested := request.RequestURI == constants.FullMgmt
                        allowAnonymous := ctlr.Config.HTTP.AccessControl.AnonymousPolicyExists()
+                       isDockerClient := strings.Contains(request.Header.Get("User-Agent"), "Docker-Client")

                        // build user access control info
                        userAc := reqCtx.NewUserAccessControl()
@@ -405,6 +406,10 @@ func (amw *AuthnMiddleware) tryAuthnHandlers(ctlr *Controller) mux.MiddlewareFun

                                        return
                                }
+                       } else if isDockerClient && request.RequestURI == "/v2/" {
+                               authFail(response, request, ctlr.Config.HTTP.Realm, delay)
+
+                               return
                        } else if allowAnonymous || isMgmtRequested {
                                // try anonymous auth only if basic auth/session was not given
                                next.ServeHTTP(response, request)

This fixes it for authenticated users, but it doesn't work for anonymous ones.

@andaaron
Copy link
Contributor

andaaron commented Feb 1, 2025

I'd guess better just to see what distribution is doing. In my case Harbor
succesfully pulled by both docker and podman on public (with and without
login), and private (with login).

This is the harbor commit fixing the issue: goharbor/harbor@08f9ffa, they had the same problem.
It look like they send a challenge back to the client even for anonymous calls, and the client gets the permissions from that authorization service before following up.

I also found some distribution docs mentioning the same logic as in https://github.com/opencontainers/wg-auth/blob/main/docs/implementations/moby.md.

So I think it's clear how they resolve this issue.
Unfortunately the usage of /v2 for authentication/authorization is not covered by the OCI spec. The spec just mentions: https://github.com/opencontainers/distribution-spec/blob/main/spec.md#determining-support
"This endpoint MAY be used for authentication/authorization purposes, but this is out of the purview of this specification.".

@andaaron
Copy link
Contributor

andaaron commented Feb 6, 2025

@vooon, a workaround I can think of is disabling anonymous access, and sharing a known user/password with everyone who needs anonymous access.
Or using podman, or another client which works.

Given the docker specific behavior for /v2, we will very likely not fix this issue.

@vooon
Copy link
Contributor Author

vooon commented Feb 7, 2025

@andaaron that's hardly possible: a mockbuild do not allow you to setup a user. Otherwise there were no need to open repos...

@andaaron
Copy link
Contributor

andaaron commented Feb 7, 2025

Doesn't podman/docker read the credentials from local files also in case of mockbuild?
In any case of podman it should work also with anonymous.

@vooon
Copy link
Contributor Author

vooon commented Feb 8, 2025

@andaaron as i remember no - it doesn't. There were a lot of headache when RH changes python macroses so they no longer works in a simple rpmbuild. There were a lot of pocking around, but easiest was just open anonymous access to the base image. Anyway it doesn't have anything proprietary, just preinstalled dev tools and few setup scripts.

The problem that I had to use Docker for GitLab CI because of funky network errors when you utilise services in tests.
But once tests passes the next step - run mock. In a docker, so it also must pull the image from the dev registry... :)

@rchincha
Copy link
Contributor

rchincha commented Feb 13, 2025

@vooon docker "boostraps" authN based on what it encounters on "/v2/" endpoint and assumes/reuses that for all other endpoints, which can get tricky when we have mixed mode authN such as "anonymous".

zot handles each endpoint independently, probably what podman also does.

@vooon
Copy link
Contributor Author

vooon commented Feb 22, 2025

@rchincha i understand, but still i have requirement to support docker and having both private and public registries.

Looks like easiest if i just catch /v2/ calls on the LB and return 401 unconditionally.
Or have a separate instances for private and public repos, which became more maintenance burden.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working rm-external Roadmap item submitted by non-maintainers
Projects
None yet
Development

No branches or pull requests

4 participants