CI: Use local cache registry #22726

edsantiago · 2024-05-16T13:16:37Z

As of containers/automation_images#357 our CI VMs include a local registry preloaded with all(*) images used in tests.

(*) where "all" means "most".

This PR sets up CI such that it will use that registry.

Signed-off-by: Ed Santiago santiago@redhat.com

None

packit-as-a-service · 2024-05-16T13:20:49Z

Ephemeral COPR build failed. @containers/packit-build please check.

Luap99 · 2024-05-16T16:20:01Z

test/e2e/pull_test.go

@@ -38,7 +38,8 @@ var _ = Describe("Podman pull", func() {

 		session := podmanTest.Podman([]string{"pull", "quay.io/libpod/ibetthisdoesntexist:there"})
 		session.WaitWithDefaultTimeout()
-		Expect(session).To(ExitWithError(125, "nitializing source docker://quay.io/libpod/ibetthisdoesntexist:there: reading manifest there in quay.io/libpod/ibetthisdoesntexist: unauthorized: access to the requested resource is not authorized"))
+		// FIXME: uncomfortable hardcoding of localhost:56789
+		Expect(session).To(ExitWithError(125, "nitializing source docker://quay.io/libpod/ibetthisdoesntexist:there: reading manifest there in localhost:56789/libpod/ibetthisdoesntexist: manifest unknown"))


heads up, tests should still pass locally and I don't think we want to setup a local registry there right?
Thus I would think we need a regex or a Or() matcher to match both strings anyway

This is thorny. As tests are currently written, e2e tests hard-force the use of test/registries.conf. No matter where they're run (CI, laptop, anywhere). This may need to be reevaluated, but I'm not bothering with any of that until I find out if this approach is viable.

yes sure keep testing, just keep in mind that the end result must still work locally

Luap99 · 2024-05-16T16:20:52Z

contrib/cirrus/runner.sh

    set -x
    # shellcheck disable=SC2154
    exec bin/podman run --rm --privileged --net=host --cgroupns=host \
        -v `mktemp -d -p /var/tmp`:/var/tmp:Z \
        --tmpfs /tmp:mode=1777 \
+        --expose 56789 \


this does not do anything, not sure if you were expecting anything with that or if this serves documentation purposes.

I need the container to be able to talk to host:56789. --port does the opposite, IIRC: host can talk to container. I'll look at logs and see what happens.

The container uses --network=host so it shares the network namespace with the host so from a network POV there should be no functional difference

Duh. Thanks, I missed that.

contrib/cirrus/local-cache-registry

...to minimize hiccups. RUN-2091 in Jira. Network registries are too unreliable; they cause too many flakes in CI. Here we set up a registry running on each VM, prepopulated with all container images used in podman and buildah tests. Related PRs: containers/podman#22726 containers/buildah#5584 Once those merge, podman and buildah CI tests will fetch images from this local registry. Signed-off-by: Ed Santiago <santiago@redhat.com>

As of containers/automation_images#357 our CI VMs include a local registry preloaded with all(*) images used in tests. * where "all" means "most". This commit installs a new registries.conf that redirects docker and quay to the new local registry. The hope is that this will reduce CI flakes. Since tests change over time, and new tests may require new images, this commit also adds a mechanism for pulling in remote images at test run time. Obviously this negates the purpose of the cache, since it introduces a flake pain point. The idea is: DO NOT DO THIS UNLESS ABSOLUTELY NECESSARY, and then, if we have to do this, hurry up and spin new CI VMs that include the new image(s). Signed-off-by: Ed Santiago <santiago@redhat.com>

This commit gets tests working under the new local-registry system: * amend a few image names, mostly just sticking to a consistent list of those images in our registry cache. Mostly minor tag updates. * trickier: pull_test: change some error messages, and remove a test that's now a NOP. Basically, with a local (unprotected) registry we always get "404 manifest unknown"; with a real registry we'll get "403 I can't tell you". * trickiest: seccomp_test: build our own images at run time, with our desired labels. Until now we've been pulling prebuilt images, but those will not copy to the local cache registry. Something about v1? Anyhow, I gave up trying to cache them, and the workaround is straightforward. Also took the liberty of strengthening a few error-message checks Signed-off-by: Ed Santiago <santiago@redhat.com>

New tool, get-local-registry-script, intended for developers to get a local registry running in their environment. This is not necessary for any tests, but may be desirable for performance reasons and/or to recreate the CI environment. Signed-off-by: Ed Santiago <santiago@redhat.com>

edsantiago

Once again I think this is ready for review. My thanks for everyone's patience.

edsantiago · 2024-07-11T12:07:16Z

test/e2e/info_test.go

+			session = podmanTest.Podman([]string{"info", "--format", "{{.Host.RemoteSocket.Exists}}"})
+			session.WaitWithDefaultTimeout()
+			Expect(session).Should(ExitCleanly())
+			Expect(session.OutputToString()).To(Equal("true"))


This is, I think, the only change since last review. The latest Debian VM image creates and recreates /run/user/XXX/podman/podman.sock at (seemingly) random times. I've given up trying to understand what is creating it, and honestly I choose to decide that this test was broken. There is no harm in the socket existing in a podman-local test. The only necessary test is that if podman-remote, then socket-must-exist.

Luap99

LGTM

openshift-ci · 2024-07-11T12:16:00Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: edsantiago, Luap99

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Luap99,edsantiago]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

rhatdan · 2024-07-11T12:37:57Z

/lgtm
I will let @edsantiago cancel the hold

edsantiago · 2024-07-11T12:39:43Z

/hold cancel

(I forgot about that). Thanks.

openshift-ci bot added release-note-none approved Indicates a PR has been approved by an approver from all required OWNERS files. labels May 16, 2024

edsantiago force-pushed the pull-from-local-registry branch 5 times, most recently from 684512b to c82b31a Compare May 16, 2024 16:14

Luap99 reviewed May 16, 2024

View reviewed changes

edsantiago force-pushed the pull-from-local-registry branch 7 times, most recently from 03825bc to 48e506f Compare May 22, 2024 14:40

Luap99 reviewed May 22, 2024

View reviewed changes

contrib/cirrus/local-cache-registry Outdated Show resolved Hide resolved

edsantiago force-pushed the pull-from-local-registry branch 10 times, most recently from 293a60b to 88fa517 Compare May 28, 2024 13:54

edsantiago force-pushed the pull-from-local-registry branch 3 times, most recently from 3bc7dbf to 80ed2a7 Compare May 30, 2024 01:57

edsantiago force-pushed the pull-from-local-registry branch 3 times, most recently from 9c82836 to 74139bd Compare July 8, 2024 21:31

This was referenced Jul 9, 2024

podman stop: rootless netns ref counter out of sync, counter is at -1, resetting it back to 0 #21569

Closed

podman pod rm: Error: removing pod XXX cgroup: <echo>: Unit machine-ETC.slice not loaded #20044

Closed

edsantiago force-pushed the pull-from-local-registry branch 2 times, most recently from d9379fc to 64007cb Compare July 10, 2024 12:51

edsantiago mentioned this pull request Jul 10, 2024

podman start --filter restart-policy=always : container state improper #23246

Closed

edsantiago force-pushed the pull-from-local-registry branch from 64007cb to 64e176b Compare July 11, 2024 01:31

edsantiago added 3 commits July 11, 2024 04:39

edsantiago force-pushed the pull-from-local-registry branch from 64e176b to 07b6994 Compare July 11, 2024 10:39

edsantiago marked this pull request as ready for review July 11, 2024 12:00

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 11, 2024

edsantiago commented Jul 11, 2024

View reviewed changes

Luap99 approved these changes Jul 11, 2024

View reviewed changes

openshift-ci bot assigned rhatdan Jul 11, 2024

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jul 11, 2024

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 11, 2024

openshift-merge-bot bot merged commit 58c8803 into containers:main Jul 11, 2024
89 checks passed

edsantiago deleted the pull-from-local-registry branch July 11, 2024 12:42

This was referenced Jul 11, 2024

Create a local registry containers/automation_images#357

Merged

CI: use local registry containers/buildah#5584

Merged

stale-locking-app bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Oct 10, 2024

stale-locking-app bot locked as resolved and limited conversation to collaborators Oct 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: Use local cache registry #22726

CI: Use local cache registry #22726

edsantiago commented May 16, 2024 •

edited

Loading

packit-as-a-service bot commented May 16, 2024

Luap99 May 16, 2024

edsantiago May 16, 2024

Luap99 May 16, 2024

Luap99 May 16, 2024

edsantiago May 16, 2024

Luap99 May 16, 2024

edsantiago May 16, 2024

edsantiago left a comment

edsantiago Jul 11, 2024

Luap99 left a comment

openshift-ci bot commented Jul 11, 2024

rhatdan commented Jul 11, 2024

edsantiago commented Jul 11, 2024

CI: Use local cache registry #22726

CI: Use local cache registry #22726

Conversation

edsantiago commented May 16, 2024 • edited Loading

packit-as-a-service bot commented May 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edsantiago left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Luap99 left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Jul 11, 2024

rhatdan commented Jul 11, 2024

edsantiago commented Jul 11, 2024

edsantiago commented May 16, 2024 •

edited

Loading