make run-on-kind Doesn't Load a locally built bpfman image into the KIND Cluster

[frobware]

When using make run-on-kind to deploy a cluster, the image referenced by ${BPFMAN_IMG} is not loaded into the KIND cluster. As a result, Kubernetes resolves and always pulls the bpfman image from quay.io, even if a locally built version of the bpfman image with the same tag exists. This behaviour occurs because the image pull policy is set to IfNotPresent, which should prevent pulling the image if it already exists locally, but since the image is tagged with :latest, Kubernetes defaults to pulling it from the remote registry.

This pull behaviour is called out in https://kind.sigs.k8s.io/docs/user/quick-start/#loading-an-image-into-your-cluster.

Steps to Reproduce

1. Build a bpfman image locally:

   Make a modification to bpfman; for example:

   $ git diff bpfman-api/src/bin/rpc/main.rs
   diff --git a/bpfman-api/src/bin/rpc/main.rs b/bpfman-api/src/bin/rpc/main.rs
   index 4cc6c9c2..73fc1be4 100644
   --- a/bpfman-api/src/bin/rpc/main.rs
   +++ b/bpfman-api/src/bin/rpc/main.rs
   @@ -85,6 +85,7 @@ fn initialize_rpc(csi_support: bool) -> anyhow::Result<()> {
   
    #[tokio::main]
    async fn main() -> anyhow::Result<()> {
   +    println!("Hello world");
        let args = Rpc::parse();
        initialize_rpc(args.csi_support)?;
        //TODO https://github.com/bpfman/bpfman/issues/881

   Build your changes:

   $ podman build -f Containerfile.bpfman.local -t quay.io/bpfman/bpfman:latest

2. Verify the locally built image exists:

   $ podman images | grep bpfman
   quay.io/bpfman/bpfman           latest      b0655bd63d58  6 seconds ago   1.81 GB
   quay.io/bpfman/bpfman-agent     latest      a54235a7988d  28 minutes ago  492 MB
   quay.io/bpfman/bpfman-operator  latest      ca65b12695e5  29 minutes ago  178 MB

3. Verify your change shows up:

   Which it doesn't.

   % oc logs -n bpfman bpfman-daemon-gfj9p
   Defaulted container "bpfman" out of: bpfman, bpfman-agent, node-driver-registrar, mount-bpffs (init)
   [INFO  bpfman_rpc::serve] Using no inactivity timer
   [INFO  bpfman_rpc::serve] Using default Unix socket
   [INFO  bpfman_rpc::serve] Listening on /run/bpfman-sock/bpfman.sock
   [INFO  bpfman_rpc::storage] CSI Plugin Listening on /run/bpfman/csi/csi.sock

4. Verify the remote SHA using skopeo without pulling the image:

   $ skopeo inspect docker://quay.io/bpfman/bpfman:latest --format '{{.Digest}}'
   sha256:ea8fc9e10bf9b83ce6e289848010bd1672a9bc4b4d41af1599ec570434a99661

5. Run make run-on-kind to deploy the cluster:

   $ make run-on-kind

6. Check the running pods in the bpfman namespace:

   $ oc get pods -n bpfman
   NAME                              READY   STATUS    RESTARTS   AGE
   bpfman-daemon-59qxl               3/3     Running   0          17s
   bpfman-operator-f967f7657-qkwxn   2/2     Running   0          21s

7. Enter the kind cluster and inspect the digest for the bpfman image:

   $ docker exec -it bpfman-deployment-control-plane /bin/bash
   
   $ crictl images | grep bpfman
   quay.io/bpfman/bpfman-agent                     latest               a54235a7988d8       492MB
   quay.io/bpfman/bpfman-operator                  latest               ca65b12695e5f       178MB
   quay.io/bpfman/bpfman                           latest               0b59611dac0a2       56.1MB
   quay.io/bpfman/csi-node-driver-registrar        v2.9.0               50013f94a28d1       10.6MB

   $ crictl inspecti quay.io/bpfman/bpfman:latest
   {
     "status": {
     "id": "sha256:0b59611dac0a2d29b3a8fa01d7526d514bb47ce073480a7f38f539b117e09f92",
     "repoTags": [
       "quay.io/bpfman/bpfman:latest"
     ],
     "repoDigests": [
       "quay.io/bpfman/bpfman@sha256:ea8fc9e10bf9b83ce6e289848010bd1672a9bc4b4d41af1599ec570434a99661"
     ],
     "size": "56137435",
     ...

   Note: The digest ea8fc9e10bf9b83ce6e289848010bd1672a9bc4b4d41af1599ec570434a99661 in the cluster matches the output from skopeo; the image was pulled from quay.io. And the repoDigests also references quay.io.

   If we contrast with the ${BPFMAN_OPERATOR_IMG} image, which is explicitly loaded into the cluster, we see that it was imported:

   root@bpfman-deployment-control-plane:/# crictl inspecti quay.io/bpfman/bpfman-operator:latest |more
   {
     "status": {
       "id": "sha256:ca65b12695e5f69db69f7cb8ecf3b070511e34afe025a271b21155a9c3ad5280",
       "repoTags": [
         "quay.io/bpfman/bpfman-operator:latest"
       ],
       "repoDigests": [
         "docker.io/library/import-2024-08-16@sha256:62561a427f71c223116dfb49e96bf748a395fc3e10ebc8fdca76b7c26805306c"
       ],
    "size": "178254706",   

Conclusion

Although a local bpfman image was built and tagged correctly, when make run-on-kind is executed, the KIND cluster pulls the bpfman image from quay.io. This happens because the image reference in the Makefile ${BPFMAN_IMG} (which defaults to quay.io/bpfman/bpfman:latest) was not explicitly loaded into the cluster; we only explicitly load the operator and agent images. Due to the use of the :latest tag, Kubernetes defaults to pulling the image from the remote registry, quay.io, bypassing the locally built bpfman image with the same tag.

Workaround

To use a locally built bpfman image, avoid tagging the image as :latest, or explicitly set imagePullPolicy: Never in the daemonset deployment.

Fix

Update the call to ./hack/kind-load-image.sh in the Makefile target load-images-kind to always load whatever is referenced in ${BPFMAN_IMG}. This will ensure that the BPFMAN_IMG is explicitly loaded into the KIND cluster, preventing Kubernetes from pulling the image from quay.io and ensuring the locally built bpfman image is used instead.

If we load the bpfman image ala:

.PHONY: load-images-kind
load-images-kind: ## Load bpfman, bpfman-agent, and bpfman-operator images into the running local kind devel cluster.
	./hack/kind-load-image.sh ${KIND_CLUSTER_NAME} ${BPFMAN_OPERATOR_IMG} ${BPFMAN_AGENT_IMG} ${BPFMAN_IMG}

$ make run-on-kind

$ oc logs -n bpfman bpfman-daemon-znxzm
Defaulted container "bpfman" out of: bpfman, bpfman-agent, node-driver-registrar, mount-bpffs (init)
Hello world
[INFO  bpfman_rpc::serve] Using no inactivity timer
[INFO  bpfman_rpc::serve] Using default Unix socket
[INFO  bpfman_rpc::serve] Listening on /run/bpfman-sock/bpfman.sock
[INFO  bpfman_rpc::storage] CSI Plugin Listening on /run/bpfman/csi/csi.sock

To avoid any surprises in local development I would recommend local image references only:

$ cat .envrc
export BPFMAN_AGENT_IMG=localhost/bpfman-operator:latest
export BPFMAN_IMG=localhost/bpfman:latest
export BPFMAN_OPERATOR_IMG=localhost/bpfman-operator:latest

$ podman build -f Containerfile.bpfman.local -t bpfman:latest

$ make run-on-kind
...
./hack/kind-load-image.sh bpfman-deployment localhost/bpfman-operator:latest localhost/bpfman-operator:latest localhost/bpfman:latest
...