Getting started docs - ErrImagePull on whalesay

[dazinator]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

Never used argo before, I attempted to follow the Getting started page. It seems to fail pulling the image. I did not expect this.

After successfully installing argo server / controller in a k8s namespace, and the cli, I submitted the test workflow as instructed here

argo submit -n argo --watch https://raw.githubusercontent.com/argoproj/argo-workflows/main/examples/hello-world.yaml

After some time I see the message transition from PodInitializing to ErrImagePull

STEP                  TEMPLATE  PODNAME            DURATION  MESSAGE
 ◷ hello-world-jwjlf  whalesay  hello-world-jwjlf  1m        PodInitializing
Name:                hello-world-jwjlf
Namespace:           argo
ServiceAccount:      unset (will run with the default ServiceAccount)
Status:              Running
Conditions:
 PodRunning          False
Created:             Tue Jul 23 16:47:37 +0100 (1 minute ago)
Started:             Tue Jul 23 16:47:37 +0100 (1 minute ago)
Duration:            1 minute 8 seconds
Progress:            0/1

STEP                  TEMPLATE  PODNAME            DURATION  MESSAGE
 ◷ hello-world-jwjlf  whalesay  hello-world-jwjlf  1m        ErrImagePull: [DEPRECATION NOTICE] Docker Image Format v1 and Docker Image manifest version 2, schema 1 support is disabled by default and will be removed in an upcoming release. Suggest the author of docker.io/docker/whalesay:latest to upgrade the image to the OCI Format or Docker Image manifest v2, schema 2. More information at https://docs.docker.com/go/deprecated-image-specs/
Name:                hello-world-jwjlf
Namespace:           argo
ServiceAccount:      unset (will run with the default ServiceAccount)
Status:              Running
Conditions:
 PodRunning          False
Created:             Tue Jul 23 16:47:37 +0100 (1 minute ago)
Started:             Tue Jul 23 16:47:37 +0100 (1 minute ago)
Duration:            1 minute 9 seconds
Progress:            0/1

then eventually to

STEP                  TEMPLATE  PODNAME            DURATION  MESSAGE
 ◷ hello-world-jwjlf  whalesay  hello-world-jwjlf  1m        ImagePullBackOff: Back-off pulling image "docker/whalesay:latest"
Name:                hello-world-jwjlf
Namespace:           argo
ServiceAccount:      unset (will run with the default ServiceAccount)
Status:              Running
Conditions:
 PodRunning          False
Created:             Tue Jul 23 16:47:37 +0100 (1 minute ago)
Started:             Tue Jul 23 16:47:37 +0100 (1 minute ago)
Duration:            1 minute 20 seconds
Progress:            0/1

this seems to cycle to ErrImagePull again.

After 4 mins the workflow still hasn't executed seemingly due to some issue with pulling the image.

STEP                  TEMPLATE  PODNAME            DURATION  MESSAGE
 ◷ hello-world-jwjlf  whalesay  hello-world-jwjlf  3m        ImagePullBackOff: Back-off pulling image "docker/whalesay:latest"
Name:                hello-world-jwjlf
Namespace:           argo
ServiceAccount:      unset (will run with the default ServiceAccount)
Status:              Running
Conditions:
 PodRunning          False
Created:             Tue Jul 23 16:47:37 +0100 (4 minutes ago)
Started:             Tue Jul 23 16:47:37 +0100 (4 minutes ago)
Duration:            4 minutes 0 seconds
Progress:            0/1

Version(s)

v3.5.8

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

As per the docs its this one here:

argo submit -n argo --watch https://raw.githubusercontent.com/argoproj/argo-workflows/main/examples/hello-world.yaml

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

Logs from in your workflow's wait container

> time="2024-07-23T15:47:37.202Z" level=info msg="Processing workflow" Phase= ResourceVersion=1179 namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:47:37.205Z" level=info msg="resolved artifact repository" artifactRepositoryRef="argo/#"
time="2024-07-23T15:47:37.205Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=hello-world-jwjlf
time="2024-07-23T15:47:37.205Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:47:37.205Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:47:37.205Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:47:37.205Z" level=info msg="Pod node hello-world-jwjlf initialized Pending" namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:47:37.212Z" level=info msg="Created pod: hello-world-jwjlf (hello-world-jwjlf)" namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:47:37.212Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:47:37.212Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:47:37.212Z" level=info msg="Workflow to be dehydrated" Workflow Size=1113
time="2024-07-23T15:47:37.220Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=1183 workflow=hello-world-jwjlf
time="2024-07-23T15:47:47.203Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1183 namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:47:47.203Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=hello-world-jwjlf
time="2024-07-23T15:47:47.203Z" level=info msg="node changed" namespace=argo new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=hello-world-jwjlf old.message= old.phase=Pending old.progress=0/1 workflow=hello-world-jwjlf
time="2024-07-23T15:47:47.203Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:47:47.203Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:47:47.203Z" level=info msg="Workflow to be dehydrated" Workflow Size=1414
time="2024-07-23T15:47:47.208Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=1203 workflow=hello-world-jwjlf
time="2024-07-23T15:47:57.209Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1203 namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:47:57.210Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=hello-world-jwjlf
time="2024-07-23T15:47:57.210Z" level=info msg="node unchanged" namespace=argo nodeID=hello-world-jwjlf workflow=hello-world-jwjlf
time="2024-07-23T15:47:57.210Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:47:57.210Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:48:45.578Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1203 namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:48:45.578Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=hello-world-jwjlf
time="2024-07-23T15:48:45.578Z" level=info msg="task-result changed" namespace=argo nodeID=hello-world-jwjlf workflow=hello-world-jwjlf
time="2024-07-23T15:48:45.578Z" level=info msg="node changed" namespace=argo new.message="ErrImagePull: [DEPRECATION NOTICE] Docker Image Format v1 and Docker Image manifest version 2, schema 1 support is disabled by default and will be removed in an upcoming release. Suggest the author of docker.io/docker/whalesay:latest to upgrade the image to the OCI Format or Docker Image manifest v2, schema 2. More information at https://docs.docker.com/go/deprecated-image-specs/" new.phase=Pending new.progress=0/1 nodeID=hello-world-jwjlf old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=hello-world-jwjlf
time="2024-07-23T15:48:45.579Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:48:45.579Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:48:45.579Z" level=info msg="Workflow to be dehydrated" Workflow Size=1806
time="2024-07-23T15:48:45.587Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=1307 workflow=hello-world-jwjlf
time="2024-07-23T15:48:55.587Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1307 namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:48:55.587Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=hello-world-jwjlf
time="2024-07-23T15:48:55.587Z" level=info msg="task-result changed" namespace=argo nodeID=hello-world-jwjlf workflow=hello-world-jwjlf
time="2024-07-23T15:48:55.588Z" level=info msg="node changed" namespace=argo new.message="ImagePullBackOff: Back-off pulling image \"docker/whalesay:latest\"" new.phase=Pending new.progress=0/1 nodeID=hello-world-jwjlf old.message="ErrImagePull: [DEPRECATION NOTICE] Docker Image Format v1 and Docker Image manifest version 2, schema 1 support is disabled by default and will be removed in an upcoming release. Suggest the author of docker.io/docker/whalesay:latest to upgrade the image to the OCI Format or Docker Image manifest v2, schema 2. More information at https://docs.docker.com/go/deprecated-image-specs/" old.phase=Pending old.progress=0/1 workflow=hello-world-jwjlf
time="2024-07-23T15:48:55.588Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:48:55.588Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:48:55.588Z" level=info msg="Workflow to be dehydrated" Workflow Size=1488
time="2024-07-23T15:48:55.593Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=1327 workflow=hello-world-jwjlf
time="2024-07-23T15:49:17.315Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1327 namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:49:17.315Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=hello-world-jwjlf
time="2024-07-23T15:49:17.315Z" level=info msg="task-result changed" namespace=argo nodeID=hello-world-jwjlf workflow=hello-world-jwjlf
time="2024-07-23T15:49:17.316Z" level=info msg="node changed" namespace=argo new.message="ErrImagePull: [DEPRECATION NOTICE] Docker Image Format v1 and Docker Image manifest version 2, schema 1 support is disabled by default and will be removed in an upcoming release. Suggest the author of docker.io/docker/whalesay:latest to upgrade the image to the OCI Format or Docker Image manifest v2, schema 2. More information at https://docs.docker.com/go/deprecated-image-specs/" new.phase=Pending new.progress=0/1 nodeID=hello-world-jwjlf old.message="ImagePullBackOff: Back-off pulling image \"docker/whalesay:latest\"" old.phase=Pending old.progress=0/1 workflow=hello-world-jwjlf
time="2024-07-23T15:49:17.316Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:49:17.316Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:49:17.316Z" level=info msg="Workflow to be dehydrated" Workflow Size=1806
time="2024-07-23T15:49:17.322Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=1365 workflow=hello-world-jwjlf
time="2024-07-23T15:49:27.323Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1365 namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:49:27.323Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=hello-world-jwjlf
time="2024-07-23T15:49:27.323Z" level=info msg="task-result changed" namespace=argo nodeID=hello-world-jwjlf workflow=hello-world-jwjlf
time="2024-07-23T15:49:27.323Z" level=info msg="node changed" namespace=argo new.message="ImagePullBackOff: Back-off pulling image \"docker/whalesay:latest\"" new.phase=Pending new.progress=0/1 nodeID=hello-world-jwjlf old.message="ErrImagePull: [DEPRECATION NOTICE] Docker Image Format v1 and Docker Image manifest version 2, schema 1 support is disabled by default and will be removed in an upcoming release. Suggest the author of docker.io/docker/whalesay:latest to upgrade the image to the OCI Format or Docker Image manifest v2, schema 2. More information at https://docs.docker.com/go/deprecated-image-specs/" old.phase=Pending old.progress=0/1 workflow=hello-world-jwjlf
time="2024-07-23T15:49:27.323Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:49:27.323Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:49:27.323Z" level=info msg="Workflow to be dehydrated" Workflow Size=1488
time="2024-07-23T15:49:27.328Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=1385 workflow=hello-world-jwjlf
time="2024-07-23T15:49:43.319Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1385 namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:49:43.319Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=hello-world-jwjlf
time="2024-07-23T15:49:43.319Z" level=info msg="task-result changed" namespace=argo nodeID=hello-world-jwjlf workflow=hello-world-jwjlf
time="2024-07-23T15:49:43.319Z" level=info msg="node changed" namespace=argo new.message="ErrImagePull: [DEPRECATION NOTICE] Docker Image Format v1 and Docker Image manifest version 2, schema 1 support is disabled by default and will be removed in an upcoming release. Suggest the author of docker.io/docker/whalesay:latest to upgrade the image to the OCI Format or Docker Image manifest v2, schema 2. More information at https://docs.docker.com/go/deprecated-image-specs/" new.phase=Pending new.progress=0/1 nodeID=hello-world-jwjlf old.message="ImagePullBackOff: Back-off pulling image \"docker/whalesay:latest\"" old.phase=Pending old.progress=0/1 workflow=hello-world-jwjlf
time="2024-07-23T15:49:43.319Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:49:43.320Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:49:43.320Z" level=info msg="Workflow to be dehydrated" Workflow Size=1806
time="2024-07-23T15:49:43.326Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=1414 workflow=hello-world-jwjlf
time="2024-07-23T15:49:56.316Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1414 namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:49:56.316Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=hello-world-jwjlf
time="2024-07-23T15:49:56.316Z" level=info msg="task-result changed" namespace=argo nodeID=hello-world-jwjlf workflow=hello-world-jwjlf
time="2024-07-23T15:49:56.317Z" level=info msg="node changed" namespace=argo new.message="ImagePullBackOff: Back-off pulling image \"docker/whalesay:latest\"" new.phase=Pending new.progress=0/1 nodeID=hello-world-jwjlf old.message="ErrImagePull: [DEPRECATION NOTICE] Docker Image Format v1 and Docker Image manifest version 2, schema 1 support is disabled by default and will be removed in an upcoming release. Suggest the author of docker.io/docker/whalesay:latest to upgrade the image to the OCI Format or Docker Image manifest v2, schema 2. More information at https://docs.docker.com/go/deprecated-image-specs/" old.phase=Pending old.progress=0/1 workflow=hello-world-jwjlf
time="2024-07-23T15:49:56.317Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:49:56.317Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:49:56.317Z" level=info msg="Workflow to be dehydrated" Workflow Size=1488
time="2024-07-23T15:49:56.323Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=1436 workflow=hello-world-jwjlf
time="2024-07-23T15:50:26.721Z" level=info msg="Alloc=6871 TotalAlloc=19966 Sys=29541 NumGC=7 Goroutines=175"
time="2024-07-23T15:50:39.318Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1436 namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:50:39.318Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=hello-world-jwjlf
time="2024-07-23T15:50:39.318Z" level=info msg="task-result changed" namespace=argo nodeID=hello-world-jwjlf workflow=hello-world-jwjlf
time="2024-07-23T15:50:39.318Z" level=info msg="node changed" namespace=argo new.message="ErrImagePull: [DEPRECATION NOTICE] Docker Image Format v1 and Docker Image manifest version 2, schema 1 support is disabled by default and will be removed in an upcoming release. Suggest the author of docker.io/docker/whalesay:latest to upgrade the image to the OCI Format or Docker Image manifest v2, schema 2. More information at https://docs.docker.com/go/deprecated-image-specs/" new.phase=Pending new.progress=0/1 nodeID=hello-world-jwjlf old.message="ImagePullBackOff: Back-off pulling image \"docker/whalesay:latest\"" old.phase=Pending old.progress=0/1 workflow=hello-world-jwjlf
time="2024-07-23T15:50:39.318Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:50:39.318Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:50:39.318Z" level=info msg="Workflow to be dehydrated" Workflow Size=1806
time="2024-07-23T15:50:39.324Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=1505 workflow=hello-world-jwjlf
time="2024-07-23T15:50:52.316Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1505 namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:50:52.316Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=hello-world-jwjlf
time="2024-07-23T15:50:52.316Z" level=info msg="task-result changed" namespace=argo nodeID=hello-world-jwjlf workflow=hello-world-jwjlf
time="2024-07-23T15:50:52.316Z" level=info msg="node changed" namespace=argo new.message="ImagePullBackOff: Back-off pulling image \"docker/whalesay:latest\"" new.phase=Pending new.progress=0/1 nodeID=hello-world-jwjlf old.message="ErrImagePull: [DEPRECATION NOTICE] Docker Image Format v1 and Docker Image manifest version 2, schema 1 support is disabled by default and will be removed in an upcoming release. Suggest the author of docker.io/docker/whalesay:latest to upgrade the image to the OCI Format or Docker Image manifest v2, schema 2. More information at https://docs.docker.com/go/deprecated-image-specs/" old.phase=Pending old.progress=0/1 workflow=hello-world-jwjlf
time="2024-07-23T15:50:52.316Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:50:52.316Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:50:52.316Z" level=info msg="Workflow to be dehydrated" Workflow Size=1488
time="2024-07-23T15:50:52.323Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=1528 workflow=hello-world-jwjlf
time="2024-07-23T15:51:02.324Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1528 namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:51:02.324Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=hello-world-jwjlf
time="2024-07-23T15:51:02.324Z" level=info msg="task-result changed" namespace=argo nodeID=hello-world-jwjlf workflow=hello-world-jwjlf
time="2024-07-23T15:51:02.324Z" level=info msg="node unchanged" namespace=argo nodeID=hello-world-jwjlf workflow=hello-world-jwjlf
time="2024-07-23T15:51:02.324Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:51:02.324Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:51:02.324Z" level=info msg="Workflow to be dehydrated" Workflow Size=1488
time="2024-07-23T15:51:02.328Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=1528 workflow=hello-world-jwjlf
time="2024-07-23T15:52:02.313Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1528 namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:52:02.313Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=hello-world-jwjlf
time="2024-07-23T15:52:02.313Z" level=info msg="task-result changed" namespace=argo nodeID=hello-world-jwjlf workflow=hello-world-jwjlf
time="2024-07-23T15:52:02.313Z" level=info msg="node changed" namespace=argo new.message="ErrImagePull: [DEPRECATION NOTICE] Docker Image Format v1 and Docker Image manifest version 2, schema 1 support is disabled by default and will be removed in an upcoming release. Suggest the author of docker.io/docker/whalesay:latest to upgrade the image to the OCI Format or Docker Image manifest v2, schema 2. More information at https://docs.docker.com/go/deprecated-image-specs/" new.phase=Pending new.progress=0/1 nodeID=hello-world-jwjlf old.message="ImagePullBackOff: Back-off pulling image \"docker/whalesay:latest\"" old.phase=Pending old.progress=0/1 workflow=hello-world-jwjlf
time="2024-07-23T15:52:02.314Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:52:02.314Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:52:02.314Z" level=info msg="Workflow to be dehydrated" Workflow Size=1806
time="2024-07-23T15:52:02.321Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=1637 workflow=hello-world-jwjlf
time="2024-07-23T15:52:12.322Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1637 namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:52:12.322Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=hello-world-jwjlf
time="2024-07-23T15:52:12.322Z" level=info msg="task-result changed" namespace=argo nodeID=hello-world-jwjlf workflow=hello-world-jwjlf
time="2024-07-23T15:52:12.322Z" level=info msg="node changed" namespace=argo new.message="ImagePullBackOff: Back-off pulling image \"docker/whalesay:latest\"" new.phase=Pending new.progress=0/1 nodeID=hello-world-jwjlf old.message="ErrImagePull: [DEPRECATION NOTICE] Docker Image Format v1 and Docker Image manifest version 2, schema 1 support is disabled by default and will be removed in an upcoming release. Suggest the author of docker.io/docker/whalesay:latest to upgrade the image to the OCI Format or Docker Image manifest v2, schema 2. More information at https://docs.docker.com/go/deprecated-image-specs/" old.phase=Pending old.progress=0/1 workflow=hello-world-jwjlf
time="2024-07-23T15:52:12.322Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:52:12.322Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:52:12.322Z" level=info msg="Workflow to be dehydrated" Workflow Size=1488
time="2024-07-23T15:52:12.331Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=1654 workflow=hello-world-jwjlf
time="2024-07-23T15:52:25.749Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1654 namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:52:25.750Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=hello-world-jwjlf
time="2024-07-23T15:52:25.750Z" level=info msg="task-result changed" namespace=argo nodeID=hello-world-jwjlf workflow=hello-world-jwjlf
time="2024-07-23T15:52:25.750Z" level=info msg="node unchanged" namespace=argo nodeID=hello-world-jwjlf workflow=hello-world-jwjlf
time="2024-07-23T15:52:25.750Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:52:25.750Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:52:25.750Z" level=info msg="Workflow to be dehydrated" Workflow Size=1488
time="2024-07-23T15:52:25.753Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=1654 workflow=hello-world-jwjlf
time="2024-07-23T15:54:50.308Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1654 namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:54:50.308Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=hello-world-jwjlf
time="2024-07-23T15:54:50.308Z" level=info msg="task-result changed" namespace=argo nodeID=hello-world-jwjlf workflow=hello-world-jwjlf
time="2024-07-23T15:54:50.308Z" level=info msg="node changed" namespace=argo new.message="ErrImagePull: [DEPRECATION NOTICE] Docker Image Format v1 and Docker Image manifest version 2, schema 1 support is disabled by default and will be removed in an upcoming release. Suggest the author of docker.io/docker/whalesay:latest to upgrade the image to the OCI Format or Docker Image manifest v2, schema 2. More information at https://docs.docker.com/go/deprecated-image-specs/" new.phase=Pending new.progress=0/1 nodeID=hello-world-jwjlf old.message="ImagePullBackOff: Back-off pulling image \"docker/whalesay:latest\"" old.phase=Pending old.progress=0/1 workflow=hello-world-jwjlf
time="2024-07-23T15:54:50.309Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:54:50.309Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:54:50.309Z" level=info msg="Workflow to be dehydrated" Workflow Size=1806
time="2024-07-23T15:54:50.320Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=1898 workflow=hello-world-jwjlf
time="2024-07-23T15:55:02.308Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1898 namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:55:02.308Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=hello-world-jwjlf
time="2024-07-23T15:55:02.308Z" level=info msg="task-result changed" namespace=argo nodeID=hello-world-jwjlf workflow=hello-world-jwjlf
time="2024-07-23T15:55:02.308Z" level=info msg="node changed" namespace=argo new.message="ImagePullBackOff: Back-off pulling image \"docker/whalesay:latest\"" new.phase=Pending new.progress=0/1 nodeID=hello-world-jwjlf old.message="ErrImagePull: [DEPRECATION NOTICE] Docker Image Format v1 and Docker Image manifest version 2, schema 1 support is disabled by default and will be removed in an upcoming release. Suggest the author of docker.io/docker/whalesay:latest to upgrade the image to the OCI Format or Docker Image manifest v2, schema 2. More information at https://docs.docker.com/go/deprecated-image-specs/" old.phase=Pending old.progress=0/1 workflow=hello-world-jwjlf
time="2024-07-23T15:55:02.309Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:55:02.309Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:55:02.309Z" level=info msg="Workflow to be dehydrated" Workflow Size=1488
time="2024-07-23T15:55:02.314Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=1918 workflow=hello-world-jwjlf
time="2024-07-23T15:55:26.714Z" level=info msg="Alloc=7487 TotalAlloc=25398 Sys=29541 NumGC=9 Goroutines=175"
time="2024-07-23T15:58:30.742Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=1918 namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:58:30.742Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=hello-world-jwjlf
time="2024-07-23T15:58:30.742Z" level=info msg="task-result changed" namespace=argo nodeID=hello-world-jwjlf workflow=hello-world-jwjlf
time="2024-07-23T15:58:30.742Z" level=info msg="node unchanged" namespace=argo nodeID=hello-world-jwjlf workflow=hello-world-jwjlf
time="2024-07-23T15:58:30.742Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:58:30.742Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-jwjlf
time="2024-07-23T15:58:30.742Z" level=info msg="Workflow to be dehydrated" Workflow Size=1488
time="2024-07-23T15:58:30.745Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=1918 workflow=hello-world-jwjlf

[dazinator]

I found some discussion related to this here: #12767 (comment)

What's the resolution?

[agilgur5]

I found some discussion related to this here: #12767 (comment)

Per that discussion, it's an upstream issue unfortunately with the image: docker/whalesay#6. Per the issue, I also tweeted out at Docker to try to get their attention that way, and it got some retweets, but no response unfortunately: https://x.com/agilgur5/status/1790084420642255219 😕

What's the resolution?

In this case, it would be to use a different image from whalesay. It may make sense to update the docs, but it's not clear what an alternative image would be to use. whalesay is also used in a lot more docs and examples than just Argo -- it's a universal hello world image used all over the ecosystem. So it would be good to fix that everywhere, but we don't control Docker's official accounts 😕 For security purposes, we also shouldn't use someone's personal rebuild of whalesay either in the official docs

Same root cause as #11858 as well

[agilgur5]

Same root cause as #11858 as well

Closing as the resolution to this would be the same as that issue

[dazinator]

Ok. I understand the problem and the desire to use whalesay. However as that isn't forthcoming, perhaps the docs and the getting started example could be re-worked, for example to use an alpine image or busybox or any other popular working image that would let you just echo some text or whatever was required. This would mean re-working the getting started example - as things stand it's not a good getting started experience. If the issue you have linked will resolve the getting started experience then I am happy this can be closed.

[agilgur5]

However as that isn't forthcoming, perhaps the docs and the getting started example could be re-worked, for example to use an alpine image or busybox or any other popular working image that would let you just echo some text or whatever was required

Yea I was thinking the exact same to switch to alpine or busybox or something else minimal with arbitrary text printing support like echo

This would mean re-working the getting started example

Unfortunately, it means re-working a lot more than that: quick search says whalesay is used ~191 times in the codebase 🫠

PRs certainly welcome though!

If the issue you have linked will resolve the getting started experience then I am happy this can be closed.

#11858 is regarding arm64 support for example images, but same root cause in that whalesay is incompatible. So we need to replace it with some image that is both arm64 and manifest v2 compatible (which is a pretty small ask these days, whalesay just hasn't been touched in 9+ years so predates multi-arch images and manifest v2).

Also I did get a response from Docker Support in docker/whalesay#6 (comment) (I emailed them after you raised this, unfortunately it's not quite a sufficient response).
And also found a drop-in alternative that would work in your case: rancher/cowsay. There's no arm64 build of it however, so not quite sufficient as a replacement for all examples, but would still be better than whalesay.