Argo can't run the DAG example from docs

[prestto]

Summary

Argo expected to run DAG example , but it just hangs after pod "A"
completes it's task.

This seems to be possibly related to this issue

Diagnostics

Kubernetes provider: OVH France, also replicated on k3ds

Argo Workflows version: 3.0.7

This is my first installation/test, so non regression as far as i'm aware.

Recreate

Install on k8s

kubectl create ns argo

# wget https://raw.githubusercontent.com/argoproj/argo-workflows/stable/manifests/namespace-install.yaml
kubectl apply -n argo -f ./namespace-install.yaml

Install CLI

# Download the binary
curl -sLO https://github.com/argoproj/argo/releases/download/v3.0.7/argo-linux-amd64.gz

# Unzip
gunzip argo-linux-amd64.gz

# Make binary executable
chmod +x argo-linux-amd64

# Move binary to path
sudo mv ./argo-linux-amd64 /usr/local/bin/argo

# Test installation
argo version

# link with server
# recommended on user panel in interface
cat >> ~/.bashrc <<EOL
export ARGO_SERVER='127.0.0.1:2746' 
export ARGO_HTTP1=true  
export ARGO_SECURE=true
export ARGO_BASE_HREF=
export ARGO_TOKEN='' 
export ARGO_NAMESPACE=argo
export ARGO_INSECURE_SKIP_VERIFY=true
# export KUBECONFIG=/dev/null
EOL

# auto complete
argo completion bash | sudo tee -a /etc/bash_completion.d/argo
cat >> ~/.bashrc <<EOL
source /etc/bash_completion.d/argo
EOL

# check it works: 
argo list

# port forward for dev
kubectl -n argo port-forward svc/argo-server 2746:2746

# dev rolebinding
kubectl create rolebinding default-admin --clusterrole=admin --serviceaccount=argo:default -n argo

submit task

# copy task into file 4_dag.yml
argo submit -n argo 4_dag.yml

Progress

Looking at the GUI i see the following progress:

• pods
  • name: dag-diamond-sd8d8
    • phase: running
    • progress 0/1
  • name: dag-diamond-sd8d8.A
    • pod name: dag-diamond-sd8d8-2426179855
    • phase: running
    • progress 0/1
    • logs:
      A

Logs

From pod:

dag-diamond-sd8d8-2426179855: A

From workflow-controller (filtered):

...
time="2021-06-21T11:28:20.608Z" level=info msg="Processing workflow" namespace=argo workflow=dag-diamond-dnqmc
time="2021-06-21T11:28:20.608Z" level=info msg="Applying shutdown deadline for pod dag-diamond-dnqmc-3685358807" namespace=argo workflow=dag-diamond-dnqmc
time="2021-06-21T11:28:20.609Z" level=info msg="Updating execution control of dag-diamond-dnqmc-3685358807: {\"deadline\":\"0001-01-01T00:00:00Z\"}" namespace=argo workflow=dag-diamond-dnqmc
time="2021-06-21T11:28:20.632Z" level=info msg="Patch pods 200"
time="2021-06-21T11:28:20.634Z" level=info msg="Signalling dag-diamond-dnqmc-3685358807 of updates" namespace=argo workflow=dag-diamond-dnqmc
time="2021-06-21T11:28:20.645Z" level=info msg="https://10.3.0.1:443/api/v1/namespaces/argo/pods/dag-diamond-dnqmc-3685358807/exec?command=sh&command=-c&command=kill+-s+USR2+%24%28pidof+argoexec%29&container=wait&stderr=true&stdout=true&tty=false"
time="2021-06-21T11:28:20.817Z" level=info msg="Create exec 101"
time="2021-06-21T11:28:20.978Z" level=info msg="Signal of dag-diamond-dnqmc-3685358807 (wait) successfully issued" namespace=argo workflow=dag-diamond-dnqmc
time="2021-06-21T11:28:23.518Z" level=info msg="Get leases 200"
time="2021-06-21T11:28:23.536Z" level=info msg="Update leases 200"
time="2021-06-21T11:28:28.553Z" level=info msg="Get leases 200"
time="2021-06-21T11:28:28.569Z" level=info msg="Update leases 200"
time="2021-06-21T11:28:30.639Z" level=info msg="Processing workflow" namespace=argo workflow=dag-diamond-dnqmc
time="2021-06-21T11:28:30.639Z" level=info msg="Applying shutdown deadline for pod dag-diamond-dnqmc-3685358807" namespace=argo workflow=dag-diamond-dnqmc
time="2021-06-21T11:28:30.639Z" level=info msg="Updating execution control of dag-diamond-dnqmc-3685358807: {\"deadline\":\"0001-01-01T00:00:00Z\"}" namespace=argo workflow=dag-diamond-dnqmc
time="2021-06-21T11:28:30.673Z" level=info msg="Patch pods 200"
time="2021-06-21T11:28:30.674Z" level=info msg="Signalling dag-diamond-dnqmc-3685358807 of updates" namespace=argo workflow=dag-diamond-dnqmc
time="2021-06-21T11:28:30.679Z" level=info msg="https://10.3.0.1:443/api/v1/namespaces/argo/pods/dag-diamond-dnqmc-3685358807/exec?command=sh&command=-c&command=kill+-s+USR2+%24%28pidof+argoexec%29&container=wait&stderr=true&stdout=true&tty=false"
time="2021-06-21T11:28:30.831Z" level=info msg="Create exec 101"
time="2021-06-21T11:28:30.943Z" level=info msg="Signal of dag-diamond-dnqmc-3685358807 (wait) successfully issued" namespace=argo workflow=dag-diamond-dnqmc
...
time="2021-06-21T11:29:19.743Z" level=info msg="Get leases 200"
time="2021-06-21T11:29:19.765Z" level=info msg="Update leases 200"
...
time="2021-06-21T11:32:45.187Z" level=info msg="Processing workflow" namespace=argo workflow=dag-diamond-sd8d8
time="2021-06-21T11:32:45.201Z" level=info msg="Get configmaps 404"
time="2021-06-21T11:32:45.201Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2021-06-21T11:32:45.201Z" level=info msg="resolved artifact repository" artifactRepositoryRef=default-artifact-repository
time="2021-06-21T11:32:45.201Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=dag-diamond-sd8d8
time="2021-06-21T11:32:45.206Z" level=info msg="DAG node dag-diamond-sd8d8 initialized Running" namespace=argo workflow=dag-diamond-sd8d8
time="2021-06-21T11:32:45.206Z" level=info msg="All of node dag-diamond-sd8d8.A dependencies [] completed" namespace=argo workflow=dag-diamond-sd8d8
time="2021-06-21T11:32:45.206Z" level=info msg="Pod node dag-diamond-sd8d8-2426179855 initialized Pending" namespace=argo workflow=dag-diamond-sd8d8
time="2021-06-21T11:32:45.246Z" level=info msg="Create events 201"
time="2021-06-21T11:32:45.475Z" level=info msg="Get leases 200"
time="2021-06-21T11:32:45.489Z" level=info msg="Update leases 200"
time="2021-06-21T11:32:45.570Z" level=info msg="Create pods 201"
time="2021-06-21T11:32:45.572Z" level=info msg="Created pod: dag-diamond-sd8d8.A (dag-diamond-sd8d8-2426179855)" namespace=argo workflow=dag-diamond-sd8d8
time="2021-06-21T11:32:45.601Z" level=info msg="Update workflows 200"
time="2021-06-21T11:32:45.603Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=24070823452 workflow=dag-diamond-sd8d8
time="2021-06-21T11:32:50.560Z" level=info msg="Get leases 200"
time="2021-06-21T11:32:50.604Z" level=info msg="Update leases 200"
time="2021-06-21T11:32:55.572Z" level=info msg="Processing workflow" namespace=argo workflow=dag-diamond-sd8d8
time="2021-06-21T11:32:55.573Z" level=info msg="Updating node dag-diamond-sd8d8-2426179855 status Pending -> Running" namespace=argo workflow=dag-diamond-sd8d8
time="2021-06-21T11:32:55.606Z" level=info msg="Update workflows 409"
time="2021-06-21T11:32:55.607Z" level=warning msg="Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"dag-diamond-sd8d8\": the object has been modified; please apply your changes to the latest version and try again Conflict" namespace=argo workflow=dag-diamond-sd8d8
time="2021-06-21T11:32:55.608Z" level=info msg="Re-applying updates on latest version and retrying update" namespace=argo workflow=dag-diamond-sd8d8
time="2021-06-21T11:32:55.647Z" level=info msg="Get leases 200"
time="2021-06-21T11:32:55.659Z" level=info msg="Get workflows 200"
time="2021-06-21T11:32:55.668Z" level=info msg="Update leases 200"
time="2021-06-21T11:32:55.680Z" level=info msg="Update workflows 200"
time="2021-06-21T11:32:55.680Z" level=info msg="Update retry attempt 1 successful" namespace=argo workflow=dag-diamond-sd8d8
time="2021-06-21T11:32:55.681Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=24070827499 workflow=dag-diamond-sd8d8
time="2021-06-21T11:33:00.686Z" level=info msg="Get leases 200"
time="2021-06-21T11:33:01.811Z" level=info msg="Update leases 200"
time="2021-06-21T11:33:05.599Z" level=info msg="Processing workflow" namespace=argo workflow=dag-diamond-sd8d8
time="2021-06-21T11:33:05.787Z" level=info msg="List workflows 200"
time="2021-06-21T11:33:05.787Z" level=info msg=healthz age=5m0s err="<nil>" instanceID= labelSelector="!workflows.argoproj.io/phase,!workflows.argoproj.io/controller-instanceid" managedNamespace=argo

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[sarabala1979]

I will take a look

[sarabala1979]

time="2021-06-21T11:32:55.607Z" level=warning msg="Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io "dag-diamond-sd8d8": the object has been modified; please apply your changes to the latest version and try again Conflict" namespace=argo workflow=dag-diamond-sd8d8
time="2021-06-21T11:32:55.608Z" level=info msg="Re-applying updates on latest version and retrying update" namespace=argo workflow=dag-diamond-sd8d8
I am not able to reproduce my environment.
Can you please check there is only one controller is running on that cluster?

[no-response]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.