Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

artifact passing problem in v2.4.1 #1672

Closed
wanghong230 opened this issue Oct 12, 2019 · 1 comment
Closed

artifact passing problem in v2.4.1 #1672

wanghong230 opened this issue Oct 12, 2019 · 1 comment
Labels
type/bug
Milestone
v2.4

Comments

@wanghong230
Copy link
Member

Is this a BUG REPORT or FEATURE REQUEST?:
BUG
What happened:

Our workflow is working fine in 2.2.x and also 2.3.x. I tried to use 2.4.1, then my workflow failed at the last with error in the controller log.

(cl-c-iksm-use2-3945-c1-bf143-98hbb-586212985) deemed errored due to child cl-c-iksm-use2-3945-c1-bf143-98hbb[4].iks-state-archive error: inputs.artifacts.iks-state-tar was not supplied"

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Controller Log:

time="2019-10-12T16:40:43Z" level=info msg="Labeled pod dev-patterns-iksmanager-usw2-ppd-qal1/cl-c-iksm-use2-3945-c1-bf143-98hbb-299044679 completed"
--
  | time="2019-10-12T16:40:42Z" level=info msg="Labeled pod dev-patterns-iksmanager-usw2-ppd-qal1/cl-c-iksm-use2-3945-c1-bf143-98hbb-299044679 completed"
  | time="2019-10-12T16:40:42Z" level=info msg="Workflow update successful" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="Checking daemoned children of " namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="Marking workflow completed" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="Updated message  -> child 'cl-c-iksm-use2-3945-c1-bf143-98hbb[4].iks-state-archive' errored" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="Updated phase Running -> Failed" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="node cl-c-iksm-use2-3945-c1-bf143-98hbb (cl-c-iksm-use2-3945-c1-bf143-98hbb) finished: 2019-10-12 16:40:42.392635364 +0000 UTC" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="Checking daemoned children of cl-c-iksm-use2-3945-c1-bf143-98hbb" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="node cl-c-iksm-use2-3945-c1-bf143-98hbb (cl-c-iksm-use2-3945-c1-bf143-98hbb) message: child 'cl-c-iksm-use2-3945-c1-bf143-98hbb[4].iks-state-archive' errored" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="node cl-c-iksm-use2-3945-c1-bf143-98hbb (cl-c-iksm-use2-3945-c1-bf143-98hbb) phase Running -> Failed" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="Outbound nodes of cl-c-iksm-use2-3945-c1-bf143-98hbb is [cl-c-iksm-use2-3945-c1-bf143-98hbb-1318224576]" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="Outbound nodes of cl-c-iksm-use2-3945-c1-bf143-98hbb-1211005337 is [cl-c-iksm-use2-3945-c1-bf143-98hbb-1318224576]" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="step group cl-c-iksm-use2-3945-c1-bf143-98hbb-586212985 was unsuccessful: child 'cl-c-iksm-use2-3945-c1-bf143-98hbb[4].iks-state-archive' errored" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="node cl-c-iksm-use2-3945-c1-bf143-98hbb[4] (cl-c-iksm-use2-3945-c1-bf143-98hbb-586212985) finished: 2019-10-12 16:40:42.392537934 +0000 UTC" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="node cl-c-iksm-use2-3945-c1-bf143-98hbb[4] (cl-c-iksm-use2-3945-c1-bf143-98hbb-586212985) message: child 'cl-c-iksm-use2-3945-c1-bf143-98hbb[4].iks-state-archive' errored" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="node cl-c-iksm-use2-3945-c1-bf143-98hbb[4] (cl-c-iksm-use2-3945-c1-bf143-98hbb-586212985) phase Running -> Error" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="Step group node cl-c-iksm-use2-3945-c1-bf143-98hbb[4] (cl-c-iksm-use2-3945-c1-bf143-98hbb-586212985) deemed errored due to child cl-c-iksm-use2-3945-c1-bf143-98hbb[4].iks-state-archive error: inputs.artifacts.iks-state-tar was not supplied" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="node cl-c-iksm-use2-3945-c1-bf143-98hbb[4].iks-state-archive (cl-c-iksm-use2-3945-c1-bf143-98hbb-1211005337) finished: 2019-10-12 16:40:42.392466101 +0000 UTC" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="node cl-c-iksm-use2-3945-c1-bf143-98hbb[4].iks-state-archive (cl-c-iksm-use2-3945-c1-bf143-98hbb-1211005337) message: inputs.artifacts.iks-state-tar was not supplied" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="node cl-c-iksm-use2-3945-c1-bf143-98hbb[4].iks-state-archive (cl-c-iksm-use2-3945-c1-bf143-98hbb-1211005337) phase Running -> Error" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="node cl-c-iksm-use2-3945-c1-bf143-98hbb[3] (cl-c-iksm-use2-3945-c1-bf143-98hbb-586654270) phase Running -> Succeeded" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="SG Outbound nodes of cl-c-iksm-use2-3945-c1-bf143-98hbb-299044679 are [cl-c-iksm-use2-3945-c1-bf143-98hbb-299044679]" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="Step group node cl-c-iksm-use2-3945-c1-bf143-98hbb[3] (cl-c-iksm-use2-3945-c1-bf143-98hbb-586654270) successful" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="SG Outbound nodes of cl-c-iksm-use2-3945-c1-bf143-98hbb-468017586 are [cl-c-iksm-use2-3945-c1-bf143-98hbb-468017586]" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="node cl-c-iksm-use2-3945-c1-bf143-98hbb[3] (cl-c-iksm-use2-3945-c1-bf143-98hbb-586654270) phase Succeeded -> Running" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="node cl-c-iksm-use2-3945-c1-bf143-98hbb[2] (cl-c-iksm-use2-3945-c1-bf143-98hbb-519396699) phase Running -> Succeeded" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="Step group node cl-c-iksm-use2-3945-c1-bf143-98hbb[2] (cl-c-iksm-use2-3945-c1-bf143-98hbb-519396699) successful" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="SG Outbound nodes of cl-c-iksm-use2-3945-c1-bf143-98hbb-2486535658 are [cl-c-iksm-use2-3945-c1-bf143-98hbb-2486535658]" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="node cl-c-iksm-use2-3945-c1-bf143-98hbb[2] (cl-c-iksm-use2-3945-c1-bf143-98hbb-519396699) phase Succeeded -> Running" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="node cl-c-iksm-use2-3945-c1-bf143-98hbb[1] (cl-c-iksm-use2-3945-c1-bf143-98hbb-1593605600) phase Running -> Succeeded" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="Step group node cl-c-iksm-use2-3945-c1-bf143-98hbb[1] (cl-c-iksm-use2-3945-c1-bf143-98hbb-1593605600) successful" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="SG Outbound nodes of cl-c-iksm-use2-3945-c1-bf143-98hbb-3745091569 are [cl-c-iksm-use2-3945-c1-bf143-98hbb-3745091569]" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="node cl-c-iksm-use2-3945-c1-bf143-98hbb[1] (cl-c-iksm-use2-3945-c1-bf143-98hbb-1593605600) phase Succeeded -> Running" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="node cl-c-iksm-use2-3945-c1-bf143-98hbb[0] (cl-c-iksm-use2-3945-c1-bf143-98hbb-1660568981) phase Running -> Succeeded" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="Step group node cl-c-iksm-use2-3945-c1-bf143-98hbb[0] (cl-c-iksm-use2-3945-c1-bf143-98hbb-1660568981) successful" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="node cl-c-iksm-use2-3945-c1-bf143-98hbb[0] (cl-c-iksm-use2-3945-c1-bf143-98hbb-1660568981) phase Succeeded -> Running" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:42Z" level=info msg="Updating node cl-c-iksm-use2-3945-c1-bf143-98hbb[4].iks-state-archive(0) (cl-c-iksm-use2-3945-c1-bf143-98hbb-1318224576) message: PodInitializing"
  | time="2019-10-12T16:40:42Z" level=info msg="Processing workflow" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:41Z" level=info msg="Workflow update successful" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:41Z" level=info msg="Workflow step group node cl-c-iksm-use2-3945-c1-bf143-98hbb[4] (cl-c-iksm-use2-3945-c1-bf143-98hbb-586212985) not yet completed" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:41Z" level=info msg="Created pod: cl-c-iksm-use2-3945-c1-bf143-98hbb[4].iks-state-archive(0) (cl-c-iksm-use2-3945-c1-bf143-98hbb-1318224576)" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:41Z" level=info msg="Pod node cl-c-iksm-use2-3945-c1-bf143-98hbb[4].iks-state-archive(0) (cl-c-iksm-use2-3945-c1-bf143-98hbb-1318224576) initialized Pending" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:41Z" level=info msg="Retry node cl-c-iksm-use2-3945-c1-bf143-98hbb[4].iks-state-archive (cl-c-iksm-use2-3945-c1-bf143-98hbb-1211005337) initialized Running" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:41Z" level=info msg="SG Outbound nodes of cl-c-iksm-use2-3945-c1-bf143-98hbb-299044679 are [cl-c-iksm-use2-3945-c1-bf143-98hbb-299044679]" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:41Z" level=info msg="StepGroup node cl-c-iksm-use2-3945-c1-bf143-98hbb[4] (cl-c-iksm-use2-3945-c1-bf143-98hbb-586212985) initialized Running" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:41Z" level=info msg="node cl-c-iksm-use2-3945-c1-bf143-98hbb[3] (cl-c-iksm-use2-3945-c1-bf143-98hbb-586654270) finished: 2019-10-12 16:40:41.040553156 +0000 UTC" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:41Z" level=info msg="node cl-c-iksm-use2-3945-c1-bf143-98hbb[3] (cl-c-iksm-use2-3945-c1-bf143-98hbb-586654270) phase Running -> Succeeded" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:41Z" level=info msg="Step group node cl-c-iksm-use2-3945-c1-bf143-98hbb[3] (cl-c-iksm-use2-3945-c1-bf143-98hbb-586654270) successful" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
  | time="2019-10-12T16:40:41Z" level=info msg="SG Outbound nodes of cl-c-iksm-use2-3945-c1-bf143-98hbb-468017586 are [cl-c-iksm-use2-3945-c1-bf143-98hbb-468017586]" namespace=dev-patterns-iksmanager-usw2-ppd-qal1 workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb

WF

Name:                cl-c-iksm-use2-3945-c1-bf143-98hbb
Namespace:           dev-patterns-iksmanager-usw2-ppd-qal1
ServiceAccount:      default
Status:              Failed
Message:             child 'cl-c-iksm-use2-3945-c1-bf143-98hbb[4].iks-state-archive' errored
Created:             Sat Oct 12 09:28:58 -0700 (1 hour ago)
Started:             Sat Oct 12 09:28:58 -0700 (1 hour ago)
Finished:            Sat Oct 12 09:40:42 -0700 (1 hour ago)
Duration:            11 minutes 44 seconds
Parameters:          
*******

�[39mSTEP�[0m                                   PODNAME                                        DURATION  MESSAGE
 �[31m✖�[0m cl-c-iksm-use2-3945-c1-bf143-98hbb                                                           child 'cl-c-iksm-use2-3945-c1-bf143-98hbb[4].iks-state-archive' errored
 ├---�[32m✔�[0m ssh-key-generate                cl-c-iksm-use2-3945-c1-bf143-98hbb-3745091569  5s        
 ├---�[32m✔�[0m aws-check                       cl-c-iksm-use2-3945-c1-bf143-98hbb-2486535658  4s        
 ├---�[32m✔�[0m cluster-create                  cl-c-iksm-use2-3945-c1-bf143-98hbb-468017586   11m       
 ├---�[32m✔�[0m iks-state-sync                  cl-c-iksm-use2-3945-c1-bf143-98hbb-299044679   7s        
 └--- iks-state-archive(0)                      cl-c-iksm-use2-3945-c1-bf143-98hbb-1318224576  1h        PodInitializing

POD is finished fine:

Name:               cl-c-iksm-use2-3945-c1-bf143-98hbb-1318224576
Namespace:          dev-patterns-iksmanager-usw2-ppd-qal1
Priority:           0
PriorityClassName:  <none>
Node:               ip-10-80-104-83.us-west-2.compute.internal/10.80.104.83
Start Time:         Sat, 12 Oct 2019 09:40:41 -0700
Labels:             workflows.argoproj.io/completed=false
                    workflows.argoproj.io/workflow=cl-c-iksm-use2-3945-c1-bf143-98hbb
Annotations:        iam.amazonaws.com/role: k8s-dev-patterns-iksmanager-usw2-ppd-qal1
                    kubernetes.io/limit-ranger:
                      LimitRanger plugin set: cpu, memory request for container wait; memory limit for container wait; memory limit for container main; cpu, mem...
                    workflows.argoproj.io/node-name: cl-c-iksm-use2-3945-c1-bf143-98hbb[4].iks-state-archive(0)
                    workflows.argoproj.io/outputs:
                      {"artifacts":[{"name":"main-logs","archiveLogs":true,"s3":{"endpoint":"s3.amazonaws.com","bucket":"iksm-bucket-us-west-2-qal1","region":"u...
                    workflows.argoproj.io/template:
                      {"name":"iks-state-archive","arguments":{},"inputs":{"artifacts":[{"name":"iks-state-tar","path":"/home/iks/iks","mode":511,"s3":{"endpoin...
Status:             Succeeded
IP:                 100.124.154.84
Controlled By:      Workflow/cl-c-iksm-use2-3945-c1-bf143-98hbb
Init Containers:
  init:
    Container ID:  docker://79b533fb42e63275685568c4dfe8ad5ea7fe7b29a7be4b2cb1aaaf9a3261240f
    Image:         docker.artifactory.a.intuit.com/dev/patterns/kubernetes/argoproj/argoexec:v2.4.1
    Image ID:      docker-pullable://docker.artifactory.a.intuit.com/dev/patterns/kubernetes/argoproj/argoexec@sha256:928c56b440d612bf4adad9dc3daade312489cc96d16767e3ed82bc8bc7ef97b3
    Port:          <none>
    Host Port:     <none>
    Command:
      argoexec
      init
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 12 Oct 2019 09:40:43 -0700
      Finished:     Sat, 12 Oct 2019 09:40:43 -0700
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  4Gi
    Requests:
      cpu:     100m
      memory:  256Mi
    Environment:
      ARGO_POD_NAME:  cl-c-iksm-use2-3945-c1-bf143-98hbb-1318224576 (v1:metadata.name)
    Mounts:
      /argo/inputs/artifacts from input-artifacts (rw)
      /argo/podmetadata from podmetadata (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-2tfnd (ro)
Containers:
  wait:
    Container ID:  docker://b3c7250f14a2addf47054601a06c6e8b5600a3d57ccd1f270a91e3d9ef6de4d4
    Image:         docker.artifactory.a.intuit.com/dev/patterns/kubernetes/argoproj/argoexec:v2.4.1
    Image ID:      docker-pullable://docker.artifactory.a.intuit.com/dev/patterns/kubernetes/argoproj/argoexec@sha256:928c56b440d612bf4adad9dc3daade312489cc96d16767e3ed82bc8bc7ef97b3
    Port:          <none>
    Host Port:     <none>
    Command:
      argoexec
      wait
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 12 Oct 2019 09:40:45 -0700
      Finished:     Sat, 12 Oct 2019 09:40:48 -0700
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  4Gi
    Requests:
      cpu:     100m
      memory:  256Mi
    Environment:
      ARGO_POD_NAME:  cl-c-iksm-use2-3945-c1-bf143-98hbb-1318224576 (v1:metadata.name)
    Mounts:
      /argo/podmetadata from podmetadata (rw)
      /mainctrfs/home/iks/iks from input-artifacts (rw,path="iks-state-tar")
      /mainctrfs/home/iks/kubeconfig from input-artifacts (rw,path="kube-admin-config")
      /var/run/docker.sock from docker-sock (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-2tfnd (ro)
  main:
    Container ID:  docker://a51979ef38dae7f4eb090566c1c031207a95a9e47114ec07a3eea4cea490fe37
    Image:         docker.artifactory.a.intuit.com/dev/patterns/kubernetes/dev/iksm-worker:2.9.3-5ae811f
    Image ID:      docker-pullable://docker.artifactory.a.intuit.com/dev/patterns/kubernetes/dev/iksm-worker@sha256:82557ee3be9795cc749b2c13828dac823c3f1f730853104fb0104df9aea9a721
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
    Args:
      aws s3 sync /home/iks/iks s3://iksm-bucket-us-west-2-qal1/clusters/iksm-use2-3945-c1.cluster.k8s.local/iks && aws s3 cp /home/iks/kubeconfig s3://iksm-bucket-us-west-2-qal1/clusters/iksm-use2-3945-c1.cluster.k8s.local/kubeconfigs/admin 
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 12 Oct 2019 09:40:45 -0700
      Finished:     Sat, 12 Oct 2019 09:40:47 -0700
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  4Gi
    Requests:
      cpu:     100m
      memory:  200Mi
    Environment:
      AWS_METADATA_SERVICE_TIMEOUT:       10
      AWS_METADATA_SERVICE_NUM_ATTEMPTS:  5
    Mounts:
      /home/iks/iks from input-artifacts (rw,path="iks-state-tar")
      /home/iks/kubeconfig from input-artifacts (rw,path="kube-admin-config")
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-2tfnd (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  podmetadata:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations -> annotations
  docker-sock:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/docker.sock
    HostPathType:  Socket
  input-artifacts:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  default-token-2tfnd:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-2tfnd
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  kops.k8s.io/instancegroup=nodes
Tolerations:     ig/nodes
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

Environment:

  • Argo version:
2.4.1
  • Kubernetes version :
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:16Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

Other debugging information (if applicable):

  • workflow result:
$ argo get <workflowname>
  • executor logs:
$ kubectl logs <failedpodname> -c init
hwang8@intuitdep88fe9 argo (master) $ k logs  cl-c-iksm-use2-3945-c1-bf143-98hbb-1318224576 init
time="2019-10-12T16:40:43Z" level=info msg="Creating a docker executor"
time="2019-10-12T16:40:43Z" level=info msg="Executor (version: v2.4.1, build_date: 2019-10-08T23:14:37Z) initialized (pod: dev-patterns-iksmanager-usw2-ppd-qal1/cl-c-iksm-use2-3945-c1-bf143-98hbb-1318224576) with template:\n{\"name\":\"iks-state-archive\",\"arguments\":{},\"inputs\":{\"artifacts\":[{\"name\":\"iks-state-tar\",\"path\":\"/home/iks/iks\",\"mode\":511,\"s3\":{\"endpoint\":\"s3.amazonaws.com\",\"bucket\":\"iksm-bucket-us-west-2-qal1\",\"region\":\"us-west-2\",\"accessKeySecret\":{\"key\":\"\"},\"secretKeySecret\":{\"key\":\"\"},\"key\":\"clusters/iksm-use2-3945-c1.cluster.k8s.local/iks.tgz\"}},{\"name\":\"kube-admin-config\",\"path\":\"/home/iks/kubeconfig\",\"mode\":511,\"s3\":{\"endpoint\":\"s3.amazonaws.com\",\"bucket\":\"iksm-bucket-us-west-2-qal1\",\"region\":\"us-west-2\",\"accessKeySecret\":{\"key\":\"\"},\"secretKeySecret\":{\"key\":\"\"},\"key\":\"clusters/iksm-use2-3945-c1.cluster.k8s.local/kubeconfig.tgz\"}}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"iam.amazonaws.com/role\":\"k8s-dev-patterns-iksmanager-usw2-ppd-qal1\"}},\"container\":{\"name\":\"\",\"image\":\"docker.artifactory.a.intuit.com/dev/patterns/kubernetes/dev/iksm-worker:2.9.3-5ae811f\",\"command\":[\"sh\",\"-c\"],\"args\":[\"aws s3 sync /home/iks/iks s3://iksm-bucket-us-west-2-qal1/clusters/iksm-use2-3945-c1.cluster.k8s.local/iks \\u0026\\u0026 aws s3 cp /home/iks/kubeconfig s3://iksm-bucket-us-west-2-qal1/clusters/iksm-use2-3945-c1.cluster.k8s.local/kubeconfigs/admin \"],\"env\":[{\"name\":\"AWS_METADATA_SERVICE_TIMEOUT\",\"value\":\"10\"},{\"name\":\"AWS_METADATA_SERVICE_NUM_ATTEMPTS\",\"value\":\"5\"}],\"resources\":{\"requests\":{\"cpu\":\"100m\",\"memory\":\"200Mi\"}},\"imagePullPolicy\":\"Always\"},\"archiveLocation\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"s3.amazonaws.com\",\"bucket\":\"iksm-bucket-us-west-2-qal1\",\"region\":\"us-west-2\",\"accessKeySecret\":{\"key\":\"\"},\"secretKeySecret\":{\"key\":\"\"},\"key\":\"clusters/iksm-use2-3945-c1.cluster.k8s.local/workflows/cl-c-iksm-use2-3945-c1-bf143-98hbb/cl-c-iksm-use2-3945-c1-bf143-98hbb-1211005337\"}},\"retryStrategy\":{\"limit\":3}}"
time="2019-10-12T16:40:43Z" level=info msg="Start loading input artifacts..."
time="2019-10-12T16:40:43Z" level=info msg="Downloading artifact: iks-state-tar"
time="2019-10-12T16:40:43Z" level=info msg="S3 Load path: /argo/inputs/artifacts/iks-state-tar.tmp, key: clusters/iksm-use2-3945-c1.cluster.k8s.local/iks.tgz"
time="2019-10-12T16:40:43Z" level=info msg="Creating minio client s3.amazonaws.com using IAM role"
time="2019-10-12T16:40:43Z" level=info msg="Getting from s3 (endpoint: s3.amazonaws.com, bucket: iksm-bucket-us-west-2-qal1, key: clusters/iksm-use2-3945-c1.cluster.k8s.local/iks.tgz) to /argo/inputs/artifacts/iks-state-tar.tmp"
time="2019-10-12T16:40:43Z" level=info msg="[tar -tf /argo/inputs/artifacts/iks-state-tar.tmp]"
time="2019-10-12T16:40:43Z" level=info msg="tar -xf /argo/inputs/artifacts/iks-state-tar.tmp -C /argo/inputs/artifacts/iks-state-tar.tmpdir"
time="2019-10-12T16:40:43Z" level=info msg="Successfully download file: /argo/inputs/artifacts/iks-state-tar"
time="2019-10-12T16:40:43Z" level=info msg="Downloading artifact: kube-admin-config"
time="2019-10-12T16:40:43Z" level=info msg="S3 Load path: /argo/inputs/artifacts/kube-admin-config.tmp, key: clusters/iksm-use2-3945-c1.cluster.k8s.local/kubeconfig.tgz"
time="2019-10-12T16:40:43Z" level=info msg="Creating minio client s3.amazonaws.com using IAM role"
time="2019-10-12T16:40:43Z" level=info msg="Getting from s3 (endpoint: s3.amazonaws.com, bucket: iksm-bucket-us-west-2-qal1, key: clusters/iksm-use2-3945-c1.cluster.k8s.local/kubeconfig.tgz) to /argo/inputs/artifacts/kube-admin-config.tmp"
time="2019-10-12T16:40:43Z" level=info msg="[tar -tf /argo/inputs/artifacts/kube-admin-config.tmp]"
time="2019-10-12T16:40:43Z" level=info msg="tar -xf /argo/inputs/artifacts/kube-admin-config.tmp -C /argo/inputs/artifacts/kube-admin-config.tmpdir"
time="2019-10-12T16:40:43Z" level=info msg="Successfully download file: /argo/inputs/artifacts/kube-admin-config"
time="2019-10-12T16:40:43Z" level=info msg="Alloc=4891 TotalAlloc=23358 Sys=70334 NumGC=7 Goroutines=12"

$ kubectl logs <failedpodname> -c wait

hwang8@intuitdep88fe9 argo (master) $ k logs  cl-c-iksm-use2-3945-c1-bf143-98hbb-1318224576 wait
time="2019-10-12T16:40:45Z" level=info msg="Creating a docker executor"
time="2019-10-12T16:40:45Z" level=info msg="Executor (version: v2.4.1, build_date: 2019-10-08T23:14:37Z) initialized (pod: dev-patterns-iksmanager-usw2-ppd-qal1/cl-c-iksm-use2-3945-c1-bf143-98hbb-1318224576) with template:\n{\"name\":\"iks-state-archive\",\"arguments\":{},\"inputs\":{\"artifacts\":[{\"name\":\"iks-state-tar\",\"path\":\"/home/iks/iks\",\"mode\":511,\"s3\":{\"endpoint\":\"s3.amazonaws.com\",\"bucket\":\"iksm-bucket-us-west-2-qal1\",\"region\":\"us-west-2\",\"accessKeySecret\":{\"key\":\"\"},\"secretKeySecret\":{\"key\":\"\"},\"key\":\"clusters/iksm-use2-3945-c1.cluster.k8s.local/iks.tgz\"}},{\"name\":\"kube-admin-config\",\"path\":\"/home/iks/kubeconfig\",\"mode\":511,\"s3\":{\"endpoint\":\"s3.amazonaws.com\",\"bucket\":\"iksm-bucket-us-west-2-qal1\",\"region\":\"us-west-2\",\"accessKeySecret\":{\"key\":\"\"},\"secretKeySecret\":{\"key\":\"\"},\"key\":\"clusters/iksm-use2-3945-c1.cluster.k8s.local/kubeconfig.tgz\"}}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"iam.amazonaws.com/role\":\"k8s-dev-patterns-iksmanager-usw2-ppd-qal1\"}},\"container\":{\"name\":\"\",\"image\":\"docker.artifactory.a.intuit.com/dev/patterns/kubernetes/dev/iksm-worker:2.9.3-5ae811f\",\"command\":[\"sh\",\"-c\"],\"args\":[\"aws s3 sync /home/iks/iks s3://iksm-bucket-us-west-2-qal1/clusters/iksm-use2-3945-c1.cluster.k8s.local/iks \\u0026\\u0026 aws s3 cp /home/iks/kubeconfig s3://iksm-bucket-us-west-2-qal1/clusters/iksm-use2-3945-c1.cluster.k8s.local/kubeconfigs/admin \"],\"env\":[{\"name\":\"AWS_METADATA_SERVICE_TIMEOUT\",\"value\":\"10\"},{\"name\":\"AWS_METADATA_SERVICE_NUM_ATTEMPTS\",\"value\":\"5\"}],\"resources\":{\"requests\":{\"cpu\":\"100m\",\"memory\":\"200Mi\"}},\"imagePullPolicy\":\"Always\"},\"archiveLocation\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"s3.amazonaws.com\",\"bucket\":\"iksm-bucket-us-west-2-qal1\",\"region\":\"us-west-2\",\"accessKeySecret\":{\"key\":\"\"},\"secretKeySecret\":{\"key\":\"\"},\"key\":\"clusters/iksm-use2-3945-c1.cluster.k8s.local/workflows/cl-c-iksm-use2-3945-c1-bf143-98hbb/cl-c-iksm-use2-3945-c1-bf143-98hbb-1211005337\"}},\"retryStrategy\":{\"limit\":3}}"
time="2019-10-12T16:40:45Z" level=info msg="Waiting on main container"
time="2019-10-12T16:40:45Z" level=info msg="main container started with container ID: a51979ef38dae7f4eb090566c1c031207a95a9e47114ec07a3eea4cea490fe37"
time="2019-10-12T16:40:45Z" level=info msg="Starting annotations monitor"
time="2019-10-12T16:40:45Z" level=info msg="docker wait a51979ef38dae7f4eb090566c1c031207a95a9e47114ec07a3eea4cea490fe37"
time="2019-10-12T16:40:45Z" level=info msg="Starting deadline monitor"
time="2019-10-12T16:40:48Z" level=info msg="Main container completed"
time="2019-10-12T16:40:48Z" level=info msg="Saving logs"
time="2019-10-12T16:40:48Z" level=info msg="Annotations monitor stopped"
time="2019-10-12T16:40:48Z" level=info msg="[docker logs a51979ef38dae7f4eb090566c1c031207a95a9e47114ec07a3eea4cea490fe37]"
time="2019-10-12T16:40:48Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: clusters/iksm-use2-3945-c1.cluster.k8s.local/workflows/cl-c-iksm-use2-3945-c1-bf143-98hbb/cl-c-iksm-use2-3945-c1-bf143-98hbb-1211005337/main.log"
time="2019-10-12T16:40:48Z" level=info msg="Creating minio client s3.amazonaws.com using IAM role"
time="2019-10-12T16:40:48Z" level=info msg="Saving from /tmp/argo/outputs/logs/main.log to s3 (endpoint: s3.amazonaws.com, bucket: iksm-bucket-us-west-2-qal1, key: clusters/iksm-use2-3945-c1.cluster.k8s.local/workflows/cl-c-iksm-use2-3945-c1-bf143-98hbb/cl-c-iksm-use2-3945-c1-bf143-98hbb-1211005337/main.log)"
time="2019-10-12T16:40:48Z" level=info msg="No output parameters"
time="2019-10-12T16:40:48Z" level=info msg="No output artifacts"
time="2019-10-12T16:40:48Z" level=info msg="Killing sidecars"
time="2019-10-12T16:40:48Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2019-10-12T16:40:48Z" level=info msg="Annotating pod with output"
time="2019-10-12T16:40:48Z" level=info msg="Alloc=6810 TotalAlloc=25800 Sys=70590 NumGC=7 Goroutines=14"
  • workflow-controller logs:
$ kubectl logs -n argo $(kubectl get pods -l app=workflow-controller -n argo -o name)
@dtaniwaki
Copy link
Member

I think this is the same issue as #1659. Could you share your workflow so I can confirm the fix for this issue too?

@jessesuen jessesuen added this to the v2.4 milestone Oct 18, 2019
@sarabala1979 sarabala1979 mentioned this issue Oct 21, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug
Projects
None yet
Milestone
v2.4
Development

No branches or pull requests
4 participants
@simster7 @dtaniwaki @wanghong230 @jessesuen