Logging does not work after pod completes

[strin]

Is this a BUG REPORT or FEATURE REQUEST?:

Bug report.

What happened:

We are running argo on Google Kubernetes Engine, with autoscaling enabled. Argo launched the container successfully, but after it completes the node gets deleted by autoscaler and the completed pod is no longer available.

What you expected to happen:

In Argo UI clicking on logs should reveal the container logs

How to reproduce it (as minimally and precisely as possible):

Launch a job on GKE with the following config enabled.

        resources:
          limits:
            nvidia.com/gpu: 1

Anything else we need to know?:

Environment:

• Argo version: v2.2.0

$ argo version

• Kubernetes version :

$ kubectl version -o yaml

clientVersion:
  buildDate: 2017-11-09T07:26:41Z
  compiler: gc
  gitCommit: f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd
  gitTreeState: clean
  gitVersion: v1.8.3
  goVersion: go1.9.2
  major: "1"
  minor: "8"
  platform: darwin/amd64
serverVersion:
  buildDate: 2018-08-02T23:42:40Z
  compiler: gc
  gitCommit: 9b635efce81582e1da13b35a7aa539c0ccb32987
  gitTreeState: clean
  gitVersion: v1.9.7-gke.5
  goVersion: go1.9.3b4
  major: "1"
  minor: 9+
  platform: linux/amd64

[aisensiy]

Yes, the log is gone when pod is gone. It is the default behavior of kubernetes. See more infomation from https://kubernetes.io/docs/concepts/cluster-administration/logging/

[strin]

@aisensiy I wonder why argo chose to remove the pods after they are done. Another option is to follow the pattern of Kubernetes jobs:

https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#job-termination-and-cleanup

It will keep the pods unless deleted by users.

[alexmt]

Workflow controller (v2.2.0) has ability to save logs in artifacts repository:

https://github.com/argoproj/argo/blob/47a9e5560229c789b70a6624f23fb4433412fbc4/docs/workflow-controller-configmap.yaml#L23-L25

UI don't use that feature yet, though. This will be resolved in #1018

[alexmt]

I guess we can close it. @strin please feel free to reopen if necessary

[robintanner]

This archiveLogs setting isn't working for me. I don't see anything in any logs for controllers, sensors, events in the namespace that would indicate why this is not working.