argo submit from a WorkflowTemplate with default exit handler produces "invalid field in spec" error

[mattskone]

Summary

Use Case

I would like to use the argo submit --from workflowtemplate/[template_name] command to run ad hoc Workflows from an existing WorkflowTemplate. This works fine. I would also like to have a default exit handler defined in the workflow controller configmap so that all of my workflows have the same exit behavior, but then when I run argo submit, the resulting workflow fails immediately with:

Status:              Failed
Message:             invalid spec: OnExit is invalid field in spec if workflow referred WorkflowTemplate reference

Steps to reproduce:

1. Install Argo Workflows 2.9.3
2. Edit the workflow controller configmap to add a default exit handler for all workflows:

data:
  workflowDefaults: |
    spec:
      onExit: whalesay-exit
      templates:
        - name: whalesay-exit
          container:
          image: docker/whalesay
          command: [cowsay]
          args: ["hello from the default exit handler"]

3. Create whalesay-template.yaml using the sample WorkflowTemplate from the Argo docs:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: workflow-template-submittable
spec:
  entrypoint: whalesay-template     # Fields other than "arguments" and "templates" not supported in v2.4 - v2.6
  arguments:
    parameters:
      - name: message
        value: hello world
  templates:
    - name: whalesay-template
      inputs:
        parameters:
          - name: message
      container:
        image: docker/whalesay
        command: [cowsay]
        args: ["{{inputs.parameters.message}}"]

4. Submit the WorkflowTemplate to your Argo cluster:

$ argo template create whalesay-template.yaml

5. Use the argo CLI to create and watch a Workflow from the WorkflowTemplate:

$ argo submit --from workflowtemplate/workflow-template-submittable --watch

Expected behavior: The Workflow is successfully started. (Note that it subsequently fails because the default service account has insufficient permissions, but that behavior is known and unrelated to this issue.)

Actual behavior: The Workflow fails immediately with the failure message shown above. In version 2.9.5, this issue reproduces with a slightly different failure message:

Status:              Failed
Message:             invalid spec: Templates is invalid field in spec if workflow referred WorkflowTemplate reference

This behavior does not reproduce in version 2.8.2.

Diagnostics

What version of Argo Workflows are you running?

2.9.3

Here's the resulting Workflow YAML:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  creationTimestamp: "2020-08-11T17:39:31Z"
  generateName: workflow-template-submittable-
  generation: 2
  labels:
    workflows.argoproj.io/completed: "true"
    workflows.argoproj.io/phase: Failed
  name: workflow-template-submittable-5shzj
  namespace: default
  resourceVersion: "31356"
  selfLink: /apis/argoproj.io/v1alpha1/namespaces/default/workflows/workflow-template-submittable-5shzj
  uid: 9e84c2db-6b92-4824-bd7c-347b401cba0b
spec:
  arguments: {}
  onExit: whalesay-exit
  templates: null
  workflowTemplateRef:
    name: workflow-template-submittable
status:
  conditions:
  - status: "True"
    type: Completed
  finishedAt: "2020-08-11T17:39:31Z"
  message: 'invalid spec: OnExit is invalid field in spec if workflow referred WorkflowTemplate
    reference'
  phase: Failed
  startedAt: "2020-08-11T17:39:31Z"
  storedTemplates:
    namespaced/workflow-template-submittable/whalesay-template:
      arguments: {}
      container:
        args:
        - '{{inputs.parameters.message}}'
        command:
        - cowsay
        image: docker/whalesay
        name: ""
        resources: {}
      inputs:
        parameters:
        - name: message
      metadata: {}
      name: whalesay-template
      outputs: {}
  storedWorkflowTemplateSpec:
    arguments:
      parameters:
      - name: message
        value: hello world
    entrypoint: whalesay-template
    templates:
    - arguments: {}
      container:
        args:
        - '{{inputs.parameters.message}}'
        command:
        - cowsay
        image: docker/whalesay
        name: ""
        resources: {}
      inputs:
        parameters:
        - name: message
      metadata: {}
      name: whalesay-template
      outputs: {}

Logs from the workflow controller:

time="2020-08-11T17:39:31Z" level=info msg="Processing workflow" namespace=default workflow=workflow-template-submittable-5shzj
time="2020-08-11T17:39:31Z" level=info msg="Updated phase  -> Running" namespace=default workflow=workflow-template-submittable-5shzj
time="2020-08-11T17:39:31Z" level=info msg="Updated phase Running -> Failed" namespace=default workflow=workflow-template-submittable-5shzj
time="2020-08-11T17:39:31Z" level=info msg="Updated message  -> invalid spec: OnExit is invalid field in spec if workflow referred WorkflowTemplate reference" namespace=default workflow=workflow-template-submittable-5shzj
time="2020-08-11T17:39:31Z" level=info msg="Marking workflow completed" namespace=default workflow=workflow-template-submittable-5shzj
time="2020-08-11T17:39:31Z" level=info msg="Checking daemoned children of " namespace=default workflow=workflow-template-submittable-5shzj
time="2020-08-11T17:39:31Z" level=info msg="Workflow update successful" namespace=default phase=Failed resourceVersion=31356 workflow=workflow-template-submittable-5shzj

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[sarabala1979]

looks like some OS-specific CLI YAML parsing issue. I can see templates: null on parsed workflow. Can you provide the your CLI version? you can get it using the argo version.

[mattskone]

argo: v2.9.3
  BuildDate: 2020-07-15T01:16:21Z
  GitCommit: 9407e19b3a1c61ad4043e382484fd0b6b15574f2
  GitTreeState: clean
  GitTag: v2.9.3
  GoVersion: go1.13.4
  Compiler: gc
  Platform: darwin/amd64

[sarabala1979]

@mattskone There was a bug in 2.8, workflow default will not be applied on submit --from #3355. This is a limitation in v2.9 submit -- from. Templates in workflow default will not be supported for submit -- from. The fix is a little bit complicated. I will talk to team and work on it.

[alexec]

Available for testing in v2.11.0-rc1.