[pipelines-in-pipelines] SIGSEGV in controller if parent PipelineRun is cancelled while child PipelineRun is starting

[emenaceb]

Expected Behavior

When a Run is changed before its child PipelineRun has started, this change should be ignored if it isn't handled.

Actual Behavior

Pip controller crashes (SEGV signal received) and restart once an again until child PipelineRun ends.

Then, when it ends, pip controller starts normally.

Log:

{"level":"info","ts":1650470954.805556,"logger":"pip-controller.github.com-tektoncd-experimental-pipelines-in-pipelines-pkg-reconciler-pip.Reconciler","caller":"pip/piprun.go:154","msg":"PipelineRun created by Run ci-builds/testpip-wcqgm-echo is still running","knative.dev/traceid":"0add69f4-14aa-46c8-a20c-243135efa84f","knative.dev/key":"ci-builds/testpip-wcqgm-echo","logging.googleapis.com/labels":{},"logging.googleapis.com/sourceLocation":{"file":"github.com/tektoncd/experimental/pipelines-in-pipelines/pkg/reconciler/pip/piprun.go","line":"154","function":"github.com/tektoncd/experimental/pipelines-in-pipelines/pkg/reconciler/pip.updateRunStatus"}}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0x14e5b0d]

goroutine 130 [running]:
github.com/tektoncd/experimental/pipelines-in-pipelines/pkg/reconciler/pip.updateRunStatus({0x1c78c78, 0xc000b02ff0}, 0xc0005f2400, 0xc000594c80)
        github.com/tektoncd/experimental/pipelines-in-pipelines/pkg/reconciler/pip/piprun.go:155 +0x4ed
github.com/tektoncd/experimental/pipelines-in-pipelines/pkg/reconciler/pip.(*Reconciler).reconcile(0x19a7e91, {0x1c78c78, 0xc000b02ff0}, 0xc0005f2400)
        github.com/tektoncd/experimental/pipelines-in-pipelines/pkg/reconciler/pip/piprun.go:129 +0x279
github.com/tektoncd/experimental/pipelines-in-pipelines/pkg/reconciler/pip.(*Reconciler).ReconcileKind(0x0, {0x1c78c78, 0xc000b02ff0}, 0xc0005f2400)
        github.com/tektoncd/experimental/pipelines-in-pipelines/pkg/reconciler/pip/piprun.go:98 +0x9f2
github.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1alpha1/run.(*reconcilerImpl).Reconcile(0xc000534dc0, {0x1c78c78, 0xc000b02fc0}, {0xc000c86fc0, 0x1c})
        github.com/tektoncd/pipeline@v0.20.0/pkg/client/injection/reconciler/pipeline/v1alpha1/run/reconciler.go:241 +0x6ed
knative.dev/pkg/controller.(*Impl).processNextWorkItem(0xc00057dd00)
        knative.dev/pkg@v0.0.0-20210114223020-f0ea5e6b9c4e/controller/controller.go:523 +0x5d5
knative.dev/pkg/controller.(*Impl).RunContext.func3()
        knative.dev/pkg@v0.0.0-20210114223020-f0ea5e6b9c4e/controller/controller.go:461 +0x68
created by knative.dev/pkg/controller.(*Impl).RunContext
        knative.dev/pkg@v0.0.0-20210114223020-f0ea5e6b9c4e/controller/controller.go:459 +0x2de

Pod changes:

NAMESPACE         NAME                                                 READY   STATUS      RESTARTS   AGE
tekton-pip-run    pip-controller-597c4c5b85-g44hm                      0/1     CrashLoopBackOff   1          15m

NAMESPACE         NAME                                                 READY   STATUS      RESTARTS   AGE
tekton-pip-run    pip-controller-597c4c5b85-g44hm                      1/1     Running     2          15m

Steps to Reproduce the Problem

1. Add following Task

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: pip-test-task
spec:
  params:
  - name: toecho
    description: Something to be echoed
  workspaces:
    - name: ws
  steps:
  - name: echo
    image: 'ubuntu'
    workingDir: $(workspaces.ws.path)
    script: |
      #!/bin/bash

      VAL="$(params.toecho)"
      if [[ "$VAL" == "error" ]] ; then
        echo "Error"
        exit -1     
      elif [[ "$VAL" == "sleep" ]] ; then
        echo "Sleeping"
        sleep 30
        echo "Wake up"
      elif [[ "$VAL" == "ls" ]] ; then
        echo "Listing directory"
        ls
        echo "Done"
      else
        echo "Printing $VAL"
      fi 

2. Add following Pipeline

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: pip-test-pipeline
spec:

  params:
    - name: value
      description: Value to be echoed

  workspaces:
    - name: ws
    
  tasks:
      - name: do-echo
        params:
          - name: toecho
            value: $(params.value)
        workspaces:
          - name: ws
            workspace: ws
        taskRef:
          name: pip-test-task

3. Create following PipelineRun

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  generateName: "testpip-"
spec:
  serviceAccountName: tekton-builds-service-account
  params:
    - name: value
      value: "I want to use pip"
  workspaces:
    - name: ws
      volumeClaimTemplate:
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1Gi

  podTemplate: {}
  timeout: 1h0m0s

  pipelineSpec:
    params:
      - name: value
        description: Value to echo
        default: "Hello World"
    workspaces:
      - name: ws
    tasks:

      - name: echo
        taskRef:
          apiVersion: tekton.dev/v1beta1
          kind: Pipeline
          name: pip-test-pipeline
        params:
          - name: value
            value: $(params.value)
        workspaces:
          - name: ws
            workspace: ws

4. While child PipelineRun is created but before it starts, cancel it. (You can cancel it using Dashboard)

5. Then Pip controller crashes

Additional Info

• Kubernetes version:

  Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:58:47Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1+k3s1", GitCommit:"75dba57f9b1de3ec0403b148c52c348e1dee2a5e", GitTreeState:"clean", BuildDate:"2021-05-21T16:12:29Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

• Tekton Pipeline version:

  Output of tkn version or kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'

v0.33.2

[jerop]

promoting it to the tekton pipelines API, so closing

tektoncd/community#701