Api-service container failing with panic: runtime error: invalid memory address or nil pointer dereference

[barambani]

What happened:
Api-service container failing with the log below

I0210 10:50:05.291958       1 interceptor.go:29] /api.RunService/CreateRun handler starting
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x138 pc=0x11a6ed6]
goroutine 272 [running]:
github.com/kubeflow/pipelines/backend/src/common/util.(*Workflow).VerifyParameters(0xc0000c1e10, 0xc000e42780, 0x4, 0xc000e42780)
        backend/src/common/util/workflow.go:65 +0x96
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).CreateRun(0xc000871860, 0xc000eca270, 0x0, 0x0, 0x
24bb800)
        backend/src/apiserver/resource/resource_manager.go:224 +0x1cf
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).CreateRun(0xc0000c0078, 0x191ca40, 0xc00021f860, 0xc00021f
890, 0xc0000c0078, 0x1, 0x1)
        backend/src/apiserver/server/run_server.go:35 +0x6a
github.com/kubeflow/pipelines/backend/api/go_client._RunService_CreateRun_Handler.func1(0x191ca40, 0xc00021f860, 0x1679080, 0xc000
21f890, 0x1, 0x0, 0xc000060700, 0x0)
        bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/go_client_go_proto%/github.com/kubeflow/pipelines/backend/api/go_cli
ent/run.pb.go:1269 +0x86
main.apiServerInterceptor(0x191ca40, 0xc00021f860, 0x1679080, 0xc00021f890, 0xc000d944e0, 0xc000d94500, 0x1589980, 0x2574b90, 0x16
fe160, 0xc000aa8b00)
        backend/src/apiserver/interceptor.go:30 +0xf4
github.com/kubeflow/pipelines/backend/api/go_client._RunService_CreateRun_Handler(0x168d080, 0xc0000c0078, 0x191ca40, 0xc00021f860
, 0xc000d36af0, 0x17e55a8, 0x0, 0x0, 0xc000a809c0, 0x188)
        bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/go_client_go_proto%/github.com/kubeflow/pipelines/backend/api/go_cli
ent/run.pb.go:1271 +0x158
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00002a300, 0x19296c0, 0xc0008fc600, 0xc000aa8b00, 0xc00046faa0, 0x24efd20, 0x0
, 0x0, 0x0)
        external/org_golang_google_grpc/server.go:966 +0x4a2
google.golang.org/grpc.(*Server).handleStream(0xc00002a300, 0x19296c0, 0xc0008fc600, 0xc000aa8b00, 0x0)
        external/org_golang_google_grpc/server.go:1245 +0xd61
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000351500, 0xc00002a300, 0x19296c0, 0xc0008fc600, 0xc000aa8b00)
        external/org_golang_google_grpc/server.go:685 +0x9f
created by google.golang.org/grpc.(*Server).serveStreams.func1
        external/org_golang_google_grpc/server.go:683 +0xa1

What did you expect to happen:

What steps did you take:
Tried to run an existing pipeline.

Anything else you would like to add:

[numerology]

Hi @barambani do you mind sharing the following info:

1. Version of KFP deployment
2. Version of KFP SDK
3. If possible can you share the your pipeline definition as well? Thanks

[AndriiNeverov]

@numerology

Seeing this while moving from KF 1.0.0 to 1.0.2:

• ran kubectl delete deploy ml-pipeline minio mysql ml-pipeline-persistenceagent ml-pipeline-ui ml-pipeline-viewer-controller-deployment ml-pipeline-scheduledworkflow ml-pipeline-ml-pipeline-visualizationserver
• applied 1.0.2 manifests on top
• tried to run "[Sample] Basic - Parallel execution"

1.0.2 manifests contain changes in pipeline image versions (0.2.0 -> 0.2.5).

If I also delete the PVC, the old pipelines are gone, I see the new ones and the issue doesn't happen. So the old pipelines doesn't seem to be upgraded properly, is that right?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

[valiantljk]

still seeing this issue.

[Bobgy]

/reopen

[k8s-ci-robot]

@Bobgy: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[wstian]

same issue.
In my case, it's because minio service started with invalid /data. so the pipeline-apiserver fail to get pipeline definition.

[Bobgy]

@wstian @valiantljk Can you report your KFP version?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.