Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Backend] Fix report workflow error messages #2815

Merged
merged 2 commits into from
Jan 9, 2020

Conversation

Bobgy
Copy link
Contributor

@Bobgy Bobgy commented Jan 8, 2020

/assign @gaoning777
/kind bug

There are two different problems:

error log 1
I0107 09:03:28.062672       1 interceptor.go:29] /api.ReportService/ReportWorkflow handler starting
I0107 09:03:28.103541       1 error.go:218] Failed to update run
github.com/kubeflow/pipelines/backend/src/apiserver/storage.(*RunStore).UpdateRun
	backend/src/apiserver/storage/run_store.go:405
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).ReportWorkflowResource
	backend/src/apiserver/resource/resource_manager.go:613
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ReportServer).ReportWorkflow
	backend/src/apiserver/server/report_server.go:39
github.com/kubeflow/pipelines/backend/api/go_client._ReportService_ReportWorkflow_Handler.func1
	bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/go_client_go_proto%/github.com/kubeflow/pipelines/backend/api/go_client/report.pb.go:174
main.apiServerInterceptor
	backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/go_client._ReportService_ReportWorkflow_Handler
	bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/go_client_go_proto%/github.com/kubeflow/pipelines/backend/api/go_client/report.pb.go:176
google.golang.org/grpc.(*Server).processUnaryRPC
	external/org_golang_google_grpc/server.go:966
google.golang.org/grpc.(*Server).handleStream
	external/org_golang_google_grpc/server.go:1245
google.golang.org/grpc.(*Server).serveStreams.func1.1
	external/org_golang_google_grpc/server.go:685
runtime.goexit
	GOROOT/src/runtime/asm_amd64.s:1333
InternalServerError: Failed to update run 6fd49a99-f198-495f-b91b-6614f3e45079. Row not found
github.com/kubeflow/pipelines/backend/src/common/util.NewInternalServerError
	backend/src/common/util/error.go:142
github.com/kubeflow/pipelines/backend/src/apiserver/storage.(*RunStore).UpdateRun
	backend/src/apiserver/storage/run_store.go:405
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).ReportWorkflowResource
	backend/src/apiserver/resource/resource_manager.go:613
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ReportServer).ReportWorkflow
	backend/src/apiserver/server/report_server.go:39
github.com/kubeflow/pipelines/backend/api/go_client._ReportService_ReportWorkflow_Handler.func1
	bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/go_client_go_proto%/github.com/kubeflow/pipelines/backend/api/go_client/report.pb.go:174
main.apiServerInterceptor
	backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/go_client._ReportService_ReportWorkflow_Handler
	bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/go_client_go_proto%/github.com/kubeflow/pipelines/backend/api/go_client/report.pb.go:176
google.golang.org/grpc.(*Server).processUnaryRPC
	external/org_golang_google_grpc/server.go:966
google.golang.org/grpc.(*Server).handleStream
	external/org_golang_google_grpc/server.go:1245
google.golang.org/grpc.(*Server).serveStreams.func1.1
	external/org_golang_google_grpc/server.go:685
runtime.goexit
	GOROOT/src/runtime/asm_amd64.s:1333
Failed to update the run.
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrap
	backend/src/common/util/error.go:211
github.com/kubeflow/pipelines/backend/src/common/util.Wrap
	backend/src/common/util/error.go:244
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).ReportWorkflowResource
	backend/src/apiserver/resource/resource_manager.go:615
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ReportServer).ReportWorkflow
	backend/src/apiserver/server/report_server.go:39
github.com/kubeflow/pipelines/backend/api/go_client._ReportService_ReportWorkflow_Handler.func1
	bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/go_client_go_proto%/github.com/kubeflow/pipelines/backend/api/go_client/report.pb.go:174
main.apiServerInterceptor
	backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/go_client._ReportService_ReportWorkflow_Handler
	bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/go_client_go_proto%/github.com/kubeflow/pipelines/backend/api/go_client/report.pb.go:176
google.golang.org/grpc.(*Server).processUnaryRPC
	external/org_golang_google_grpc/server.go:966
google.golang.org/grpc.(*Server).handleStream
	external/org_golang_google_grpc/server.go:1245
google.golang.org/grpc.(*Server).serveStreams.func1.1
	external/org_golang_google_grpc/server.go:685
runtime.goexit
	GOROOT/src/runtime/asm_amd64.s:1333
Report workflow failed.
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrap
	backend/src/common/util/error.go:211
github.com/kubeflow/pipelines/backend/src/common/util.Wrap
	backend/src/common/util/error.go:244
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ReportServer).ReportWorkflow
	backend/src/apiserver/server/report_server.go:41
github.com/kubeflow/pipelines/backend/api/go_client._ReportService_ReportWorkflow_Handler.func1
	bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/go_client_go_proto%/github.com/kubeflow/pipelines/backend/api/go_client/report.pb.go:174
main.apiServerInterceptor
	backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/go_client._ReportService_ReportWorkflow_Handler
	bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/go_client_go_proto%/github.com/kubeflow/pipelines/backend/api/go_client/report.pb.go:176
google.golang.org/grpc.(*Server).processUnaryRPC
	external/org_golang_google_grpc/server.go:966
google.golang.org/grpc.(*Server).handleStream
	external/org_golang_google_grpc/server.go:1245
google.golang.org/grpc.(*Server).serveStreams.func1.1
	external/org_golang_google_grpc/server.go:685
runtime.goexit
	GOROOT/src/runtime/asm_amd64.s:1333
/api.ReportService/ReportWorkflow call failed
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrapf
	backend/src/common/util/error.go:206
github.com/kubeflow/pipelines/backend/src/common/util.Wrapf
	backend/src/common/util/error.go:231
main.apiServerInterceptor
	backend/src/apiserver/interceptor.go:32
github.com/kubeflow/pipelines/backend/api/go_client._ReportService_ReportWorkflow_Handler
	bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/go_client_go_proto%/github.com/kubeflow/pipelines/backend/api/go_client/report.pb.go:176
google.golang.org/grpc.(*Server).processUnaryRPC
	external/org_golang_google_grpc/server.go:966
google.golang.org/grpc.(*Server).handleStream
	external/org_golang_google_grpc/server.go:1245
google.golang.org/grpc.(*Server).serveStreams.func1.1
	external/org_golang_google_grpc/server.go:685
runtime.goexit
	GOROOT/src/runtime/asm_amd64.s:1333
error log 2
I0107 10:55:38.115884       1 interceptor.go:29] /api.ReportService/ReportWorkflow handler starting
E0107 10:55:38.330257       1 error.go:256] InternalError: workflows.argoproj.io "conditional-execution-pipeline-x4v6v" not found
Failed to add PersistedFinalState label to workflow
github.com/kubeflow/pipelines/backend/src/common/util.Wrap
        backend/src/common/util/error.go:246
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).ReportWorkflowResource
        backend/src/apiserver/resource/resource_manager.go:673
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ReportServer).ReportWorkflow
        backend/src/apiserver/server/report_server.go:39
github.com/kubeflow/pipelines/backend/api/go_client._ReportService_ReportWorkflow_Handler.func1
        bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/go_client_go_proto%/github.com/kubeflow/pipelines/backend/api/go_client/report.pb.go:174
main.apiServerInterceptor
        backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/go_client._ReportService_ReportWorkflow_Handler
        bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/go_client_go_proto%/github.com/kubeflow/pipelines/backend/api/go_client/report.pb.go:176
google.golang.org/grpc.(*Server).processUnaryRPC
        external/org_golang_google_grpc/server.go:966
google.golang.org/grpc.(*Server).handleStream
        external/org_golang_google_grpc/server.go:1245
google.golang.org/grpc.(*Server).serveStreams.func1.1
        external/org_golang_google_grpc/server.go:685
runtime.goexit
        GOROOT/src/runtime/asm_amd64.s:1333
Report workflow failed.
github.com/kubeflow/pipelines/backend/src/common/util.Wrap
        backend/src/common/util/error.go:246
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ReportServer).ReportWorkflow
        backend/src/apiserver/server/report_server.go:41
github.com/kubeflow/pipelines/backend/api/go_client._ReportService_ReportWorkflow_Handler.func1
        bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/go_client_go_proto%/github.com/kubeflow/pipelines/backend/api/go_client/report.pb.go:174
main.apiServerInterceptor
        backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/go_client._ReportService_ReportWorkflow_Handler
        bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/go_client_go_proto%/github.com/kubeflow/pipelines/backend/api/go_client/report.pb.go:176
google.golang.org/grpc.(*Server).processUnaryRPC
        external/org_golang_google_grpc/server.go:966
google.golang.org/grpc.(*Server).handleStream
        external/org_golang_google_grpc/server.go:1245
google.golang.org/grpc.(*Server).serveStreams.func1.1
        external/org_golang_google_grpc/server.go:685
runtime.goexit
        GOROOT/src/runtime/asm_amd64.s:1333
/api.ReportService/ReportWorkflow call failed
github.com/kubeflow/pipelines/backend/src/common/util.Wrapf
        backend/src/common/util/error.go:233
main.apiServerInterceptor
        backend/src/apiserver/interceptor.go:32
github.com/kubeflow/pipelines/backend/api/go_client._ReportService_ReportWorkflow_Handler
        bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/go_client_go_proto%/github.com/kubeflow/pipelines/backend/api/go_client/report.pb.go:176
google.golang.org/grpc.(*Server).processUnaryRPC
        external/org_golang_google_grpc/server.go:966
google.golang.org/grpc.(*Server).handleStream
        external/org_golang_google_grpc/server.go:1245
google.golang.org/grpc.(*Server).serveStreams.func1.1
        external/org_golang_google_grpc/server.go:685
runtime.goexit
        GOROOT/src/runtime/asm_amd64.s:1333

This change is Reviewable

@IronPan
Copy link
Member

IronPan commented Jan 9, 2020

/lgtm
/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: IronPan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 41a1da2 into kubeflow:master Jan 9, 2020
@Bobgy Bobgy changed the title [Backend] Report workflow should use its namespace [Backend] Fix report workflow error messages Jan 9, 2020
@Bobgy Bobgy deleted the add-ns-in-report-workflow branch January 9, 2020 00:21
@gaoning777
Copy link
Contributor

Thanks Yuan for the fix.
/lgtm

rui5i pushed a commit to rui5i/pipelines that referenced this pull request Jan 16, 2020
* Report workflow should use its namespace

* Use rows matched instead of rows affected for sql update
Jeffwan pushed a commit to Jeffwan/pipelines that referenced this pull request Dec 9, 2020
* Report workflow should use its namespace

* Use rows matched instead of rows affected for sql update
magdalenakuhn17 pushed a commit to magdalenakuhn17/pipelines that referenced this pull request Oct 22, 2023
Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants