Error in retrieving list of pipelines (KubeFlow v0.6.1)

[kimchitsigai]

What happened:
I've installed kubeflow v0.6.1, with kfctl_existing_arrikto.yaml.
From the KubeFlow Central Dashboard, I click on Pipelines and I get the error message + Details, shown in the attached screenshot.

What did you expect to happen:
I expect an empty list of pipelines or the list of samples, being displayed without error message.

What steps did you take:

Anything else you would like to add:
I've also attached the output from kubectl describe the pipeline-related pods.

pipeline_log.txt

[kimchitsigai]

I'm adding the outputs from kubectl logs for the pipeline related pods.
There seems to be something wrong with ml-pipeline:
Failed to commit transaction to initialize default experiment table

pipeline_log2.txt

[rmgogogo]

I also got few times same issue on it. On our radar now.

-- please ignore following --
b/144082994

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

[akshayuppal3]

Having the same issue ..

[akshayuppal3]

had to do with low on disk space..

[ReggieCarey]

this is still a bug. Stop closing it.

[Bobgy]

/reopen

Can you provide more details and error message?

[k8s-ci-robot]

@Bobgy: Reopened this issue.

In response to this:

/reopen

Can you provide more details and error message?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[google-oss-robot]

@Bobgy: Reopened this issue.

In response to this:

/reopen

Can you provide more details and error message?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[bvboca]

have the same issue on pipeline Version: 1.0.4
{"error":"Invalid input error: Invalid resource references for experiment. ListExperiment requires filtering by namespace.","message":"Invalid input error: Invalid resource references for experiment. ListExperiment requires filtering by namespace.","code":3,"details":[{"@type":"type.googleapis.com/api.Error","error_message":"Invalid resource references for experiment. ListExperiment requires filtering by namespace.","error_details":"Invalid input error: Invalid resource references for experiment. ListExperiment requires filtering by namespace."}]}

[ReggieCarey]

Can I suggest that the volunteers supporting this product stop ignoring this glaring issue and make it priority no. 1. We are now at Kubeflow 1.2.0 and this is still broken and it is the core functionality of Kubeflow.

See issue #5223 for my details on this bug.

[bvboca]

I did a fresh install on a clean kubernetes 1.16. The issue is not seem for now. Before this, I install kubeflow on an existing istio cluster and skip some steps. Maybe some functions are broken in my installation.

[Bobgy]

Kubeflow 1.2 on dex is not supported by any party participating, I'd suggest waiting for 1.3 where @yanniszark said they will maintain.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[santhosh261190]

This is due to the race conditions while creating the resources. In my case, once after all the resources are up and running i did executed "kubectl delete namespace kubeflow", this will re-create and possibly the issue will go away.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[DeepFlame-JR]

In my case, the error message was "Error: failed to retrieve list of pipelines" and detail message "No healthy upstream".
And I got the logs in pod of ml-pipeline-xxxx

I0329 08:27:53.106414       6 client_manager.go:160] Initializing client manager
I0329 08:27:53.106781       6 config.go:57] Config DBConfig.ExtraParams not specified, skipping
[mysql] 2020/12/04 00:28:19 packets.go:36: unexpected EOF
[mysql] 2020/12/04 00:28:23 packets.go:36: unexpected EOF
[mysql] 2020/12/04 00:28:26 packets.go:36: unexpected EOF
[mysql] 2020/12/04 00:28:32 packets.go:36: unexpected EOF
[mysql] 2020/12/04 00:28:38 packets.go:36: unexpected EOF

I solved this problem by just commading kubectl delete po ml-pipeline-xxx
Then new pod will be created by ml-pipeline deployment.