How to handle Argo as an dependency for Kubeflow pipelines.

[NikeNano]

Summary

For Kubeflow pipelines Argo Workflows are used. During attempts to do validation of Argo Workflows in the backend of Kubeflow pipelines there are some issues with dependencies on com_github_colinmarc_hdfs that comes from having argo as a dependency.

For Kubeflow pipelines Bazel is used. To reproduce pull the branch from here: kubeflow/pipelines#3965

Then execute: bazel run //:gazelle -- update-repos --from_file=go.mod & bazel build --action_env=PATH --define=grpc_no_ares=true //backend/...

The received error message is:

Analysis of target '//backend/src/apiserver:apiserver' failed; build aborted: no such package '@com_github_colinmarc_hdfs//': failed to fetch com_github_colinmarc_hdfs: # cd /private/var/tmp/_bazel_niklashansson/b0db32648c4d9475e405b778807c3054/external/com_github_colinmarc_hdfs; git checkout v1.1.4-0.20180805212432-9746310a4d31
error: pathspec 'v1.1.4-0.20180805212432-9746310a4d31' did not match any file(s) known to git
2020/08/18 16:32:04 exit status 1

How could this dependency be handle correctly? I am rather new to both golang and Bazel so sorry if I am missing something obviously.

Thank you!

Motivation

Why do you need to know this, any examples or use cases you could include?

[alexec]

The validate library transitively imports this HDFS dependency:

	github.com/colinmarc/hdfs v1.1.4-0.20180805212432-9746310a4d31

[alexec]

That tag does not exist, try pinning to v1.13:

go get github.com/colinmarc/hdfs@v1.1.3

[NikeNano]

Thanks @alexec. When i pin to 1.13 I get the following issues when I run the test for the validation argo workflows in the Kubeflow backend.

 ../../../../../../../../pkg/mod/github.com/argoproj/argo@v2.3.0+incompatible/workflow/artifacts/hdfs/util.go:21:10: options.KerberosClient undefined (type hdfs.ClientOptions has no field or method KerberosClient)
../../../../../../../../pkg/mod/github.com/argoproj/argo@v2.3.0+incompatible/workflow/artifacts/hdfs/util.go:22:10: options.KerberosServicePrincipleName undefined (type hdfs.ClientOptions has no field or method KerberosServicePrincipleName)

Link to test.

The test works when I have github.com/colinmarc/hdfs v1.1.4-0.20180805212432-9746310a4d31, which confuses me a bit ....

[NikeNano]

Get the same error as #1983 if i pin to v1.1.3 issues discuss but don't get it to work with their solution either.

[NikeNano]

Do you know more about the issue @alexec, thanks for your help!

[alexec]

hi @NikeNano I'm afraid I don't know how to fix this issue. The is Go modules issue rather than an Argo issue. You'll know if you've tried to import the Kubernetes dependencies Go modules is not the seamless experience of Maven.

[NikeNano]

Thanks for the help :)

You'll know if you've tried to import the Kubernetes dependencies Go modules is not the seamless experience of Maven.

I am starting to learn that now .... :(

[dtzar]

I believe #3991 would help the overall situation.
Also @NikeNano per kubeflow/pipelines#4477 you shouldn't be using Bazel which also can cause problems.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.