argocd repo add fails when using --enable-oci with self-signed certificate

[rogfut]

If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argocd slack channel.

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

argocd repo add fails with X509 certificate error when adding a private repo with self signed cert and using --enable-oci flag. Attempted workaround to use --insecure-skip-server-verification works to add the registry, but argocd app create fails with X509 certificate error.

To Reproduce

1. Add the ca.crt to argo's cert list
   argocd cert tls-add --from ~/ca.pem helmrepo.rogerfutrell.com

2. Add the helm repo
   argocd repo add helmrepo.rogerfutrell.com --type helm --name test-helm --enable-oci --username <username> --password <password>

3. argocd repo add throws x509: certificate signed by unknown authority

Attempted Workaround

1. Add the helm repo with the insecure flag
   argocd repo add helmrepo.rogerfutrell.com --type helm --name test-helm --enable-oci --username <username> --password <password> --insecure-skip-server-verification

2. argocd repo list -o yaml indicates a successful connection

- connectionState:
    attemptedAt: "2021-06-30T01:00:11Z"
    message: ""
    status: Successful
  enableOCI: true
  insecure: true
  name: test-helm
  repo: helmrepo.rogerfutrell.com
  type: helm
  username: <username>

3. create the argocd app via UI or via cli

argocd app create myapp --repo helmrepo.rogerfutrell.com --helm-chart test-helm/myapp --revision 0.1.2-bdebug --dest-namespace myapp --dest-server https://kubernetes.default.svc --insecure

FATA[0000] rpc error: code = InvalidArgument desc = application spec is invalid: InvalidSpecError: Unable to get app details: rpc error: code = Unknown desc = `helm chart pull helmrepo.rogerfutrell.com/test-helm/myapp:0.1.2-bdebug` failed exit status 1: Error: failed to do request: Head "https://helmrepo.rogerfutrell.com/v2/test-helm/myapp/manifests/0.1.2-bdebug": x509: certificate signed by unknown authority

Expected behavior

1. argocd-server pod should be using the ca certificate helmrepo.rogerfutrell.com that is mounted to the pod when performing helm registry login

2. argocd create app should be using the ca certificate helmrepo.rogerfutrell.com that is mounted to the pod when performing helm chart pull
   argocd is able to execute the helm chart pull and ignores x509 errors because the repo was added with --insecure-skip-server-verification.

Version

argocd: v2.0.3+8d2b13d.dirty
  BuildDate: 2021-05-27T19:54:02Z
  GitCommit: 8d2b13d733e1dff7d1ad2c110ed31be4804406e2
  GitTreeState: dirty
  GoVersion: go1.16.4
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.0.4+0842d44
  BuildDate: 2021-06-23T01:27:53Z
  GitCommit: 0842d448107eb1397b251e63ec4d4bc1b4efdd6e
  GitTreeState: clean
  GoVersion: go1.16
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v3.9.4 2021-02-09T19:22:10Z
  Helm Version: v3.5.1+g32c2223
  Kubectl Version: v0.20.4
  Jsonnet Version: v0.17.0

Logs

argocd repo add helmrepo.rogerfutrell.com --type helm --name test-helm --enable-oci --username <username> --password <pass>

FATA[0000] rpc error: code = Unknown desc = `helm registry login helmrepo.rogerfutrell.com --username ****** --password ******` failed exit status 1: WARNING: Using --password ****** the CLI is insecure. Use --password-stdin.
time="2021-06-30T03:41:25Z" level=info msg="Error logging in to v2 endpoint, trying next endpoint: Get \"https://helmrepo.rogerfutrell.com/v2/\": x509: certificate signed by unknown authority"
Error: Get "https://helmrepo.rogerfutrell.com/v2/": x509: certificate signed by unknown authority

 argocd app create myapp --repo helmrepo.rogerfutrell.com --helm-chart test-helm/myapp --revision 0.1.2-bdebug --dest-namespace myapp --dest-server https://kubernetes.default.svc --insecure

 FATA[0000] rpc error: code = InvalidArgument desc = application spec is invalid: InvalidSpecError: Unable to get app details: rpc error: code = Unknown desc = `helm chart pull helmrepo.rogerfutrell.com/test-helm/myapp:0.1.2-bdebug` failed exit status 1: Error: failed to do request: Head "https://helmrepo.rogerfutrell.com/v2/test-helm/myapp/manifests/0.1.2-bdebug": x509: certificate signed by unknown authority

More Info

This same ca cert works with my private registry when I am not using --enable-oci. Issue seems specific to --enable-oci.

This works in helm in my laptop's bash shell, and helm is using my ca-cert in /etc/ssl/certs as expected:

export HELM_EXPERIMENTAL_OCI=1

helm registry login helmrepo.rogerfutrell.com --username <username> --password <password>
WARNING: Using --password via the CLI is insecure. Use --password-stdin.
Login succeeded

helm chart pull helmrepo.rogerfutrell.com/test-helm/myapp:0.1.2-bdebug
0.1.2-bdebug: Pulling from helmrepo.rogerfutrell.com/test-helm/myapp
ref:     helmrepo.rogerfutrell.com/test-helm/myapp:0.1.2-bdebug
digest:  c9bf173e4a564ec87ba1ac80aa5beab381ab2db48566d6bc038127d323686344
size:    2.7 KiB
name:    myapp
version: 0.1.2-bdebug
Status: Chart is up to date for helmrepo.rogerfutrell.com/test-helm/myapp:0.1.2-bdebug

Helm version

version.BuildInfo{Version:"v3.5.3", GitCommit:"041ce5a2c17a58be0fcd5f5e16fb3e7e95fea622", GitTreeState:"dirty", GoVersion:"go1.15.8"}

[alexmt]

Fixed by #6458

Thank you to @elucidator