Unable to connect to private helm OCI repo x509: certificate signed by unknown authority

[rogfut]

If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argocd slack channel.

Checklist:

• [ x ] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [ x ] I've included steps to reproduce the bug.
• [ x ] I've pasted the output of argocd version.

Describe the bug

originally logged in #6599

ArgoCD is unable to connect to my private Harbor Helm OCI repository with self-signed certificate. It is able to connect to another non-OCI helm repository on the same harbor instance with the same certificate and credentials.

To Reproduce

1. exec into the argocd-server pod
   k exec -it argocd-server-5cd74676dc-78w5n -n argocd -- /bin/bash

2. Verify the cert exists and has the correct data
   cat /app/config/tls/oci.example.com

3. verify with openssl that the cert is valid when connecting to repo
   openssl s_client -CAfile /app/config/tls/oci.example.com oci.example.com:443

 ...
 Verify return code: 0 (ok)
 ...

4. helm registry login fails with x509 certificate error

export HELM_EXPERIMENTAL_OCI=1
helm registry login -u username oci.example.com/chartrepo/test

INFO[0006] Error logging in to v2 endpoint, trying next endpoint: Get "https://oci.example.com/chartrepo/test/v2/": x509: certificate signed by unknown authority
Error: Get "https://oci.example.com/chartrepo/test/v2/": x509: certificate signed by unknown authority

5. try this on the helm client on my local machine and it works. Note that my local machine has the same certificate installed to /etc/ssl/certs

export HELM_EXPERIMENTAL_OCI=1
helm registry login -u username oci.example.com/chartrepo/test
Password:
Login succeeded

Expected behavior

I would expect that somehow argocd-server would use the certificate installed at /app/config/tls/oci.example.com when running helm registry login. Since there is no --ca-file option for helm registry login, the way I have achieved this in my local environment is to add the self-signed certificate to /etc/ssl/certs. I don't see a way to do this with argocd-server. Should argocd-server be installing the certificate to /etc/ssl/certs?

More Info

For reference, I have my self-signed ca cert installed properly per the documentation

Version

argocd: v2.0.5+4c94d88.dirty
  BuildDate: 2021-07-25T14:45:00Z
  GitCommit: 4c94d886f56bcb2f9d5b3251fdc049c2d1354b88
  GitTreeState: dirty
  GoVersion: go1.16.6
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.1.2+7af9dfb
  BuildDate: 2021-09-02T18:05:23Z
  GitCommit: 7af9dfb3524c13e941ab604e36e49a617fe47d2e
  GitTreeState: clean
  GoVersion: go1.16.5
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v4.2.0 2021-06-30T22:49:26Z
  Helm Version: v3.6.0+g7f2df64
  Kubectl Version: v0.21.0
  Jsonnet Version: v0.17.0

Logs

Output from `argocd repo add oci.example.com/chartrepo/test --type helm --name test --enable-oci --username username

time="2021-09-16T05:47:13Z" level=info msg="Error logging in to v2 endpoint, trying next endpoint: Get \"https://oci.example.com/v2/\": x509: certificate signed by unknown authority"
Error: Get "https://oci.example.com/v2/": x509: certificate signed by unknown authority

[rogfut]

Someone in the slack channel had mentioned that this isn't a bug but instead I need to install the cert into /etc/ssl/certs via configmap or secret in argocd-server, but I'm struggling to find the documentation on how to do so. Any cert I install as a secret goes to /app/config/tls but does not get added to /etc/ssl/certs or /etc/ssl/certs/ca-certificates.crt.

I followed This document

Is there documentation on adding a self-signed certificate to /etc/ssl/certs for the argocd-server pod?

[rogfut]

Closing this as an environment issue.

In case anyone finds this, the solution I found was to use a configMap to mount the self-signed certificate to /etc/ssl/certs on the argocd-server pod. I did this by patching the argocd-server deployment in the argocd provided install.yaml.

Here is the patch.yaml I made to the Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-server
spec:
  template:
    spec:
      containers:
      - name: argocd-server
        volumeMounts:
        - name: harbor-cert-cm
          mountPath: /etc/ssl/certs/harbor-cert.pem
          subPath: harbor-cert.pem
          readOnly: false
      volumes:
      - name: harbor-cert-cm
        configMap:
          name: harbor-cert-cm
          items:
          - key: harbor-ca.pem
            path: harbor-cert.pem

[mmerickel]

I disagree that this should be closed. The CA cert is configured in argo the way they document - but it doesn't apply to OCI repositories. This should be considered a bug.

[mmerickel]

Using the snippets above, I had to mount the certificate on both argo-server and argo-repo-server before I was able to get an OCI repository to sync.