x509: certificate signed by unknown authority

[zibuyule]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug
I deployed a self built private Harbor and have configured a self signed TLS certificate for it.

I configured a repo for argocd using the parameter " --insecure-skip-server-verification", and it appears to be successfully connected.

The command is as follows:

argocd repo add harbor.sredevops.me --type helm --name will-harbor-helm --enable-oci --username admin --insecure-skip-server-verification

The chart in my Harbor is as follows:

I have created an application as follows:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: helm-nginx-harbor
  namespace: argocd
spec:
  project: default
  source:
    chart: helm/nginx
    repoURL: harbor.sredevops.me
    targetRevision: "0.24"
    helm:
      releaseName: my-nginx
  destination:
    server: "https://kubernetes.default.svc"
    namespace: helm-app
  syncPolicy:
    syncOptions:
      - CreateNamespace=true

The Configmap also looks normal.

I always thought that if the repo connection was successful, the application should not show 'x509: certificate signed by unknown authority'.

To Reproduce

Expected behavior

The expected behavior is that the application can proceed smoothly. Ultimately, a chart named nginx will be installed.

Screenshots

Version
argocd： 2.72

**Logs**

Paste any relevant application logs here.

[dan-m8t]

#12371

[augeivv]

We have the problem as well and cannot continue without the feature.
We use the current version 2.8.4.

Is there a solution for this?

[dromie]

@augeivv
This issue is "fixed" and your v2.8.4 should contain the fix, BUT.......

when you update your argocd-tls-certs-cm configmap with the TLS certificate, this configmap is mounted into argocd-server pod.
And it can take up two minutes from you updating your configmap update till that update gets mounted into the argocd-server pod. k8s-doc
And argo-cd won't refresh your application automatically, you need to hit the refresh button.

Option B: update your argocd-tls-certs-cm configmap, wait 2 minutes, and deploy your application.

Feel free to test this, you can use my testenv (https://gist.github.com/dromie/0199dada8f1d080c83c292e2e3e30f85)

[FrancescoAzzariti]

@dromie
Sorry to bother you, but i also still have this problem.
I have ArgoCD 2.8.2

• The Cert is in the argocd-tls-certs-cm for more then 24h
• I added the repository declaratively like this:

apiVersion: v1
kind: Secret
metadata:
  labels:
    argocd.argoproj.io/secret-type: repository
  name: argocd-repo-test-helm-oci
  namespace: argocd-namespace  
data:
  project: integrator
  enableOCI: true
  insecure: true
  name: myrepo.example.com
  password: somepassword
  type: helm
  url: myrepo.example.com
  username: someuser
type: Opaque

(the insecure: true is equivalent to the --insecure-skip-server-verification parameter, right?)

The GUI shows the Repo connected succesfully, but i still get the x509 Error after creating an Application with a Helm Chart from this Repo

Additional Info:

• I use Jfrog Artifactory for the Private Repo
• If I use a normal Helm Repository (not OCI) on the Jfrog Artifactory, it works fine

Am i missing something?
Thanks!

[dromie]

@FrancescoAzzariti
In the argocd-server pod's container you should be able to find the "openssl" command, in my testsetup the following command shows "Verification: OK", what does your's say?

openssl s_client -debug -showcerts -CAfile /app/config/tls/myregistry -connect myregistry:443

[FrancescoAzzariti]

I just checked - mine says:
Verify return code: 2 (unable to get issuer certificate)

i'll try adding the cert of the issuer / the whole chain

EDIT:
@dromie
I added the chain to the configmap (selfsigned PKI), waited until the cert was in the Pod and now I get
Verification: OK

After this I retried to apply the Application and i still get:
tls: failed to verify certificate: x509: certificate signed by unknown authority

EDIT2:
I found a solution that works for us:
I mounted the configmap which includes the cert+chain provided by trustmanager into the "reposerver" and "server" pod to /etc/ssl/certs/pkichain.pem

[ElouanLD]

Hi, we are facing the exact same problem on argocd v2.9.0. Helm version in pods is v3.12.1.

We have one Harbor instance signed with a private CA in our network.

We are trying to setup argocd to pull charts from an application but we always face the "x509: certificate signed by unknown authority".

Local charts calling for the main metallb chart stored on Harbor :

apiVersion: v2
name: metallb
description: A network load-balancer implementation for Kubernetes using standard routing protocols
type: application
version: 0.0.1
appVersion: 0.0.1

dependencies:
- name: helm/metallb
  repository: oci://registry.harbor.ats-sre.ngxp.domaintests.loc
  version: 0.13.9 

I have performed the following tasks :

• Setup the registry in argocd (argocd repo add registry.harbor.ats-sre.ngxp.domaintests.loc --type helm --name harbor --username test --enable-oci --insecure-skip-server-verification)
• Add the harbor certificate to argocd "Repository certificates and known hosts"
• Wait 2 minutes
• Checked that the configmap correclty contains the harbor certificate
• Checked that the harbor certificate is correctly mounted in the /app/config/tls/ folder of both "reposerver" and "server" pods
• Checked that openssl correctly responds "Verification: OK" to previous openssl command (openssl s_client -debug -showcerts -CAfile /app/config/tls/myregistry -connect myregistry:443)

@FrancescoAzzariti you seemed to have found a way to make it work. Can you explain how did you perform the resolving task ? I don't quite understand what you mean by "I mounted the configmap which includes the cert+chain provided by trustmanager into the "reposerver" and "server" pod to /etc/ssl/certs/pkichain.pem". Can you provide more details ? (commands or created confimaps ?) Thank you !

Ps : To expand on this bug, we found that by creating an application from the UI did correctly pull the helm chart. From the "Repositories" settings panel, you can create an application from a specific registry by clicking on the 3 dots and Create application. If we do that, there is no issue, the helm chart is correctly retrieved from Harbor. It seems that the "--insecure-skip-verify" of helm is correctly passed and used here but not if we point to the repository in a chart or application.

[augeivv]

Hello,
First of all, thank you very much for the information.
I have done some more research as I am still having these problems.

I assume we are currently talking about different possibilities. ArgoCD and Helm Charts.

Roughly speaking, I have the same setup as @ElouanLD
And can agree that it doesn't work with version 2.8.4 or newer either.

If I link directly to the Helm Chart repository in my ArgoCD application, the deployment works. However, if I link to a Git repository in the ArgoCD application, which contains a local chart, I still get the x509 error.

I have also tried @FrancescoAzzariti solution. But without success.

[FrancescoAzzariti]

The ConfigMap ist created by Trustmanager ( https://github.com/cert-manager/trust-manager ) and placed inside the argo-cd namespace.

Afterwards I updated the values.yaml of the helmchart by which ArgoCD is installed:
https://github.com/argoproj/argo-helm/blob/main/charts/argo-cd/values.yaml#L2197-L2206

This looks something like this for me:

repoServer:
  # -- Additional volumeMounts to the Repo Server main container
  volumeMounts:
    - mountPath: /etc/ssl/certs/pkichain.pem
      name: pki-ca-certs
      subPath: root-certs.crt

  # -- Additional volumes to the Repo Server pod
  volumes:
    - name: pki-ca-certs
      configMap:
        name: pki-ca-certs

I hope this helps :)
@ElouanLD @augeivv

[augeivv]

@FrancescoAzzariti

Thank you for your help. It works.
I had bound the certificates to the server pod. But not to the repo server.

[benjsc]

Facing the same issue We have a application with a source being a gitrepo, referencing a helmchart, with a dependancy on another helmchart which in an oci registry.

This causes argocd to need to run helm dependency build_ but that doesn't pick up the certs or honour the tls ignore and argocd-server reports the error:

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = `helm dependency build` failed exit status 1: Error: could not download oci://<registryhere>: failed to do request: Head "https://<registryhere>/v2/<chartnamehere>/manifests/<chartversionhere>": tls: failed to verify certificate: x509: certificate signed by unknown authority

For anyone following if you get the error due to helm dependency build failing, your after this bug #6477

[BigKAA]

Hello
It doesn't work for me either.

argocd:v2.10.0

{"execID":"092dc","level":"error","msg":"`helm pull oci://registry.kryukov.local/charts/devcontainer --version 0.1.0 --destination /tmp/0a234fff-856f-4fbe-af0d-6e6cc2404793` failed exit status 1: Error: failed to do request: Head \"https://registry.kryukov.local/v2/charts/devcontainer/manifests/0.1.0\": tls: failed to verify certificate: x509: certificate signed by unknown authority","time":"2024-02-28T10:21:14Z"}

in containers. The file registry.kryukov.local contains the CA certificate

argocd@argo-cd-argocd-server-5f8bcf7999-kt2pj:~$ ls /app/config/tls
registry.kryukov.local

argocd@argo-cd-argocd-repo-server-7bdbfcc9b4-qsxh5:/etc/ssl/certs$ ls /app/config/tls
registry.kryukov.local

[lucatr]

Facing the same issue, pretty much same setup as @FrancescoAzzariti. Might very well be a chain issue as well, and I'll try to fix that accordingly.

In the argocd GUI I see the error certificate signed by unknown authority. This shouldn't pop up if using --insecure-skip-server-verification (respectively setting insecure to true) in the repo configuration, right? Otherwise I don't understand what the flag is there for.

[svghadi]

I think this has been fixed with #16656. Self-signed certs + helm oci registry worked for me with Argo CD v2.10.11. Try latest version of 2.10.x or 2.11.x. However, there is a bug (#19138) that affects certain scenarios, but it should work in most cases.