ApplicationSet controller receives x509: certificate signed by unknown authority, when using SCM, PR or Git generator with private bitbucket server

[LanceR1303]

Slack conversation thread : https://cloud-native.slack.com/archives/C01U45M2SVB/p1654569552211179
Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug
Receive x509: certificate signed by unknown authority when application set controller with an SCM generator attempts to connect to a private bitbucket server.

To Reproduce

1. Follow installation for release v2.4.0-rc4 under https://github.com/argoproj/argo-cd/releases
2. Follow steps to setup TLS certificates as documented https://argo-cd.readthedocs.io/en/release-1.8/user-guide/private-repositories/#self-signed-untrusted-tls-certificates
3. Create an ApplicationSet CRD with an SCM generator for the Bitbucket Server SCM. The bitbucket server needs to be a private repository with TLS connectivity via self signed certificates which should have been registered in the above step

Expected behavior

Expected the application set controller to be able to communicate with bitbucket server API's to list repositories and automate ArgoCD application generation.

Screenshots

Version

{
    "Version": "v2.4.0-rc2+cd5b2af",
    "BuildDate": "2022-05-18T12:42:45Z",
    "GitCommit": "cd5b2af358f3574ea151fr8c24b1d52dafe59c85",
    "GitTreeState": "clean",
    "GoVersion": "go1.18.2",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v4.4.1 2021-11-11T23:36:27Z",
    "HelmVersion": "v3.8.1+g5cb9af4",
    "KubectlVersion": "v0.23.1",
    "JsonnetVersion": "v0.18.0"
}

Logs

• applicationset-controller logs

time="2022-06-08T00:45:28Z" level=info msg="Starting configmap/secret informers"
time="2022-06-08T00:45:28Z" level=info msg="Configmap/secret informer synced"
time="2022-06-08T00:45:28Z" level=info msg="Starting webhook server"
time="2022-06-08T00:45:28Z" level=info msg="Starting manager"
time="2022-06-08T00:45:29Z" level=error msg="error generating params" error="error listing repos: error listing repositories for: Get \"https://<myBitBucketServerHost>/rest/api/1.0/projects/<myProject>/repos?limit=100\": x509: certificate signed by unknown authority" generator="&{0xc000c6c1e0 <nil>}"

• Volume and Volume Mounts of the applicationset-controller

 Mounts:
      /app/config/ssh from ssh-known-hosts (rw)
      /app/config/tls from tls-certs (rw)
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from argocd-applicationset-controller-token-59cjk (ro)


Volumes:
  tls-certs:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      argocd-tls-certs-cm
    Optional:  false
  ssh-known-hosts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      argocd-ssh-known-hosts-cm
    Optional:  false

[LanceR1303]

Observations:

1. We attempted to deploy our application with an App of Apps deployment pattern and we observed no issues. We are using the same argocd-tls-certs-cm configmap for the App of Apps deployment strategy and we observe that the argocd-application-controller and argocd-repo-server are able to communicate with our internal bitbucket server correctly.
2. While looking through the code we can see that
   a) In argocd_repo_server.go , the system makes use of tlsConfigCustomizer class to instantiate the TLS certificates. The implementation of which is in tls.go
   b) In argocd_application_controller.go, the system uses the util function tls.LoadX509CertPool to setup the TLS trust store
   c) In argocd_applicationset_controller.go, we couldn't locate code for loading the certificates and suspect that this could be the problem?
   d) The bitbucket SCM generator bitbucket_server.go uses https://github.com/gfleury/go-bitbucket-v1 . On running through the code in this library we did not find any specific instantiation of TLS certificates as well.

[LanceR1303]

ApplicationSet CRD with status events

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  name: appset
  namespace: argocd
spec:
  generators:
    - scmProvider:
        bitbucketServer:
          allBranches: false
          api: 'MyBitBucketHost'
          basicAuth:
            passwordRef:
              key: MyPass
              secretName: argocd-generator-token
            username: MyName
          project: Apps
  template:
    metadata:
      name: 'My Application'
    spec:
      destination:
        namespace: apps
        server: 'https://kubernetes.default.svc'
      project: apps
      source:
        helm:
          parameters:
            - name: namespace
              value: default 
          releaseName: releaseName
        path: ''{{ path }}'
        repoURL: '{{ url }}'
        targetRevision: '{{ branch }}'
      syncPolicy:
        syncOptions:
          - CreateNamespace=true
status:
  conditions:
    - lastTransitionTime: '2022-06-01T10:28:16Z'
      message: >-
        error listing repos: error listing repositories for : Get
        "https://<myBitBucketServerHost>/rest/api/1.0/projects/<myProject>/repos?limit=100":
        x509: certificate signed by unknown authority
      reason: ApplicationGenerationFromParamsError
      status: 'True'
      type: ErrorOccurred
    - lastTransitionTime: '2022-06-01T10:28:16Z'
      message: >-
        error listing repos: error listing repositories for : Get
        "https://<myBitBucketServerHost>/rest/api/1.0/projects/<myProject>/repos?limit=100":
        x509: certificate signed by unknown authority
      reason: ErrorOccurred
      status: 'False'
      type: ParametersGenerated
    - lastTransitionTime: '2022-06-01T10:28:16Z'
      message: >-
        error listing repos: error listing repositories for: Get
        "https://<myBitBucketServerHost>/rest/api/1.0/projects/<myProject>/repos?limit=100":
        x509: certificate signed by unknown authority
      reason: ApplicationGenerationFromParamsError
      status: 'False'
      type: ResourcesUpToDate

[rishabh625]

Yeah it's a bug

[flaviohrocha]

+1 same problem using GitLab on-premisses

[D-BlindSide]

+1 same problem...

[PiePra]

+1 same

[zetti12345]

Is there a workaround possible?

[crenshaw-dev]

I'm not aware of any workaround. We need to provide a way to set TLS configurations for various endpoints.

For certain generators (PR generator?) it might make sense to pull the TLS config from any already-configured repo creds in the argocd namespace. For others (SCM generator), we'll probably need a separate ConfigMap (maybe reuse argocd-cm?) to store the TLS config.

As always, PRs welcome. If a core contributor manages to pick this up, they'll drop a note here. 🙂

[tabnul]

a blocker for us to use applicationsets. is there a way to allow unsafe certificates instead of trusting them?

[hnikt-jonasfh]

+1, this is a showstopper for us. We're experiencing this issue when using a Helm chart source with a cert signed by our custom CA, despite having explicitly mounted the cert chain in /etc/ssl/certs on the filesystem of the pods of the controller, server, repo server and applicationset controller. Argo CD 2.5.3+0c7de21 installed with helm chart v5.9.1

[guydolinksy]

Ran into this issue today, this is the workaround I found (keep in mind we're using an on premise setup of argocd and gitlab and thus don't need the usual certificates):

In the values.yaml of the argocd deployment add:

applicationSet:
extraVolumeMounts:
- mountPath: /etc/ssl/certs
name: tls-certs

This will mount the tls certificates from argocd's config-map into the default ssl certificate folder of golang's http client, getting around the issue. It does however override the other certificates so this isn't ideal, maybe someone can further improve on it in the future, at least until an option is made available to append these tls certificates.

Best of luck!
Guy

[hnikt-jonasfh]

Ran into this issue today, this is the workaround I found (keep in mind we're using an on premise setup of argocd and gitlab and thus don't need the usual certificates):

In the values.yaml of the argocd deployment add:

applicationSet: extraVolumeMounts: - mountPath: /etc/ssl/certs name: tls-certs

This will mount the tls certificates from argocd's config-map into the default ssl certificate folder of golang's http client, getting around the issue. It does however override the other certificates so this isn't ideal, maybe someone can further improve on it in the future, at least until an option is made available to append these tls certificates.

Best of luck! Guy

Hi,
Thank you! That solved the issue for us :) We have the certs saved in a secret (in the same namespace as argo) named "certificates", so for us the full spec is something along these lines:

applicationSet:
extraVolumeMounts:
- name: certificates
mountPath: /etc/ssl/certs/cert1.crt
subPath: cert1.crt
- name: certificates
mountPath: /etc/ssl/certs/cert2.crt
subPath: cert2.crt
extraVolumes:
- name: certificates
secret:
secretName: certificates
defaultMode: 420

This way we also get to keep whatever default certificates argo throws in /etc/ssl/certs! Hope this helps others who are having the same issues.

I noticed there are quite a few issues registered here with similar or directly related problems:

• Adding Helm https repo needs a ca-file optional input. #3539
• Helm repositories with self signed certs doesn't work when used as umbrella charts #3693
• x509: certificate signed by unknown authority #6101
• Failed to create application due to error "x509: certificate signed by unknown authority" #6376
• helm chart dependencies fail to download from private repo #6477

I still find it odd that certificates added to argo proper (e.g. via the UI or declaratively in the configmap for trusted tls certificates) do not register as valid when it comes to Argo CD using the Helm client. That seems to be a common thread with all these issues.

[rumstead]

We have a self-signed CA that needs to apply for all URLs so we BYOI.

Isn't as clean as a volume mount but also works.

FROM quay.io/argoproj/argocd:$ARGOCD_VERSION
# Become the root user for installing the certs
USER root
# We have a base image that grabs certs for us, you can get them however
COPY --from=cacerts /tmp/corpcert.cer /usr/local/share/ca-certificates/internal-ca-bundle.crt
RUN update-ca-certificates
# Fallback to the argo user
USER $ARGOCD_USER_ID

[gaeljw]

I'm not entirely sure but I believe I'm having the same issue when using Helm through Kustomize. But not when using Helm directly.

#13154

[sarsatis]

I'm not aware of any workaround. We need to provide a way to set TLS configurations for various endpoints.

For certain generators (PR generator?) it might make sense to pull the TLS config from any already-configured repo creds in the argocd namespace. For others (SCM generator), we'll probably need a separate ConfigMap (maybe reuse argocd-cm?) to store the TLS config.

As always, PRs welcome. If a core contributor manages to pick this up, they'll drop a note here. 🙂

Ran into this issue today, this is the workaround I found (keep in mind we're using an on premise setup of argocd and gitlab and thus don't need the usual certificates):
In the values.yaml of the argocd deployment add:
applicationSet: extraVolumeMounts: - mountPath: /etc/ssl/certs name: tls-certs
This will mount the tls certificates from argocd's config-map into the default ssl certificate folder of golang's http client, getting around the issue. It does however override the other certificates so this isn't ideal, maybe someone can further improve on it in the future, at least until an option is made available to append these tls certificates.
Best of luck! Guy

Hi, Thank you! That solved the issue for us :) We have the certs saved in a secret (in the same namespace as argo) named "certificates", so for us the full spec is something along these lines:

applicationSet:
extraVolumeMounts:

• name: certificates
  mountPath: /etc/ssl/certs/cert1.crt
  subPath: cert1.crt
• name: certificates
  mountPath: /etc/ssl/certs/cert2.crt
  subPath: cert2.crt
  extraVolumes:
• name: certificates
  secret:
  secretName: certificates
  defaultMode: 420

This way we also get to keep whatever default certificates argo throws in /etc/ssl/certs! Hope this helps others who are having the same issues.

I noticed there are quite a few issues registered here with similar or directly related problems:

• Adding Helm https repo needs a ca-file optional input. #3539
• Helm repositories with self signed certs doesn't work when used as umbrella charts #3693
• x509: certificate signed by unknown authority #6101
• Failed to create application due to error "x509: certificate signed by unknown authority" #6376
• helm chart dependencies fail to download from private repo #6477

I still find it odd that certificates added to argo proper (e.g. via the UI or declaratively in the configmap for trusted tls certificates) do not register as valid when it comes to Argo CD using the Helm client. That seems to be a common thread with all these issues.

Hi @guydolinksy @crenshaw-dev @D-BlindSide @zetti12345 @flaviohrocha

We are also facing the same issue. We are using Openshift operator to install ArogoCd i tried to update applicationSet by creating the secret as mentioned above but after i add volume section operator reloads and removes the volumes which i added. Please help if there is any other workaround or how to solve this issue

[ThomasVitt]

We have the same issue under Openshift 4.13 running the Openshift GitOps Operator (ArgoCD v2.6.7).
I connected repos from both BitBucket and GitLab on-prem servers with HTTPS and successfully created applicationsets with the "git"-generator. When doing the same with the "scmProvider" generator, I get the message:

  conditions:
  - lastTransitionTime: "2023-08-08T11:31:09Z"
    message: 'error listing repos: error listing repositories for REPO: Get "https://<url>/rest/api/1.0/projects/REPO/repos?limit=100":
      x509: certificate signed by unknown authority'
    reason: ApplicationGenerationFromParamsError
    status: "True"
    type: ErrorOccurred
  - lastTransitionTime: "2023-08-08T11:31:09Z"
    message: 'error listing repos: error listing repositories for REPO: Get "https://<url>/rest/api/1.0/projects/REPO/repos?limit=100":
      x509: certificate signed by unknown authority'
    reason: ErrorOccurred
    status: "False"
    type: ParametersGenerated
  - lastTransitionTime: "2023-08-08T11:31:09Z"
    message: 'error listing repos: error listing repositories for REPO: Get "https://<url>/rest/api/1.0/projects/REPO/repos?limit=100":
      x509: certificate signed by unknown authority'
    reason: ApplicationGenerationFromParamsError
    status: "False"
    type: ResourcesUpToDate

The repositories inside ArgoCD are not "insecure", so the certificates are implemented correct. But the scmProvider generator obviously has problems with them.

[ThomasVitt]

I just noticed that a pull request has been merged into release-2.8:
#14462
#14348
Do they solve the problem?

[crenshaw-dev]

They might solve it for GitLab. We'd need equivalent PRs for other SCM providers.

I do think it's a miss that ApplicationSet doesn't know how to use the existing configured certificates. There's a similar problem with repositories, because there's incongruence between accessing an SCM's git interface and accessing its broader API.

[PiePra]

My simple workaround for the Pull Request Generator was to add the Cert at settings / Repository certificates and known hosts.
Those certificates will be present in the configmap argocd-tls-certs-cm in namespace argocd. I then added an additional volumeMount to Applicationset Deployment at /etc/ssl/certs like

        volumeMounts:
        ...
        - mountPath: /etc/ssl/certs
          name: tls-certs

[jdoylei]

I had the same experience as @ThomasVitt with ApplicationSet generators - the git generator connects successfully, but the scmProvider generator (with bitbucketServer) fails with "x509: certificate signed by unknown authority". Both generators are configured with the same protocol (https:) and the same Bitbucket Server hostname. The Bitbucket Server hostname matches an Argo CD "repository certificate" stored in argocd-tls-certs-cm, which must be sufficient for the git generator.

[appiepollo14]

@ThomasVitt still running into this error on 2.8 with BitBucket Server. No workaround available for us so this is an impediment.

Anyone working on this? Any planning for this one?

[appiepollo14]

@crenshaw-dev in regards to: #9607 (comment) I would like to provide a pr with the following in mind:

Add a configMapKeyRef on the PullRequest and SCMProvider configuration. This CM should hold the tls certs to be trusted. When the appSetController runs in the same namespace as argo applicationController it will be possible to configure the tls certs through argo. Those certs are stored in the argocd-tls-certs-cm and can then, via this configuration , be used by the applicationSet.

It this way we also tackle another issue with the solutions provided in earlies PR's. That is that we can use an envVar to point to certs on the file system, however, the RedHat GitOps operator doesn't allow extraVolumeMounts, see redhat-developer/gitops-operator#620 Which means that workaround is not usably for the users of that operator. By using a CM, we overcome that issue entirely. Would love to give this a try, please provide any feedback.

[appiepollo14]

@ishitasequeira this shouldn't be closed. There are still providers / generators which hold no implementation of insecure tls or passing private certificates. For example the pr mentioned above: #18501

[mixxor]

For those who can't use the newest Argo CD version with caRef support yet, here's a working solution for custom certificates, tested with Bitbucket Server:

1. Add your CA certificate to the argocd-tls-certs-cm ConfigMap (via UI or IaC)

2. Add this to your Argo CD Helm values:

applicationSet:
  extraVolumeMounts:
    - name: tls-certs           # Uses the existing certificates volume
      mountPath: /etc/ssl/certs/your-scm-fqdn.crt
      subPath: your-scm-fqdn    # Must match your key in argocd-tls-certs-cm

equivalent you can patch it by changing the application-set-deployment

This mounts your certificate from the existing tls-certs volume to a specific path that ApplicationSet will recognize.

3. Restart Application-Set-Controller.