Difficulty adding a Helm repository hosted by an ArgoCD managed instance of Harbor

[jsolbrig]

Checklist:

• [ x ] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [ x ] I've included steps to reproduce the bug.
• [ x ] I've pasted the output of argocd version.

Describe the bug

One of the apps that I am managing with ArgoCD is Harbor. The Harbor instance hosts additional helm charts that are needed by other apps that ArgoCD will manage. Harbor stands up cleanly and I can pull from the helm repository using helm from the command line. However, when I add Harbor as a helm repository in ArgoCD:

> cat my-ca.crt | argocd cert add-tls harbor... --insecure
Enter TLS certificate data in PEM format. Press CTRL-D when finished.
Created entry with 1 PEM certificates for repository server harbor...
> argocd repo add https://harbor... --name harbor --type helm --username *** --password *** --insecure-skip-server-verification
repository 'https://harbor...' added

then attempt to sync an application that depends on a chart that is hosted in Harbor's helm repository it fails with:

> argocd app sync my-app
`helm dependency build` failed exit status 1: Error: could not find : chart my-app not found in https://harbor...

If I then exec into the argocd-repo-server pod and run helm repo add, the problem is fixed.

> kubectl exec -it -n argocd argocd-repo-server-7d69cb45f4-kxf52 -- helm repo add harbor https://harbor... \
        --username=*** --password=*** --insecure-skip-tls-verify --ca-file=/app/config/tls/harbor...
"harbor" has been added to your repositories
> argocd app sync my-app
Successful sync

What I'd really like to do

I'd really like to be able to add the repository declaratively so that I can get the CA certificate and Harbor's login info from secrets, but that doesn't work well since ArgoCD expects all of its repos to be available. Updating argocd-cm doesn't seem to have any effect once ArgoCD is initialized.

To Reproduce

• Create an ArgoCD-managed instance of Harbor.
• Add a helm chart to it.
• Add the Harbor-hosted helm repository to argocd via argocd repo add and argocd cert add.
• Build an application using the chart hosted in harbor.
• Attempt to sync that application via argocd app sync.

I can try to put together a repository for this if needed.

Expected behavior

ArgoCD will find the chart in Harbor and correctly deploy the application.

Screenshots

If applicable, add screenshots to help explain your problem.

Version

argocd: v1.7.3+b4c79cc
  BuildDate: 2020-09-01T23:17:31Z
  GitCommit: b4c79ccb88173604c3786dcd34e83a9d7e8919a5
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
argocd-server: v1.7.3+b4c79cc
  BuildDate: 2020-09-01T23:19:02Z
  GitCommit: b4c79ccb88173604c3786dcd34e83a9d7e8919a5
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: {Version:kustomize/v3.6.1 GitCommit:c97fa946d576eb6ed559f17f2ac43b3b5a8d5dbd BuildDate:2020-05-27T20:47:35Z GoOs:linux GoArch:amd64}
  Helm Version: version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
  Kubectl Version: v1.17.8

[jessesuen]

We know that cert propagation to the repo-server will take potentially minutes, because kubelet sync the changes to pods only periodically. Was sufficient time waited to verify the cert did not take affect? I would say 5 minutes would be sufficient.

[jsolbrig]

I retried this while waiting more than 5 minutes between each command:

16:01:12 > cat ~/cert_authority/cluster.crt | argocd cert add-tls "harbor.***" --insecure
Enter TLS certificate data in PEM format. Press CTRL-D when finished.
Created entry with 1 PEM certificates for repository server harbor.***

16:07:10 > argocd repo add "https://harbor.***" --name harbor --type \
    helm --username *** --password *** --insecure-skip-server-verification
repository 'https://harbor.***' added

16:16:18 > argocd app sync cs-job-system
TIMESTAMP  GROUP        KIND   NAMESPACE                  NAME    STATUS   HEALTH        HOOK  MESSAGE

Name:               ***
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          ***
URL:                http://localhost:9001/applications/***
Repo:               git@***.git
Target:             helm3
Path:               apps/***
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Unknown
Health Status:      Healthy

Operation:          Sync
Sync Revision:
Phase:              Error
Start:              2020-09-03 16:12:10 -0600 MDT
Finished:           2020-09-03 16:12:10 -0600 MDT
Duration:           0s
Message:            ComparisonError: rpc error: code = Unknown desc = `helm dependency build` failed exit status 1: Error: could not find : chart *** not found in https://harbor.***
FATA[0001] Operation has completed with phase: Error

It works once I do this:

> repo_server=$(kubectl get pod -n argocd -l app.kubernetes.io/name=argocd-repo-server \
    -o jsonpath="{.items[0].metadata.name}")
> kubectl exec -it -n argocd $repo_server -- helm repo add harbor \
    "https://harbor.***" --username=*** \
    --password=*** --insecure-skip-tls-verify \
    --ca-file=/app/config/tls/harbor.***

[jessesuen]

Thanks we'll look into it

[jsolbrig]

@jessesuen Let me know if there is any other info I can provide. I'm able to reproduce this on demand, so I should be able to give any information you need.

[sll552]

@jessesuen any news on this issue? we are hitting it as well. Helm repo works when manually adding inside the argo-repo-server container und using helm from the commandline there, but not through syncing with argo. Also happens with argo 1.7.9

[aschaber1]

I didn't get much further, but I think you might have been using the wrong helm URL. I have tried the command below and at least got it working. But then got stuck on another issue (OpenShift SCCs).
Which makes this particular issue probably obsolete?

$ argocd app create harbor --repo https://github.com/goharbor/harbor-helm.git --path . --dest-namespace harbor --dest-server https://kubernetes.default.svc

$ argocd app get harbor
Name:               harbor
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          harbor
URL:                https://<argocd-url>/applications/harbor
Repo:               https://github.com/goharbor/harbor-helm.git
Target:
Path:               .
SyncWindow:         Sync Allowed
Sync Policy:        <none>
Sync Status:        OutOfSync from  (80f97cf)
Health Status:      Degraded

$ argocd version
argocd: v1.8.7+eb3d1fb.dirty
  BuildDate: 2021-03-07T20:25:24Z
  GitCommit: eb3d1fb84b9b77cdffd70b14c4f949f1c64a9416
  GitTreeState: dirty
  GoVersion: go1.16
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v2.0.0+f5119c0
  BuildDate: 2021-04-07T06:00:33Z
  GitCommit: f5119c06686399134b3f296d44445bcdbc778d42
  GitTreeState: clean
  GoVersion: go1.16
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v3.9.4 2021-02-09T19:22:10Z
  Helm Version: v3.5.1+g32c2223
  Kubectl Version: v0.20.4
  Jsonnet Version: v0.17.0