[OCIRepository] GCP Artifact Registry Auth - Helm OCI 403 error

[kbelosevic]

Hello,

I am not sure what am I doing wrong but following error message is received:

2022-06-27T17:21:41.655Z error HelmChart/chart-name.flux-system - Reconciler error chart pull error: chart pull error: failed to get chart version for remote reference: GET "https://europe-west6-docker.pkg.dev/v2/gcp-project-name/artifact-registry-name/chart-name/tags/list": GET "https://europe-west6-docker.pkg.dev/v2/token?scope=repository%3Agcp-project-name%2Fartifact-registry-name%2Fchart-name%3Apull&service=europe-west6-docker.pkg.dev": unexpected status code 403: denied: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/project-name/locations/europe-west6/repositories/artifact-registry-name" (or it may not exist)

Versions:

NAME                         IMAGES
helm-controller              ghcr.io/fluxcd/helm-controller:v0.22.1
image-automation-controller  ghcr.io/fluxcd/image-automation-controller:v0.23.2
image-reflector-controller   ghcr.io/fluxcd/image-reflector-controller:v0.19.1
kustomize-controller         ghcr.io/fluxcd/kustomize-controller:v0.26.1
notification-controller      ghcr.io/fluxcd/notification-controller:v0.24.0
source-controller            ghcr.io/fluxcd/source-controller:v0.25.5

Config in flux:
HelmRepository

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: chart-name
  namespace: flux-system
spec:
  type: oci
  interval: 5m0s
  url: oci://europe-west6-docker.pkg.dev/gcp-project-name/artifact-registry-name

HelmRelease

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
    name: hr-name
    namespace: namespace
spec:
    interval: 5m
    chart:
        spec:
            chart: chart-name
            version: v0.0.0-latest
            sourceRef:
                kind: HelmRepository
                name: chart-name
                namespace: flux-system
            interval: 1m
    values:
        nameOverride: ""

Status shows ok:

kubectl get helmrepository                                                                                                                                                                                                                                                      ─╯
NAME         URL                                                            AGE     READY   STATUS
chart-name   oci://europe-west6-docker.pkg.dev/gcp-project-name/artifact-registry-name   5h17m   True    Helm repository is ready



kubectl describe helmrepository chart-name

Name:         chart-name
Namespace:    flux-system
Labels:       kustomize.toolkit.fluxcd.io/name=flux-system
              kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:  reconcile.fluxcd.io/requestedAt: 2022-06-27T18:44:09.65307+02:00
API Version:  source.toolkit.fluxcd.io/v1beta2
Kind:         HelmRepository
Metadata:
  Creation Timestamp:  2022-06-27T12:33:33Z
  Finalizers:
    finalizers.fluxcd.io
  Generation:  5
  Managed Fields:
    API Version:  source.toolkit.fluxcd.io/v1beta2
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          f:kustomize.toolkit.fluxcd.io/name:
          f:kustomize.toolkit.fluxcd.io/namespace:
      f:spec:
        f:interval:
        f:type:
        f:url:
    Manager:      kustomize-controller
    Operation:    Apply
    Time:         2022-06-27T16:52:12Z
    API Version:  source.toolkit.fluxcd.io/v1beta2
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"finalizers.fluxcd.io":
    Manager:      source-controller
    Operation:    Update
    Time:         2022-06-27T12:33:33Z
    API Version:  source.toolkit.fluxcd.io/v1beta2
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:reconcile.fluxcd.io/requestedAt:
    Manager:      flux
    Operation:    Update
    Time:         2022-06-27T15:36:12Z
    API Version:  source.toolkit.fluxcd.io/v1beta2
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
        f:lastHandledReconcileAt:
        f:observedGeneration:
    Manager:         source-controller
    Operation:       Update
    Subresource:     status
    Time:            2022-06-27T15:36:12Z
  Resource Version:  67802887
  UID:               06ee2b8e-166c-42d1-8871-72a0a5e9199e
Spec:
  Interval:  5m0s
  Timeout:   60s
  Type:      oci
  URL:       oci://europe-west6-docker.pkg.dev/gcp-project-name/artifact-registry-name
Status:
  Conditions:
    Last Transition Time:     2022-06-27T12:33:34Z
    Message:                  Helm repository is ready
    Observed Generation:      5
    Reason:                   Succeeded
    Status:                   True
    Type:                     Ready
  Last Handled Reconcile At:  2022-06-27T18:44:09.65307+02:00
  Observed Generation:        5
Events:                       <none>

Workload identity service account attached to kustomize-controller, source-controller and image-reflector-controller all have roles/artifactregistry.reader attached to them in artifact-registry-project, where GKE itself is deployed in separated project.

GCP Audit log shows:

{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "status": {
      "code": 2,
      "message": "permission denied by IAM"
    },
    "authenticationInfo": {},
    "requestMetadata": {
      "callerIp": "gce-internal-ip",
      "callerSuppliedUserAgent": "Helm/3.9,gzip(gfe)",
      "requestAttributes": {},
      "destinationAttributes": {}
    },
    "serviceName": "artifactregistry.googleapis.com",
    "methodName": "Docker-GetTags",
    "authorizationInfo": [
      {
        "resource": "projects/gcp-project-name/locations/europe-west6/repositories/artifact-registry-name",
        "permission": "artifactregistry.repositories.downloadArtifacts",
        "resourceAttributes": {}
      }
    ],
    "resourceName": "projects/gcp-project-name/locations/europe-west6/repositories/artifact-registry-name",
    "request": {
      "requestUrl": "/v2/gcp-project-name/artifact-registry-name/chart-name/tags/list",
      "requestMethod": "GET",
      "@type": "type.googleapis.com/google.logging.type.HttpRequest"
    },
    "resourceLocation": {
      "currentLocations": [
        "europe-west6"
      ],
      "originalLocations": [
        "europe-west6"
      ]
    }
  },
  "insertId": "18oh9rfc7nc",
  "resource": {
    "type": "audited_resource",
    "labels": {
      "method": "Docker-GetTags",
      "project_id": "gcp-project-name",
      "service": "artifactregistry.googleapis.com"
    }
  },
  "timestamp": "2022-06-27T17:21:41.435208412Z",
  "severity": "ERROR",
  "logName": "projects/gcp-project-name/logs/cloudaudit.googleapis.com%2Fdata_access",
  "receiveTimestamp": "2022-06-27T17:21:41.962287203Z"
}

While manual listing works:

gcloud artifacts docker images list europe-west6-docker.pkg.dev/gcp-project-name/artifact-registry-name/chart-name 

Listing items under project gcp-project-name, location europe-west6, repository artifact-registry-name.

Attaching same service-accounts in k8s to google/cloud-sdk pod allows listing and accessing the registry.

[darkowlzz]

Hi, OCI HelmRepository doesn't support auto-login at the moment. It's planned to be added in the future, but at the moment, you have to pass credentials in a secret reference as shows in https://fluxcd.io/docs/components/source/helmrepositories/#basic-access-authentication.
As a workaround, you can create a short lived access token using the workload identity in a CronJob and put it into a secret that HelmRepository can use. An example of this is in image automation authentication docs https://fluxcd.io/docs/guides/cron-job-image-auth/#using-access-token-short-lived . The example is for GCR, but it can be adapted to work with artifact registry as well.

The reason HelmRepository status shows Ready is because if no secret is provided, it doesn't know if it needs to perform a registry login. Only the HelmChart performs pull operation. HelmRepository performs login only if a secret is provided. HelmChart uses the secret provided in the HelmRepository.

[kbelosevic]

Thank you for the information.
Is there any ETA on when auto-login or similar will be available such as native AWS, GCP login with image-reflector-controller ?

[darkowlzz]

There's on going work to extract that code from image-reflector and use it in other projects. It's being tracked in fluxcd/image-reflector-controller#264 .
There's no ETA as far as I know. In order to make sure we don't break anything, we are adding integration tests in fluxcd/image-reflector-controller#275 and the registry login code has already been refactored into a new package fluxcd/image-reflector-controller#276. I believe once those are merged, we'd move the registry package into a separate repository with its own integration tests with all the cloud providers and then use that in source-controller for Helm OCI.

[kbelosevic]

Got it, thanks @darkowlzz
Will close this one then.

Also, temp resolution to this was to construct docker credentials like:

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: melon-core
  namespace: flux-system
spec:
  type: oci
  interval: 5m0s
  url: oci://europe-west6-docker.pkg.dev/gcp-project-name/artifact-registry-name
  secretRef:
    name: artifact-registry-credentials


---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: source-controller
  namespace: flux-system
rules:
- apiGroups: [""]
  resources:
  - secrets
  verbs:
  - get
  - create
  - patch
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: source-controller
  namespace: flux-system
subjects:
- kind: ServiceAccount
  name: source-controller
roleRef:
  kind: Role
  name: source-controller
  apiGroup: ""
---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: artifact-registry-credentials-sync
  namespace: flux-system
spec:
  suspend: false
  schedule: "*/35 * * * *"
  failedJobsHistoryLimit: 1
  successfulJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: source-controller
          restartPolicy: Never
          containers:
          - image: google/cloud-sdk
            name: create-secret
            imagePullPolicy: IfNotPresent
            env:
            - name: SECRET_NAME
              value: artifact-registry-credentials
            - name: GCR_REGISTRY
              value: https://europe-west6-docker.pkg.dev
            command:
            - /bin/bash
            - -ce
            - |-
              kubectl create secret docker-registry $SECRET_NAME \
                --dry-run=client \
                --docker-server="$GCR_REGISTRY" \
                --docker-username=oauth2accesstoken \
                --docker-password="$(gcloud auth print-access-token)" \
                -o yaml | kubectl apply -f - 

Then for the HelmRelease:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
    name: hr-name
    namespace: namespace
spec:
    interval: 5m
    chart:
        spec:
            chart: chart-name
            version: 0.0.0-latest ##### "v" as a prefix was removed as for some reason source-controller was unable to fetch any tag that has "string" as a first "sign"
            sourceRef:
                kind: HelmRepository
                name: chart-name
                namespace: flux-system
            interval: 5m
    values:
        nameOverride: ""