[Feature] Add support for Microsoft Entra Pod Identity as an authentication mechanism

[yurhasko]

What problem are you facing?

Currently, there's no option to scope rich Crossplane permissions to a certain ServiceAccount, like EKS Pod Identity or IRSA in AWS do.

What could help solve your problem?

Would be nice to have Microsoft Entra Pod Identity as a more secure authentication mechanism option.

[patst]

Actually there is, we are using it in production quite a while.

You need a workload identity enabled cluster, a K8S ServiceAccount mapped to a UserIdentity and then you can use a providerconfig like this:

apiVersion: azure.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: azure-provisioning-tenantABC
  namespace: crossplane-system
  labels:
    azure.workload.identity/use: "true" 
spec:
  clientID: {{ .Values.clientId }}
  credentials:
    source: OIDCTokenFile
  oidcTokenFilePath: /var/run/secrets/azure/tokens/azure-identity-token
  subscriptionID: {{ .Values.subscriptionId }}
  tenantID: {{ .Values.tenantId }}

We use this as DeploymentRuntimeConfig:

apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: provider-azure-family-config
spec:
  deploymentTemplate:
    spec:
      selector: {}
      template:
        metadata:
          labels:
            azure.workload.identity/use: "true"
            gap.hdi.global/scrape-metrics: "true"
        spec: 
          # { optional stuff to configure proxy and security settings}
          serviceAccountName: crossplane-azure-provisioning # we created this serviceaccount and tied it to an azure identity for workload identity federation usage

Provider Config:

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: azure-provider-network
spec:
  runtimeConfigRef:
    name: provider-azure-family-config
  package: "<custom image location>"
  # otherwise providers tries to resolve azure family from xpkg.upbound.io
  skipDependencyResolution: true

We have a lot of tenants in the cluster and each tenant has a dedicated ProviderConfig.

Hope that helps.

This uses WorkloadIdentity which is the successor of PodIdentity, which is preview and deprecated (or already removed?).

[yurhasko]

Just to clarify - so this way, access is scoped to workloads that use the ServiceAccount specified in the DeploymentRuntimeConfig, same way as with EKS Pod Identity for AWS provider? Or they are applied at the kubelet level?

[patst]

Just to clarify - so this way, access is scoped to workloads that use the ServiceAccount specified in the DeploymentRuntimeConfig, same way as with EKS Pod Identity for AWS provider? Or they are applied at the kubelet level?

Yes, even one step further down the road. It is scoped to the ServicePrincipal in EntraId.

Example:
We have a UserManagedIdentity tenant-abc in EntraId and can assign this identity permissions to to certain things (e.g. provisioning resources, when we are talking about the Azure Crossplane Providers or accessing the Kubernetes Cluster, reading ConfigMaps in a namespace if we are talking about the Crossplane Kubernetes Provider.

The UserManagedIdentity has FederatedCredentials configured, which allows the ServiceAccount to get access token (see https://learn.microsoft.com/en-us/graph/api/resources/federatedidentitycredentials-overview)

The ServiceAccount of the crossplane provider will get the access tokens depending on the credentials you configured in the DeploymentRuntimeConfig.

Maybe a good read is the basic flow for WorkloadIdentityFederation in general: https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview?tabs=dotnet#how-it-works

In the end the crossplane providers are just pods using the mechanism to fetch the access token.

[yurhasko]

Many thanks, I will try it out. Would be nice to have this documented in https://github.com/crossplane-contrib/provider-upjet-azure/blob/main/AUTHENTICATION.md (I can potentially add a PR for it it as soon as I get this working)

[yurhasko]

So I did the following:

• via Terraform:
  • enabled OIDC and Workload Identity on AKS cluster
  • configured user-assigned managed identity (with Contributor)
  • created federation identity credential
• via K8s manifests / Helm charts:
  • a ServiceAccount (myself, not automatically by Crossplane) for providers' Pods
  • Crossplane (from a Helm chart), installed 1.17.0, which works fine for EKS clusters with EKS Pod Identity, so used it just to be consistent across the clouds atm)
  • a ProviderConfig with the necessary azure.workload.identity/use: "true" label and all other stuff
  • a DeploymentRuntimeConfig (however, it differs from yours - validation didn't pass until I specified containers: [] (was failing with The DeploymentRuntimeConfig "provider-azure-runtime-config" is invalid: spec.deploymentTemplate.spec.template.spec.containers: Required value).
  • the xpkg.upbound.io/upbound/provider-azure-storage provider (tried multiple versions from 1.8.* to 1.10.2) with skipDependencyResolution: true, so it didn't install the Azure provider family.

And I'm stuck on the following issue despite trying different configurations / provider versions:

k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229:
  Failed to watch *v1beta1.ProviderConfig: failed to list *v1beta1.ProviderConfig:
  providerconfigs.azure.upbound.io is forbidden:
  User "system:serviceaccount:crossplane-system:provider-azure" cannot list resource "providerconfigs"
  in API group "azure.upbound.io" at the cluster scope

Considering that you've mentioned that you created the ServiceAccount with the Workload Identity label as a prerequisite (it was specified in DeploymentRuntimeConfig, but isn't managed by the Crossplane resources, as far as I can tell), can you advise whether you've attached some additional ClusterRole / modified automatically created one?

Many thanks in advance

[patst]

So I did the following:

• via Terraform:

  • enabled OIDC and Workload Identity on AKS cluster
  • configured user-assigned managed identity (with Contributor)
  • created federation identity credential

• via K8s manifests / Helm charts:

  • a ServiceAccount (myself, not automatically by Crossplane) for providers' Pods
  • Crossplane (from a Helm chart), installed 1.17.0, which works fine for EKS clusters with EKS Pod Identity, so used it just to be consistent across the clouds atm)
  • a ProviderConfig with the necessary azure.workload.identity/use: "true" label and all other stuff
  • a DeploymentRuntimeConfig (however, it differs from yours - validation didn't pass until I specified containers: [] (was failing with The DeploymentRuntimeConfig "provider-azure-runtime-config" is invalid: spec.deploymentTemplate.spec.template.spec.containers: Required value).
  • the xpkg.upbound.io/upbound/provider-azure-storage provider (tried multiple versions from 1.8.* to 1.10.2) with skipDependencyResolution: true, so it didn't install the Azure provider family.

And I'm stuck on the following issue despite trying different configurations / provider versions:

k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229:
  Failed to watch *v1beta1.ProviderConfig: failed to list *v1beta1.ProviderConfig:
  providerconfigs.azure.upbound.io is forbidden:
  User "system:serviceaccount:crossplane-system:provider-azure" cannot list resource "providerconfigs"
  in API group "azure.upbound.io" at the cluster scope

Considering that you've mentioned that you created the ServiceAccount with the Workload Identity label as a prerequisite (it was specified in DeploymentRuntimeConfig, but isn't managed by the Crossplane resources, as far as I can tell), can you advise whether you've attached some additional ClusterRole / modified automatically created one?

Many thanks in advance

Ah sorry, yes the containers part is required as well. I Only removed it from the example because it contained a lot of our environment specific configuration.

Maybe try to remove the skipDependencyResolution: true part. We only have it because we are running in an environment where we cannot access the xpkg Container Registry. If you have that as well, make sure to install the azure-family-provider manually as well.

Afaik there is no additional cluster role binding required.