Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Env-Injector with Workload Identity does not work without proxy sidecar annotation #692

Open
chrisorisawayi opened this issue Mar 27, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@chrisorisawayi
Copy link

chrisorisawayi commented Mar 27, 2024

Components and versions
Select which component(s) the bug relates to with [X].

[ ] Controller, version: x.x.x (docker image tag)
[x] Env-Injector (webhook), version: 1.6.0 (docker image tag)
[ ] Other

Describe the bug
Mutated pods using Workload Identity are unable to access keyvault without using Workload Identity proxy sidecar.

To Reproduce

  • Create managed identity - name wi-test in a subscription and resource group wi-test-rg
  • Deploy akv2k8s helm chart (with authentication service disabled):
controller:
  enabled: false
env_injector:
  authService: false # Note: Auth service is disabled as each pod (multi-tenanted namespaces) will use own credentials to auth with own keyvault
  envImage:
    repository: acrpath/akv2k8s/azure-keyvault-env
    tag: 1.6.1
  image:
    repository: acrpath/akv2k8s/azure-keyvault-webhook
    tag: 1.6.1
  • Create AzureIdentity
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentity
metadata:
  name: wi-test
  namespace: wi-test
spec:
  type: 0
  resourceID: /subscriptions/xxxxxxxx-yyyy-zzzz-aaaa-xxxxxxxxxx/resourceGroups/wi-test-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/wi-test
  clientID: <managed-identity-client-id>
  • Create AzureIdentityBinding
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
  name: wi-test
  namespace: wi-test
spec:
  azureIdentity: wi-test
  selector: wi-test
  • Create secret to inject
apiVersion: spv.no/v2beta1
kind: AzureKeyVaultSecret
metadata:
  name: secret-inject-test
  namespace: wi-test
  labels:
    app.kubernetes.io/akv2k8s: xxxxtestkeyvault
spec:
  vault:
    name: xxxxxtestkeyvault # name of key vault
    object:
      name: akv2k8s-secret-test # name of the akv object
      type: secret # akv object type
  • Create Federated Identity Credential (associated with managed identity created earlier and with AKS cluster OIDC url)
az identity federated-credential create \
  --name "wi-test" \
  --identity-name "wi-test" \
  --resource-group "wi-test-rg" \
  --issuer "aks cluster oidc url" \
  --subject "system:serviceaccount:wi-test:wi-test"
  • Create service account
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    azure.workload.identity/client-id: xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx
  name: wi-test
  namespace: test
  • Create pod
apiVersion: v1
kind: Pod
metadata:
  name: wi-demo-akvs
  namespace: test
  labels:
    azure.workload.identity/use: "true"
    app: wi-demo-akvs
spec:
  serviceAccountName: wi-test
  containers:
    - name: wi-demo-akvs
      image: ghcr.io/azure/azure-workload-identity/msal-go
      args:
        - --subscriptionid=xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx
        - --clientid=xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx
        - --resourcegroup=wi-test-rg
      env:
        - name: ENV_INJECTOR_LOG_LEVEL
          value: trace
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: MY_POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: MY_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: KEYVAULT_URL
          value: https://xxxxtestkeyvault.vault.azure.net/
        - name: SECRET_NAME
          value: akv2k8s-secret-test
        - name: AKV2K8S_TEST_SECRET
          value: "secret-inject-test@azurekeyvault" # ref to akvs
  nodeSelector:
    kubernetes.io/os: linux
  • Once deployed, pod starts up and the init container starts but fails to access keyvault and pod restarts after the keyvault access timeout expires (with error in the pod logs about failing to access keyvault)
  • By adding the below annotation to the pod, everything works as normal and keyvault secret is successfully retrieved
  annotations:
    azure.workload.identity/inject-proxy-sidecar: "true"

Expected behavior

  • Pod starts up and successfully access keyvault secret.
  • Dependency on using the proxy side car suggest the init-container spvest/azure-keyvault-env is not using the latest version of Azure Identity client library or not working as expected

Additional context
All examples given for akv2k8s working with Workload Identity (including the feature release) appear to be limited to those with AuthService enabled whereby the env-injector pod is expected to authenticate using its own service account rather than authentication being initiated from client pods across multiple namespaces and each with its own managed identity, federated credential and service account as obtains in a multi-tenant setup.

@chrisorisawayi chrisorisawayi added the bug Something isn't working label Mar 27, 2024
@chrisorisawayi chrisorisawayi changed the title [BUG] Env-Injector with Workload Identity does not work without proxy sidecar if using separate credentials [BUG] Env-Injector with Workload Identity does not work without proxy sidecar if using client auth Mar 27, 2024
@chrisorisawayi chrisorisawayi changed the title [BUG] Env-Injector with Workload Identity does not work without proxy sidecar if using client auth [BUG] Env-Injector with Workload Identity does not work without proxy sidecar annotation Mar 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant