Provider generates invalid values with Experimental manifest stores feature turned on

[jw-maynard]

Terraform, Provider, Kubernetes and Helm Versions

Terraform version: v1.0.11
Provider version: v2.4.1
Kubernetes version: v1.20

Affected Resource(s)

• helm_release 1.2.6
• helm_repository https://aws.github.io/eks-charts

Terraform Configuration Files

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key.
locals {
  namespace = "kube-system"
}

resource "helm_release" "aws-load-balancer-controller" {
  timeout          = 600
  name             = "aws-load-balancer-controller"
  repository       = "https://aws.github.io/eks-charts"
  chart            = "aws-load-balancer-controller"
  version          = "1.2.6"
  namespace        = local.namespace
  create_namespace = true
  values = [templatefile("${path.module}/templates/values.yml.tpl", {
    cluster_name         = "kubernetes01"
    region               = "us-east-1"
    vpc_id               = "********"
    service_account_role = "********"
  })]
}

# Values YAML Template

clusterName: ${cluster_name}
serviceAccount:
    annotations:
        eks.amazonaws.com/role-arn: "${service_account_role}"
region: ${region}
vpcId: ${vpc_id}

Debug Output

NOTE: In addition to Terraform debugging, please set HELM_DEBUG=1 to enable debugging info from helm.

Panic Output

Steps to Reproduce

1. terraform plan
2. terraform apply

Expected Behavior

Currently in our config there are no changes that should need to be applied so the plan should show no changes

Actual Behavior

Many changes are shown and when applied an error occurs:

Error: Provider produced inconsistent final plan
When expanding the plan for module.kubernetes_cluster.module.aws-load-balancer-controller["service"].helm_release.aws-load-balancer-controller to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/helm" produced an invalid new value for .manifest: was cty.StringVal(...). This is a bug in the provider, which should be reported in the provider's own issue tracker.

The string value (with some data redacted) is below:

"{\"clusterrole.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1/aws-load-balancer-controller-role\":{\"apiVersion\":\"rbac.authorization.k8s.io/v1\",\"kind\":\"ClusterRole\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller-role\"},\"rules\":[{\"apiGroups\":[\"elbv2.k8s.aws\"],\"resources\":[\"targetgroupbindings\"],\"verbs\":[\"create\",\"delete\",\"get\",\"list\",\"patch\",\"update\",\"watch\"]},{\"apiGroups\":[\"elbv2.k8s.aws\"],\"resources\":[\"ingressclassparams\"],\"verbs\":[\"get\",\"list\",\"watch\"]},{\"apiGroups\":[\"\"],\"resources\":[\"events\"],\"verbs\":[\"create\",\"patch\"]},{\"apiGroups\":[\"\"],\"resources\":[\"pods\"],\"verbs\":[\"get\",\"list\",\"watch\"]},{\"apiGroups\":[\"networking.k8s.io\"],\"resources\":[\"ingressclasses\"],\"verbs\":[\"get\",\"list\",\"watch\"]},{\"apiGroups\":[\"\",\"extensions\",\"networking.k8s.io\"],\"resources\":[\"services\",\"ingresses\"],\"verbs\":[\"get\",\"list\",\"patch\",\"update\",\"watch\"]},{\"apiGroups\":[\"\"],\"resources\":[\"nodes\",\"secrets\",\"namespaces\",\"endpoints\"],\"verbs\":[\"get\",\"list\",\"watch\"]},{\"apiGroups\":[\"elbv2.k8s.aws\",\"\",\"extensions\",\"networking.k8s.io\"],\"resources\":[\"targetgroupbindings/status\",\"pods/status\",\"services/status\",\"ingresses/status\"],\"verbs\":[\"update\",\"patch\"]}]},\"clusterrolebinding.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1/aws-load-balancer-controller-rolebinding\":{\"apiVersion\":\"rbac.authorization.k8s.io/v1\",\"kind\":\"ClusterRoleBinding\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller-rolebinding\"},\"roleRef\":{\"apiGroup\":\"rbac.authorization.k8s.io\",\"kind\":\"ClusterRole\",\"name\":\"aws-load-balancer-controller-role\"},\"subjects\":[{\"kind\":\"ServiceAccount\",\"name\":\"aws-load-balancer-controller\",\"namespace\":\"kube-system\"}]},\"kube-system/deployment.apps/apps/v1/aws-load-balancer-controller\":{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller\",\"namespace\":\"kube-system\"},\"spec\":{\"replicas\":2,\"selector\":{\"matchLabels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\"}},\"template\":{\"metadata\":{\"annotations\":{\"prometheus.io/port\":\"8080\",\"prometheus.io/scrape\":\"true\"},\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\"}},\"spec\":{\"affinity\":{\"podAntiAffinity\":{\"preferredDuringSchedulingIgnoredDuringExecution\":[{\"podAffinityTerm\":{\"labelSelector\":{\"matchExpressions\":[{\"key\":\"app.kubernetes.io/name\",\"operator\":\"In\",\"values\":[\"aws-load-balancer-controller\"]}]},\"topologyKey\":\"kubernetes.io/hostname\"},\"weight\":100}]}},\"containers\":[{\"args\":[\"--cluster-name=kubernetes01\",\"--ingress-class=alb\",\"--aws-region=us-east-1\",\"--aws-vpc-id=vpc-REDACTED\"],\"command\":[\"/controller\"],\"image\":\"602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-load-balancer-controller:v2.2.3\",\"imagePullPolicy\":\"IfNotPresent\",\"livenessProbe\":{\"failureThreshold\":2,\"httpGet\":{\"path\":\"/healthz\",\"port\":61779,\"scheme\":\"HTTP\"},\"initialDelaySeconds\":30,\"timeoutSeconds\":10},\"name\":\"aws-load-balancer-controller\",\"ports\":[{\"containerPort\":9443,\"name\":\"webhook-server\",\"protocol\":\"TCP\"},{\"containerPort\":8080,\"name\":\"metrics-server\",\"protocol\":\"TCP\"}],\"resources\":{},\"securityContext\":{\"allowPrivilegeEscalation\":false,\"readOnlyRootFilesystem\":true,\"runAsNonRoot\":true},\"volumeMounts\":[{\"mountPath\":\"/tmp/k8s-webhook-server/serving-certs\",\"name\":\"cert\",\"readOnly\":true}]}],\"priorityClassName\":\"system-cluster-critical\",\"securityContext\":{\"fsGroup\":65534},\"serviceAccountName\":\"aws-load-balancer-controller\",\"terminationGracePeriodSeconds\":10,\"volumes\":[{\"name\":\"cert\",\"secret\":{\"defaultMode\":420,\"secretName\":\"aws-load-balancer-tls\"}}]}}}},\"kube-system/role.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1/aws-load-balancer-controller-leader-election-role\":{\"apiVersion\":\"rbac.authorization.k8s.io/v1\",\"kind\":\"Role\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller-leader-election-role\",\"namespace\":\"kube-system\"},\"rules\":[{\"apiGroups\":[\"\"],\"resources\":[\"configmaps\"],\"verbs\":[\"create\"]},{\"apiGroups\":[\"\"],\"resourceNames\":[\"aws-load-balancer-controller-leader\"],\"resources\":[\"configmaps\"],\"verbs\":[\"get\",\"patch\",\"update\"]}]},\"kube-system/rolebinding.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1/aws-load-balancer-controller-leader-election-rolebinding\":{\"apiVersion\":\"rbac.authorization.k8s.io/v1\",\"kind\":\"RoleBinding\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller-leader-election-rolebinding\",\"namespace\":\"kube-system\"},\"roleRef\":{\"apiGroup\":\"rbac.authorization.k8s.io\",\"kind\":\"Role\",\"name\":\"aws-load-balancer-controller-leader-election-role\"},\"subjects\":[{\"kind\":\"ServiceAccount\",\"name\":\"aws-load-balancer-controller\",\"namespace\":\"kube-system\"}]},\"kube-system/secret/v1/aws-load-balancer-tls\":{\"kind\":\"Secret\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"aws-load-balancer-tls\",\"namespace\":\"kube-system\",\"creationTimestamp\":null,\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"}},\"data\":{\"ca.crt\":\"KHNlbnNpdGl2ZSB2YWx1ZSBiZDg2NWZlOGM4Nzg0MDdmKQ==\",\"tls.crt\":\"KHNlbnNpdGl2ZSB2YWx1ZSBmZmNhN2NlNDk1ZjBlYzdlKQ==\",\"tls.key\":\"KHNlbnNpdGl2ZSB2YWx1ZSAxMTNjNTg2NjIxMjk2Zjk5KQ==\"},\"type\":\"kubernetes.io/tls\"},\"kube-system/service/v1/aws-load-balancer-webhook-service\":{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-webhook-service\",\"namespace\":\"kube-system\"},\"spec\":{\"ports\":[{\"port\":443,\"targetPort\":\"webhook-server\"}],\"selector\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\"}}},\"kube-system/serviceaccount/v1/aws-load-balancer-controller\":{\"apiVersion\":\"v1\",\"automountServiceAccountToken\":true,\"kind\":\"ServiceAccount\",\"metadata\":{\"annotations\":{\"eks.amazonaws.com/role-arn\":\"arn:aws:iam::REDACTED:role/REDACTED\"},\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller\",\"namespace\":\"kube-system\"}},\"mutatingwebhookconfiguration.admissionregistration.k8s.io/admissionregistration.k8s.io/v1/aws-load-balancer-webhook\":{\"apiVersion\":\"admissionregistration.k8s.io/v1\",\"kind\":\"MutatingWebhookConfiguration\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-webhook\"},\"webhooks\":[{\"admissionReviewVersions\":[\"v1beta1\"],\"clientConfig\":{\"caBundle\":\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURRRENDQWlpZ0F3SUJBZ0lSQUl2REtkbWEyeWRPcUdWdG1BY3MwcFF3RFFZSktvWklodmNOQVFFTEJRQXcKS2pFb01DWUdBMVVFQXhNZllYZHpMV3h2WVdRdFltRnNZVzVqWlhJdFkyOXVkSEp2Ykd4bGNpMWpZVEFlRncweQpNVEV5TURFeE9UTTVNVGxhRncwek1URXhNamt4T1RNNU1UbGFNQ294S0RBbUJnTlZCQU1USDJGM2N5MXNiMkZrCkxXSmhiR0Z1WTJWeUxXTnZiblJ5YjJ4c1pYSXRZMkV3Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXcKZ2dFS0FvSUJBUURkdk1WWHExcm1JOS9MQi9WUzN0RjE5S2czM280enlCd1ZvOEV4QWc4OURTNU9kWnVySktOTQpPRk4vTG9GV3hrSFNTdndveGRZN1N3RjhyMlZ5NzhZU0R1U2RoUFFNcUc3M29ENW5XUDRkOEZ6OHlndTFFRTlTClkvUVl5QnhEMmVaeDg1QXBLS2h0R1p4Z25lSXRHYVBkdHpldCs2NFJHVlNNampGU1cvQ0xEYVJCYktRbHNLK1gKNG9jdjVUREZVRVRhN0JoYmpPTFNudHNLUW10ZTVtQnVmK1lKNVZ6c3JHYVBCb1NMU2VyWUZibzhFRnlBZFM3ZwpPQWxjdVRwY1dXemdSR29OQzR3RElLRlljcTVrL0g2NUd6U2hsRHJISGd1aXJ6ZkhvVkFabXV2bWdKbHVGZmFwCmxYcE1DUUFuVkVXZ05WU29FZW1INUlGNmxLaC9WN29kQWdNQkFBR2pZVEJmTUE0R0ExVWREd0VCL3dRRUF3SUMKcERBZEJnTlZIU1VFRmpBVUJnZ3JCZ0VGQlFjREFRWUlLd1lCQlFVSEF3SXdEd1lEVlIwVEFRSC9CQVV3QXdFQgovekFkQmdOVkhRNEVGZ1FVWmtJL0pIc2VpNlBGVnVoQTFRMWxzK2ZZOWZjd0RRWUpLb1pJaHZjTkFRRUxCUUFECmdnRUJBSDNiRXNwTzRvdXVyd0VWYlU2TU9OSWJNMjRLbTc4dkxjTEF5RUhIcmlML3B2S3Z3M2dDb1N0WkJJVUcKWEFLWEV0cXJObjN3azZGMFlsWTNpM0JNUEZqRFJiNllUaXNKRE1NWXdFQ25mdEQzcTBTdUljZ2xsOGNpNTg3agpPbVNYZzlGa0ZGYWJLNlpPNmErT2Eva01oLzR5NW5sSnBTUTQ2bXgvQjZpZ1hOQTgvcThtM2FqVEFqN3NQWEZ0CnRLTmhYWWtCczFwTmZsYlFUdm01b0cwZzRSL2lXb3Vqa2pCaXA5MEZ3N1FDd05rZmswN0dFWk0xYmdJZ0oyS0MKS29nVndzbk9uOUp5QkhxL3hrRVRlQXNpWW53MUFWeTJPalBnM2w0ZjdxbDR6Q3ZVejhPYjRIYjFNUWt5eUQzQgp6WEduTW1DcDVWeFNhb2RjeEdFODlDVGpjV3c9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K\",\"service\":{\"name\":\"aws-load-balancer-webhook-service\",\"namespace\":\"kube-system\",\"path\":\"/mutate-v1-pod\"}},\"failurePolicy\":\"Fail\",\"name\":\"mpod.elbv2.k8s.aws\",\"namespaceSelector\":{\"matchExpressions\":[{\"key\":\"elbv2.k8s.aws/pod-readiness-gate-inject\",\"operator\":\"In\",\"values\":[\"enabled\"]}]},\"objectSelector\":{\"matchExpressions\":[{\"key\":\"app.kubernetes.io/name\",\"operator\":\"NotIn\",\"values\":[\"aws-load-balancer-controller\"]}]},\"rules\":[{\"apiGroups\":[\"\"],\"apiVersions\":[\"v1\"],\"operations\":[\"CREATE\"],\"resources\":[\"pods\"]}],\"sideEffects\":\"None\"},{\"admissionReviewVersions\":[\"v1beta1\"],\"clientConfig\":{\"caBundle\":\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURRRENDQWlpZ0F3SUJBZ0lSQUl2REtkbWEyeWRPcUdWdG1BY3MwcFF3RFFZSktvWklodmNOQVFFTEJRQXcKS2pFb01DWUdBMVVFQXhNZllYZHpMV3h2WVdRdFltRnNZVzVqWlhJdFkyOXVkSEp2Ykd4bGNpMWpZVEFlRncweQpNVEV5TURFeE9UTTVNVGxhRncwek1URXhNamt4T1RNNU1UbGFNQ294S0RBbUJnTlZCQU1USDJGM2N5MXNiMkZrCkxXSmhiR0Z1WTJWeUxXTnZiblJ5YjJ4c1pYSXRZMkV3Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXcKZ2dFS0FvSUJBUURkdk1WWHExcm1JOS9MQi9WUzN0RjE5S2czM280enlCd1ZvOEV4QWc4OURTNU9kWnVySktOTQpPRk4vTG9GV3hrSFNTdndveGRZN1N3RjhyMlZ5NzhZU0R1U2RoUFFNcUc3M29ENW5XUDRkOEZ6OHlndTFFRTlTClkvUVl5QnhEMmVaeDg1QXBLS2h0R1p4Z25lSXRHYVBkdHpldCs2NFJHVlNNampGU1cvQ0xEYVJCYktRbHNLK1gKNG9jdjVUREZVRVRhN0JoYmpPTFNudHNLUW10ZTVtQnVmK1lKNVZ6c3JHYVBCb1NMU2VyWUZibzhFRnlBZFM3ZwpPQWxjdVRwY1dXemdSR29OQzR3RElLRlljcTVrL0g2NUd6U2hsRHJISGd1aXJ6ZkhvVkFabXV2bWdKbHVGZmFwCmxYcE1DUUFuVkVXZ05WU29FZW1INUlGNmxLaC9WN29kQWdNQkFBR2pZVEJmTUE0R0ExVWREd0VCL3dRRUF3SUMKcERBZEJnTlZIU1VFRmpBVUJnZ3JCZ0VGQlFjREFRWUlLd1lCQlFVSEF3SXdEd1lEVlIwVEFRSC9CQVV3QXdFQgovekFkQmdOVkhRNEVGZ1FVWmtJL0pIc2VpNlBGVnVoQTFRMWxzK2ZZOWZjd0RRWUpLb1pJaHZjTkFRRUxCUUFECmdnRUJBSDNiRXNwTzRvdXVyd0VWYlU2TU9OSWJNMjRLbTc4dkxjTEF5RUhIcmlML3B2S3Z3M2dDb1N0WkJJVUcKWEFLWEV0cXJObjN3azZGMFlsWTNpM0JNUEZqRFJiNllUaXNKRE1NWXdFQ25mdEQzcTBTdUljZ2xsOGNpNTg3agpPbVNYZzlGa0ZGYWJLNlpPNmErT2Eva01oLzR5NW5sSnBTUTQ2bXgvQjZpZ1hOQTgvcThtM2FqVEFqN3NQWEZ0CnRLTmhYWWtCczFwTmZsYlFUdm01b0cwZzRSL2lXb3Vqa2pCaXA5MEZ3N1FDd05rZmswN0dFWk0xYmdJZ0oyS0MKS29nVndzbk9uOUp5QkhxL3hrRVRlQXNpWW53MUFWeTJPalBnM2w0ZjdxbDR6Q3ZVejhPYjRIYjFNUWt5eUQzQgp6WEduTW1DcDVWeFNhb2RjeEdFODlDVGpjV3c9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K\",\"service\":{\"name\":\"aws-load-balancer-webhook-service\",\"namespace\":\"kube-system\",\"path\":\"/mutate-elbv2-k8s-aws-v1beta1-targetgroupbinding\"}},\"failurePolicy\":\"Fail\",\"name\":\"mtargetgroupbinding.elbv2.k8s.aws\",\"rules\":[{\"apiGroups\":[\"elbv2.k8s.aws\"],\"apiVersions\":[\"v1beta1\"],\"operations\":[\"CREATE\",\"UPDATE\"],\"resources\":[\"targetgroupbindings\"]}],\"sideEffects\":\"None\"}]},\"validatingwebhookconfiguration.admissionregistration.k8s.io/admissionregistration.k8s.io/v1/aws-load-balancer-webhook\":{\"apiVersion\":\"admissionregistration.k8s.io/v1\",\"kind\":\"ValidatingWebhookConfiguration\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-webhook\"},\"webhooks\":[{\"admissionReviewVersions\":[\"v1beta1\"],\"clientConfig\":{\"caBundle\":\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURRRENDQWlpZ0F3SUJBZ0lSQUl2REtkbWEyeWRPcUdWdG1BY3MwcFF3RFFZSktvWklodmNOQVFFTEJRQXcKS2pFb01DWUdBMVVFQXhNZllYZHpMV3h2WVdRdFltRnNZVzVqWlhJdFkyOXVkSEp2Ykd4bGNpMWpZVEFlRncweQpNVEV5TURFeE9UTTVNVGxhRncwek1URXhNamt4T1RNNU1UbGFNQ294S0RBbUJnTlZCQU1USDJGM2N5MXNiMkZrCkxXSmhiR0Z1WTJWeUxXTnZiblJ5YjJ4c1pYSXRZMkV3Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXcKZ2dFS0FvSUJBUURkdk1WWHExcm1JOS9MQi9WUzN0RjE5S2czM280enlCd1ZvOEV4QWc4OURTNU9kWnVySktOTQpPRk4vTG9GV3hrSFNTdndveGRZN1N3RjhyMlZ5NzhZU0R1U2RoUFFNcUc3M29ENW5XUDRkOEZ6OHlndTFFRTlTClkvUVl5QnhEMmVaeDg1QXBLS2h0R1p4Z25lSXRHYVBkdHpldCs2NFJHVlNNampGU1cvQ0xEYVJCYktRbHNLK1gKNG9jdjVUREZVRVRhN0JoYmpPTFNudHNLUW10ZTVtQnVmK1lKNVZ6c3JHYVBCb1NMU2VyWUZibzhFRnlBZFM3ZwpPQWxjdVRwY1dXemdSR29OQzR3RElLRlljcTVrL0g2NUd6U2hsRHJISGd1aXJ6ZkhvVkFabXV2bWdKbHVGZmFwCmxYcE1DUUFuVkVXZ05WU29FZW1INUlGNmxLaC9WN29kQWdNQkFBR2pZVEJmTUE0R0ExVWREd0VCL3dRRUF3SUMKcERBZEJnTlZIU1VFRmpBVUJnZ3JCZ0VGQlFjREFRWUlLd1lCQlFVSEF3SXdEd1lEVlIwVEFRSC9CQVV3QXdFQgovekFkQmdOVkhRNEVGZ1FVWmtJL0pIc2VpNlBGVnVoQTFRMWxzK2ZZOWZjd0RRWUpLb1pJaHZjTkFRRUxCUUFECmdnRUJBSDNiRXNwTzRvdXVyd0VWYlU2TU9OSWJNMjRLbTc4dkxjTEF5RUhIcmlML3B2S3Z3M2dDb1N0WkJJVUcKWEFLWEV0cXJObjN3azZGMFlsWTNpM0JNUEZqRFJiNllUaXNKRE1NWXdFQ25mdEQzcTBTdUljZ2xsOGNpNTg3agpPbVNYZzlGa0ZGYWJLNlpPNmErT2Eva01oLzR5NW5sSnBTUTQ2bXgvQjZpZ1hOQTgvcThtM2FqVEFqN3NQWEZ0CnRLTmhYWWtCczFwTmZsYlFUdm01b0cwZzRSL2lXb3Vqa2pCaXA5MEZ3N1FDd05rZmswN0dFWk0xYmdJZ0oyS0MKS29nVndzbk9uOUp5QkhxL3hrRVRlQXNpWW53MUFWeTJPalBnM2w0ZjdxbDR6Q3ZVejhPYjRIYjFNUWt5eUQzQgp6WEduTW1DcDVWeFNhb2RjeEdFODlDVGpjV3c9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K\",\"service\":{\"name\":\"aws-load-balancer-webhook-service\",\"namespace\":\"kube-system\",\"path\":\"/validate-elbv2-k8s-aws-v1beta1-targetgroupbinding\"}},\"failurePolicy\":\"Fail\",\"name\":\"vtargetgroupbinding.elbv2.k8s.aws\",\"rules\":[{\"apiGroups\":[\"elbv2.k8s.aws\"],\"apiVersions\":[\"v1beta1\"],\"operations\":[\"CREATE\",\"UPDATE\"],\"resources\":[\"targetgroupbindings\"]}],\"sideEffects\":\"None\"},{\"admissionReviewVersions\":[\"v1beta1\"],\"clientConfig\":{\"caBundle\":\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURRRENDQWlpZ0F3SUJBZ0lSQUl2REtkbWEyeWRPcUdWdG1BY3MwcFF3RFFZSktvWklodmNOQVFFTEJRQXcKS2pFb01DWUdBMVVFQXhNZllYZHpMV3h2WVdRdFltRnNZVzVqWlhJdFkyOXVkSEp2Ykd4bGNpMWpZVEFlRncweQpNVEV5TURFeE9UTTVNVGxhRncwek1URXhNamt4T1RNNU1UbGFNQ294S0RBbUJnTlZCQU1USDJGM2N5MXNiMkZrCkxXSmhiR0Z1WTJWeUxXTnZiblJ5YjJ4c1pYSXRZMkV3Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXcKZ2dFS0FvSUJBUURkdk1WWHExcm1JOS9MQi9WUzN0RjE5S2czM280enlCd1ZvOEV4QWc4OURTNU9kWnVySktOTQpPRk4vTG9GV3hrSFNTdndveGRZN1N3RjhyMlZ5NzhZU0R1U2RoUFFNcUc3M29ENW5XUDRkOEZ6OHlndTFFRTlTClkvUVl5QnhEMmVaeDg1QXBLS2h0R1p4Z25lSXRHYVBkdHpldCs2NFJHVlNNampGU1cvQ0xEYVJCYktRbHNLK1gKNG9jdjVUREZVRVRhN0JoYmpPTFNudHNLUW10ZTVtQnVmK1lKNVZ6c3JHYVBCb1NMU2VyWUZibzhFRnlBZFM3ZwpPQWxjdVRwY1dXemdSR29OQzR3RElLRlljcTVrL0g2NUd6U2hsRHJISGd1aXJ6ZkhvVkFabXV2bWdKbHVGZmFwCmxYcE1DUUFuVkVXZ05WU29FZW1INUlGNmxLaC9WN29kQWdNQkFBR2pZVEJmTUE0R0ExVWREd0VCL3dRRUF3SUMKcERBZEJnTlZIU1VFRmpBVUJnZ3JCZ0VGQlFjREFRWUlLd1lCQlFVSEF3SXdEd1lEVlIwVEFRSC9CQVV3QXdFQgovekFkQmdOVkhRNEVGZ1FVWmtJL0pIc2VpNlBGVnVoQTFRMWxzK2ZZOWZjd0RRWUpLb1pJaHZjTkFRRUxCUUFECmdnRUJBSDNiRXNwTzRvdXVyd0VWYlU2TU9OSWJNMjRLbTc4dkxjTEF5RUhIcmlML3B2S3Z3M2dDb1N0WkJJVUcKWEFLWEV0cXJObjN3azZGMFlsWTNpM0JNUEZqRFJiNllUaXNKRE1NWXdFQ25mdEQzcTBTdUljZ2xsOGNpNTg3agpPbVNYZzlGa0ZGYWJLNlpPNmErT2Eva01oLzR5NW5sSnBTUTQ2bXgvQjZpZ1hOQTgvcThtM2FqVEFqN3NQWEZ0CnRLTmhYWWtCczFwTmZsYlFUdm01b0cwZzRSL2lXb3Vqa2pCaXA5MEZ3N1FDd05rZmswN0dFWk0xYmdJZ0oyS0MKS29nVndzbk9uOUp5QkhxL3hrRVRlQXNpWW53MUFWeTJPalBnM2w0ZjdxbDR6Q3ZVejhPYjRIYjFNUWt5eUQzQgp6WEduTW1DcDVWeFNhb2RjeEdFODlDVGpjV3c9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K\",\"service\":{\"name\":\"aws-load-balancer-webhook-service\",\"namespace\":\"kube-system\",\"path\":\"/validate-networking-v1beta1-ingress\"}},\"failurePolicy\":\"Fail\",\"matchPolicy\":\"Equivalent\",\"name\":\"vingress.elbv2.k8s.aws\",\"rules\":[{\"apiGroups\":[\"networking.k8s.io\"],\"apiVersions\":[\"v1beta1\"],\"operations\":[\"CREATE\",\"UPDATE\"],\"resources\":[\"ingresses\"]}],\"sideEffects\":\"None\"}]}}"), but now cty.StringVal("{\"clusterrole.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1/aws-load-balancer-controller-role\":{\"apiVersion\":\"rbac.authorization.k8s.io/v1\",\"kind\":\"ClusterRole\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller-role\"},\"rules\":[{\"apiGroups\":[\"elbv2.k8s.aws\"],\"resources\":[\"targetgroupbindings\"],\"verbs\":[\"create\",\"delete\",\"get\",\"list\",\"patch\",\"update\",\"watch\"]},{\"apiGroups\":[\"elbv2.k8s.aws\"],\"resources\":[\"ingressclassparams\"],\"verbs\":[\"get\",\"list\",\"watch\"]},{\"apiGroups\":[\"\"],\"resources\":[\"events\"],\"verbs\":[\"create\",\"patch\"]},{\"apiGroups\":[\"\"],\"resources\":[\"pods\"],\"verbs\":[\"get\",\"list\",\"watch\"]},{\"apiGroups\":[\"networking.k8s.io\"],\"resources\":[\"ingressclasses\"],\"verbs\":[\"get\",\"list\",\"watch\"]},{\"apiGroups\":[\"\",\"extensions\",\"networking.k8s.io\"],\"resources\":[\"services\",\"ingresses\"],\"verbs\":[\"get\",\"list\",\"patch\",\"update\",\"watch\"]},{\"apiGroups\":[\"\"],\"resources\":[\"nodes\",\"secrets\",\"namespaces\",\"endpoints\"],\"verbs\":[\"get\",\"list\",\"watch\"]},{\"apiGroups\":[\"elbv2.k8s.aws\",\"\",\"extensions\",\"networking.k8s.io\"],\"resources\":[\"targetgroupbindings/status\",\"pods/status\",\"services/status\",\"ingresses/status\"],\"verbs\":[\"update\",\"patch\"]}]},\"clusterrolebinding.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1/aws-load-balancer-controller-rolebinding\":{\"apiVersion\":\"rbac.authorization.k8s.io/v1\",\"kind\":\"ClusterRoleBinding\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller-rolebinding\"},\"roleRef\":{\"apiGroup\":\"rbac.authorization.k8s.io\",\"kind\":\"ClusterRole\",\"name\":\"aws-load-balancer-controller-role\"},\"subjects\":[{\"kind\":\"ServiceAccount\",\"name\":\"aws-load-balancer-controller\",\"namespace\":\"kube-system\"}]},\"kube-system/deployment.apps/apps/v1/aws-load-balancer-controller\":{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller\",\"namespace\":\"kube-system\"},\"spec\":{\"replicas\":2,\"selector\":{\"matchLabels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\"}},\"template\":{\"metadata\":{\"annotations\":{\"prometheus.io/port\":\"8080\",\"prometheus.io/scrape\":\"true\"},\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\"}},\"spec\":{\"affinity\":{\"podAntiAffinity\":{\"preferredDuringSchedulingIgnoredDuringExecution\":[{\"podAffinityTerm\":{\"labelSelector\":{\"matchExpressions\":[{\"key\":\"app.kubernetes.io/name\",\"operator\":\"In\",\"values\":[\"aws-load-balancer-controller\"]}]},\"topologyKey\":\"kubernetes.io/hostname\"},\"weight\":100}]}},\"containers\":[{\"args\":[\"--cluster-name=kubernetes01\",\"--ingress-class=alb\",\"--aws-region=us-east-1\",\"--aws-vpc-id=vpc-REDACTED\"],\"command\":[\"/controller\"],\"image\":\"602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-load-balancer-controller:v2.2.3\",\"imagePullPolicy\":\"IfNotPresent\",\"livenessProbe\":{\"failureThreshold\":2,\"httpGet\":{\"path\":\"/healthz\",\"port\":61779,\"scheme\":\"HTTP\"},\"initialDelaySeconds\":30,\"timeoutSeconds\":10},\"name\":\"aws-load-balancer-controller\",\"ports\":[{\"containerPort\":9443,\"name\":\"webhook-server\",\"protocol\":\"TCP\"},{\"containerPort\":8080,\"name\":\"metrics-server\",\"protocol\":\"TCP\"}],\"resources\":{},\"securityContext\":{\"allowPrivilegeEscalation\":false,\"readOnlyRootFilesystem\":true,\"runAsNonRoot\":true},\"volumeMounts\":[{\"mountPath\":\"/tmp/k8s-webhook-server/serving-certs\",\"name\":\"cert\",\"readOnly\":true}]}],\"priorityClassName\":\"system-cluster-critical\",\"securityContext\":{\"fsGroup\":65534},\"serviceAccountName\":\"aws-load-balancer-controller\",\"terminationGracePeriodSeconds\":10,\"volumes\":[{\"name\":\"cert\",\"secret\":{\"defaultMode\":420,\"secretName\":\"aws-load-balancer-tls\"}}]}}}},\"kube-system/role.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1/aws-load-balancer-controller-leader-election-role\":{\"apiVersion\":\"rbac.authorization.k8s.io/v1\",\"kind\":\"Role\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller-leader-election-role\",\"namespace\":\"kube-system\"},\"rules\":[{\"apiGroups\":[\"\"],\"resources\":[\"configmaps\"],\"verbs\":[\"create\"]},{\"apiGroups\":[\"\"],\"resourceNames\":[\"aws-load-balancer-controller-leader\"],\"resources\":[\"configmaps\"],\"verbs\":[\"get\",\"patch\",\"update\"]}]},\"kube-system/rolebinding.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1/aws-load-balancer-controller-leader-election-rolebinding\":{\"apiVersion\":\"rbac.authorization.k8s.io/v1\",\"kind\":\"RoleBinding\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller-leader-election-rolebinding\",\"namespace\":\"kube-system\"},\"roleRef\":{\"apiGroup\":\"rbac.authorization.k8s.io\",\"kind\":\"Role\",\"name\":\"aws-load-balancer-controller-leader-election-role\"},\"subjects\":[{\"kind\":\"ServiceAccount\",\"name\":\"aws-load-balancer-controller\",\"namespace\":\"kube-system\"}]},\"kube-system/secret/v1/aws-load-balancer-tls\":{\"kind\":\"Secret\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"aws-load-balancer-tls\",\"namespace\":\"kube-system\",\"creationTimestamp\":null,\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"}},\"data\":{\"ca.crt\":\"KHNlbnNpdGl2ZSB2YWx1ZSA5YzljYjg3YjliMDliOTYzKQ==\",\"tls.crt\":\"KHNlbnNpdGl2ZSB2YWx1ZSAzNzhlYTFjODQ3OTRkMWRlKQ==\",\"tls.key\":\"KHNlbnNpdGl2ZSB2YWx1ZSA5Njk1MDMzNjlhYzE4YTJhKQ==\"},\"type\":\"kubernetes.io/tls\"},\"kube-system/service/v1/aws-load-balancer-webhook-service\":{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-webhook-service\",\"namespace\":\"kube-system\"},\"spec\":{\"ports\":[{\"port\":443,\"targetPort\":\"webhook-server\"}],\"selector\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\"}}},\"kube-system/serviceaccount/v1/aws-load-balancer-controller\":{\"apiVersion\":\"v1\",\"automountServiceAccountToken\":true,\"kind\":\"ServiceAccount\",\"metadata\":{\"annotations\":{\"eks.amazonaws.com/role-arn\":\"arn:aws:iam::REDACTED:role/REDACTED\"},\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-controller\",\"namespace\":\"kube-system\"}},\"mutatingwebhookconfiguration.admissionregistration.k8s.io/admissionregistration.k8s.io/v1/aws-load-balancer-webhook\":{\"apiVersion\":\"admissionregistration.k8s.io/v1\",\"kind\":\"MutatingWebhookConfiguration\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-webhook\"},\"webhooks\":[{\"admissionReviewVersions\":[\"v1beta1\"],\"clientConfig\":{\"caBundle\":\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURQekNDQWllZ0F3SUJBZ0lRTTk0MXA0bHpuMXRhV0Jra2hiREozVEFOQmdrcWhraUc5dzBCQVFzRkFEQXEKTVNnd0pnWURWUVFERXg5aGQzTXRiRzloWkMxaVlXeGhibU5sY2kxamIyNTBjbTlzYkdWeUxXTmhNQjRYRFRJeApNVEl3TVRFNU5EQXlNMW9YRFRNeE1URXlPVEU1TkRBeU0xb3dLakVvTUNZR0ExVUVBeE1mWVhkekxXeHZZV1F0ClltRnNZVzVqWlhJdFkyOXVkSEp2Ykd4bGNpMWpZVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0MKQVFvQ2dnRUJBTUY2QkhUNmtaMjJZek0rTjNoVWdGUzJyaUdpaHRrTkp4MGd4RnVidVZJaHRjWkZ3VGdUVWU1MApWd3BRV3dDeXFmenNSNUI0SWY3VjcxMk1SaUordzZzaWl5d2l3bURzN25DSTBNUFlyL2RYZmhTTGFka3BNcHFJCnByRi9abjFKODgxN0dxcG8rckE2ZTg0cWFtT2lTMWFLUmUzSXdFMHg2MzZwM2l0MFNRTmtkWStpU1dXMUs3WFIKb054OFlVUFNKWlVweExiTlA2TVVwbklZc283NzdFQzN5SDc0RElhTWxRdEpuMTFiendTTzJBL2thQ0NlMTZZSwpUcTNhM3VQOStVZ0hxQ3M2S2Qxa0k4cTJqdzBrWUtBV3RTTGxXZHNweVZEU1lRaFVHVVY2VE8wRUZ4WHlYUUdjCkkyaFlkNEZFNXZrODhZdlNuMzk0YXM2UUdoWm1NdmtDQXdFQUFhTmhNRjh3RGdZRFZSMFBBUUgvQkFRREFnS2sKTUIwR0ExVWRKUVFXTUJRR0NDc0dBUVVGQndNQkJnZ3JCZ0VGQlFjREFqQVBCZ05WSFJNQkFmOEVCVEFEQVFILwpNQjBHQTFVZERnUVdCQlRQcVpwOUZIMlNBRE5HWFVDV2hRSmszN0xxMXpBTkJna3Foa2lHOXcwQkFRc0ZBQU9DCkFRRUF2U25sSTBpVzIyYlJiZWZoMjlzY1ZpeDR5am5qamRSVXhvSFJ0UU9jTlA1eWtTSGU2dnJHK3dabFpkRlIKZjZzeHpkNVk5NnZnZ2o1eWNaa0pmUEhISnpRczliSThEV0lmdG5HeGVLQ2FPeHROT1VhL2lRYU5EK3ZJanozSgpod2hPRXptanJxR2dMWW9Fb3ZHZnBteGkwQ3lua3hLdkFMZWdoNDgyRkFDci9INjJMU1YrN1ZtZkxTV2UrQlk1CkJVVnNFQUVyNWdiQ1l2QjdjdjVVcUU5QXNyMmNqdUtoUURnalhndUZaVFVCOWl0T1hxUEptNG1WKzlzMnd6Nm0KNVJHTUFZSllMdjI0MUZVZVg5R3hjTkVzdWVJdWdlOXZmUEJFQUFKelRlbWdVaC9qT2JyeUdJRC9YKzZzNzlrZApEc1JTeFhGWWVKVW1tYTlySTVibEFQTU50QT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K\",\"service\":{\"name\":\"aws-load-balancer-webhook-service\",\"namespace\":\"kube-system\",\"path\":\"/mutate-v1-pod\"}},\"failurePolicy\":\"Fail\",\"name\":\"mpod.elbv2.k8s.aws\",\"namespaceSelector\":{\"matchExpressions\":[{\"key\":\"elbv2.k8s.aws/pod-readiness-gate-inject\",\"operator\":\"In\",\"values\":[\"enabled\"]}]},\"objectSelector\":{\"matchExpressions\":[{\"key\":\"app.kubernetes.io/name\",\"operator\":\"NotIn\",\"values\":[\"aws-load-balancer-controller\"]}]},\"rules\":[{\"apiGroups\":[\"\"],\"apiVersions\":[\"v1\"],\"operations\":[\"CREATE\"],\"resources\":[\"pods\"]}],\"sideEffects\":\"None\"},{\"admissionReviewVersions\":[\"v1beta1\"],\"clientConfig\":{\"caBundle\":\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURQekNDQWllZ0F3SUJBZ0lRTTk0MXA0bHpuMXRhV0Jra2hiREozVEFOQmdrcWhraUc5dzBCQVFzRkFEQXEKTVNnd0pnWURWUVFERXg5aGQzTXRiRzloWkMxaVlXeGhibU5sY2kxamIyNTBjbTlzYkdWeUxXTmhNQjRYRFRJeApNVEl3TVRFNU5EQXlNMW9YRFRNeE1URXlPVEU1TkRBeU0xb3dLakVvTUNZR0ExVUVBeE1mWVhkekxXeHZZV1F0ClltRnNZVzVqWlhJdFkyOXVkSEp2Ykd4bGNpMWpZVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0MKQVFvQ2dnRUJBTUY2QkhUNmtaMjJZek0rTjNoVWdGUzJyaUdpaHRrTkp4MGd4RnVidVZJaHRjWkZ3VGdUVWU1MApWd3BRV3dDeXFmenNSNUI0SWY3VjcxMk1SaUordzZzaWl5d2l3bURzN25DSTBNUFlyL2RYZmhTTGFka3BNcHFJCnByRi9abjFKODgxN0dxcG8rckE2ZTg0cWFtT2lTMWFLUmUzSXdFMHg2MzZwM2l0MFNRTmtkWStpU1dXMUs3WFIKb054OFlVUFNKWlVweExiTlA2TVVwbklZc283NzdFQzN5SDc0RElhTWxRdEpuMTFiendTTzJBL2thQ0NlMTZZSwpUcTNhM3VQOStVZ0hxQ3M2S2Qxa0k4cTJqdzBrWUtBV3RTTGxXZHNweVZEU1lRaFVHVVY2VE8wRUZ4WHlYUUdjCkkyaFlkNEZFNXZrODhZdlNuMzk0YXM2UUdoWm1NdmtDQXdFQUFhTmhNRjh3RGdZRFZSMFBBUUgvQkFRREFnS2sKTUIwR0ExVWRKUVFXTUJRR0NDc0dBUVVGQndNQkJnZ3JCZ0VGQlFjREFqQVBCZ05WSFJNQkFmOEVCVEFEQVFILwpNQjBHQTFVZERnUVdCQlRQcVpwOUZIMlNBRE5HWFVDV2hRSmszN0xxMXpBTkJna3Foa2lHOXcwQkFRc0ZBQU9DCkFRRUF2U25sSTBpVzIyYlJiZWZoMjlzY1ZpeDR5am5qamRSVXhvSFJ0UU9jTlA1eWtTSGU2dnJHK3dabFpkRlIKZjZzeHpkNVk5NnZnZ2o1eWNaa0pmUEhISnpRczliSThEV0lmdG5HeGVLQ2FPeHROT1VhL2lRYU5EK3ZJanozSgpod2hPRXptanJxR2dMWW9Fb3ZHZnBteGkwQ3lua3hLdkFMZWdoNDgyRkFDci9INjJMU1YrN1ZtZkxTV2UrQlk1CkJVVnNFQUVyNWdiQ1l2QjdjdjVVcUU5QXNyMmNqdUtoUURnalhndUZaVFVCOWl0T1hxUEptNG1WKzlzMnd6Nm0KNVJHTUFZSllMdjI0MUZVZVg5R3hjTkVzdWVJdWdlOXZmUEJFQUFKelRlbWdVaC9qT2JyeUdJRC9YKzZzNzlrZApEc1JTeFhGWWVKVW1tYTlySTVibEFQTU50QT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K\",\"service\":{\"name\":\"aws-load-balancer-webhook-service\",\"namespace\":\"kube-system\",\"path\":\"/mutate-elbv2-k8s-aws-v1beta1-targetgroupbinding\"}},\"failurePolicy\":\"Fail\",\"name\":\"mtargetgroupbinding.elbv2.k8s.aws\",\"rules\":[{\"apiGroups\":[\"elbv2.k8s.aws\"],\"apiVersions\":[\"v1beta1\"],\"operations\":[\"CREATE\",\"UPDATE\"],\"resources\":[\"targetgroupbindings\"]}],\"sideEffects\":\"None\"}]},\"validatingwebhookconfiguration.admissionregistration.k8s.io/admissionregistration.k8s.io/v1/aws-load-balancer-webhook\":{\"apiVersion\":\"admissionregistration.k8s.io/v1\",\"kind\":\"ValidatingWebhookConfiguration\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"aws-load-balancer-controller\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"aws-load-balancer-controller\",\"app.kubernetes.io/version\":\"v2.2.3\",\"helm.sh/chart\":\"aws-load-balancer-controller-1.2.6\"},\"name\":\"aws-load-balancer-webhook\"},\"webhooks\":[{\"admissionReviewVersions\":[\"v1beta1\"],\"clientConfig\":{\"caBundle\":\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURQekNDQWllZ0F3SUJBZ0lRTTk0MXA0bHpuMXRhV0Jra2hiREozVEFOQmdrcWhraUc5dzBCQVFzRkFEQXEKTVNnd0pnWURWUVFERXg5aGQzTXRiRzloWkMxaVlXeGhibU5sY2kxamIyNTBjbTlzYkdWeUxXTmhNQjRYRFRJeApNVEl3TVRFNU5EQXlNMW9YRFRNeE1URXlPVEU1TkRBeU0xb3dLakVvTUNZR0ExVUVBeE1mWVhkekxXeHZZV1F0ClltRnNZVzVqWlhJdFkyOXVkSEp2Ykd4bGNpMWpZVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0MKQVFvQ2dnRUJBTUY2QkhUNmtaMjJZek0rTjNoVWdGUzJyaUdpaHRrTkp4MGd4RnVidVZJaHRjWkZ3VGdUVWU1MApWd3BRV3dDeXFmenNSNUI0SWY3VjcxMk1SaUordzZzaWl5d2l3bURzN25DSTBNUFlyL2RYZmhTTGFka3BNcHFJCnByRi9abjFKODgxN0dxcG8rckE2ZTg0cWFtT2lTMWFLUmUzSXdFMHg2MzZwM2l0MFNRTmtkWStpU1dXMUs3WFIKb054OFlVUFNKWlVweExiTlA2TVVwbklZc283NzdFQzN5SDc0RElhTWxRdEpuMTFiendTTzJBL2thQ0NlMTZZSwpUcTNhM3VQOStVZ0hxQ3M2S2Qxa0k4cTJqdzBrWUtBV3RTTGxXZHNweVZEU1lRaFVHVVY2VE8wRUZ4WHlYUUdjCkkyaFlkNEZFNXZrODhZdlNuMzk0YXM2UUdoWm1NdmtDQXdFQUFhTmhNRjh3RGdZRFZSMFBBUUgvQkFRREFnS2sKTUIwR0ExVWRKUVFXTUJRR0NDc0dBUVVGQndNQkJnZ3JCZ0VGQlFjREFqQVBCZ05WSFJNQkFmOEVCVEFEQVFILwpNQjBHQTFVZERnUVdCQlRQcVpwOUZIMlNBRE5HWFVDV2hRSmszN0xxMXpBTkJna3Foa2lHOXcwQkFRc0ZBQU9DCkFRRUF2U25sSTBpVzIyYlJiZWZoMjlzY1ZpeDR5am5qamRSVXhvSFJ0UU9jTlA1eWtTSGU2dnJHK3dabFpkRlIKZjZzeHpkNVk5NnZnZ2o1eWNaa0pmUEhISnpRczliSThEV0lmdG5HeGVLQ2FPeHROT1VhL2lRYU5EK3ZJanozSgpod2hPRXptanJxR2dMWW9Fb3ZHZnBteGkwQ3lua3hLdkFMZWdoNDgyRkFDci9INjJMU1YrN1ZtZkxTV2UrQlk1CkJVVnNFQUVyNWdiQ1l2QjdjdjVVcUU5QXNyMmNqdUtoUURnalhndUZaVFVCOWl0T1hxUEptNG1WKzlzMnd6Nm0KNVJHTUFZSllMdjI0MUZVZVg5R3hjTkVzdWVJdWdlOXZmUEJFQUFKelRlbWdVaC9qT2JyeUdJRC9YKzZzNzlrZApEc1JTeFhGWWVKVW1tYTlySTVibEFQTU50QT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K\",\"service\":{\"name\":\"aws-load-balancer-webhook-service\",\"namespace\":\"kube-system\",\"path\":\"/validate-elbv2-k8s-aws-v1beta1-targetgroupbinding\"}},\"failurePolicy\":\"Fail\",\"name\":\"vtargetgroupbinding.elbv2.k8s.aws\",\"rules\":[{\"apiGroups\":[\"elbv2.k8s.aws\"],\"apiVersions\":[\"v1beta1\"],\"operations\":[\"CREATE\",\"UPDATE\"],\"resources\":[\"targetgroupbindings\"]}],\"sideEffects\":\"None\"},{\"admissionReviewVersions\":[\"v1beta1\"],\"clientConfig\":{\"caBundle\":\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURQekNDQWllZ0F3SUJBZ0lRTTk0MXA0bHpuMXRhV0Jra2hiREozVEFOQmdrcWhraUc5dzBCQVFzRkFEQXEKTVNnd0pnWURWUVFERXg5aGQzTXRiRzloWkMxaVlXeGhibU5sY2kxamIyNTBjbTlzYkdWeUxXTmhNQjRYRFRJeApNVEl3TVRFNU5EQXlNMW9YRFRNeE1URXlPVEU1TkRBeU0xb3dLakVvTUNZR0ExVUVBeE1mWVhkekxXeHZZV1F0ClltRnNZVzVqWlhJdFkyOXVkSEp2Ykd4bGNpMWpZVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0MKQVFvQ2dnRUJBTUY2QkhUNmtaMjJZek0rTjNoVWdGUzJyaUdpaHRrTkp4MGd4RnVidVZJaHRjWkZ3VGdUVWU1MApWd3BRV3dDeXFmenNSNUI0SWY3VjcxMk1SaUordzZzaWl5d2l3bURzN25DSTBNUFlyL2RYZmhTTGFka3BNcHFJCnByRi9abjFKODgxN0dxcG8rckE2ZTg0cWFtT2lTMWFLUmUzSXdFMHg2MzZwM2l0MFNRTmtkWStpU1dXMUs3WFIKb054OFlVUFNKWlVweExiTlA2TVVwbklZc283NzdFQzN5SDc0RElhTWxRdEpuMTFiendTTzJBL2thQ0NlMTZZSwpUcTNhM3VQOStVZ0hxQ3M2S2Qxa0k4cTJqdzBrWUtBV3RTTGxXZHNweVZEU1lRaFVHVVY2VE8wRUZ4WHlYUUdjCkkyaFlkNEZFNXZrODhZdlNuMzk0YXM2UUdoWm1NdmtDQXdFQUFhTmhNRjh3RGdZRFZSMFBBUUgvQkFRREFnS2sKTUIwR0ExVWRKUVFXTUJRR0NDc0dBUVVGQndNQkJnZ3JCZ0VGQlFjREFqQVBCZ05WSFJNQkFmOEVCVEFEQVFILwpNQjBHQTFVZERnUVdCQlRQcVpwOUZIMlNBRE5HWFVDV2hRSmszN0xxMXpBTkJna3Foa2lHOXcwQkFRc0ZBQU9DCkFRRUF2U25sSTBpVzIyYlJiZWZoMjlzY1ZpeDR5am5qamRSVXhvSFJ0UU9jTlA1eWtTSGU2dnJHK3dabFpkRlIKZjZzeHpkNVk5NnZnZ2o1eWNaa0pmUEhISnpRczliSThEV0lmdG5HeGVLQ2FPeHROT1VhL2lRYU5EK3ZJanozSgpod2hPRXptanJxR2dMWW9Fb3ZHZnBteGkwQ3lua3hLdkFMZWdoNDgyRkFDci9INjJMU1YrN1ZtZkxTV2UrQlk1CkJVVnNFQUVyNWdiQ1l2QjdjdjVVcUU5QXNyMmNqdUtoUURnalhndUZaVFVCOWl0T1hxUEptNG1WKzlzMnd6Nm0KNVJHTUFZSllMdjI0MUZVZVg5R3hjTkVzdWVJdWdlOXZmUEJFQUFKelRlbWdVaC9qT2JyeUdJRC9YKzZzNzlrZApEc1JTeFhGWWVKVW1tYTlySTVibEFQTU50QT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K\",\"service\":{\"name\":\"aws-load-balancer-webhook-service\",\"namespace\":\"kube-system\",\"path\":\"/validate-networking-v1beta1-ingress\"}},\"failurePolicy\":\"Fail\",\"matchPolicy\":\"Equivalent\",\"name\":\"vingress.elbv2.k8s.aws\",\"rules\":[{\"apiGroups\":[\"networking.k8s.io\"],\"apiVersions\":[\"v1beta1\"],\"operations\":[\"CREATE\",\"UPDATE\"],\"resources\":[\"ingresses\"]}],\"sideEffects\":\"None\"}]}}"

Important Factoids

References

• Compare currently deployed helm release metadata to decide if the helm_release should be updated or not #523

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[nitrocode]

I get the same issue with the datadog-agent helm chart.

│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.datadog_agent.helm_release.this[0] to
│ include new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/helm" produced an invalid new value for
│ .manifest: was

...

│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

[jw-maynard]

I did some digging into my issue today and was able to diff the Plan manifests against the Apply manifests to find out what's inconsistent. In my case the aws-load-balancer-controller chart will generate a cert bundle for it's webhook when you run the chart. It looks like the helm is being run during the plan and then again during apply and since it's run twice these auto generated certs are being created twice and therefore have different values which causes the inconstancy.

[jw-maynard]

@BBBmau @jrhouston I don't know if this is feasible, but maybe the plan step could store a complete copy of the Helm values in the plan and then during apply the provider would feed a full set of values into Helm. Also not sure if this strategy would have undesirable side effects in helm.

[github-actions]

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!

[brandon-gradient]

This does seem to be connected to the experimental manifest feature when using helm charts that have some random generation for output values. In addition to the same aws-load-balancer-controller chart generating differing cert bundles, I also ran into this with the grafana/grafana chart and using the basic auth. It would generate a random password on each run of the chart which would differ when the command is run again. It seems like the manifest feature's output would need to be passed on to helm wholesale similar to a terraform apply "plan.output" command

[TheKangaroo]

I ran into this with another chart using the randNumeric function, but I replaced it with the now function and got the same error.
At this point I'm pretty sure it's impossible to combine the experiments { manifest = true } feature and any helm function that produces different outputs for two subsequent plans.
This is, in my opinion, a major drawback of the provider at the moment. It should at least be mentioned in the documentation of this experimental feature.
On the other hand, I don't know how anyone would use the helm provider without the experimental feature enabled, since you can never tell from the plan what will actually change in your cluster.
I think the information about this issue is a bit scattered over a lot of issues and mixed with other (already solved) issues with similar symptoms.
I hope this issue gets some attention from the maintainers, as it is a showstopper for many use cases.