(eks): helm install - kube config is insecure

[mrsiejas]

When deploying stack with helm chart, getting the following error related to insecure kube config created by Lambda:

9:45:21 AM | CREATE_FAILED        | Custom::AWSCDK-EKS-HelmChart          | FluentBit/Resource/Default
Received response status [FAILED] from custom resource. Message returned: Error: b'WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /tmp/kubeconfig\nWARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /tmp/kubeconfig\nError: UPGRADE FAILED: an
other operation (install/upgrade/rollback) is in progress\n'

Code snippet:

        chart = self.add_chart(
            id='FluentBit',
            chart='fluent-bit',
            namespace='monitoring',
            repository='https://charts.helm.sh/stable',
            release='fluent-bit',
            values={
                'existingConfigMap': 'fluent-bit-config',
                'image': {
                    'fluent_bit': {
                        'tag': '1.5.6'
                    }
                },
                'parsers': {
                    'enabled': True
                },
                'serviceAccount': {
                    'create': False,
                    'name': service_account.service_account_name
                },
                'tolerations': [
                    {
                        'operator': 'Exists'
                    }
                ],
                'priorityClassName': 'system-node-critical'
            },
            version='2.10.0',
            wait=True,
            timeout=core.Duration.minutes(15)
        )

Reproduction Steps

1. running on CDK v1.93, intend to update existing fluentbit chart deployed on EKS
2. run cdk deploy <stack_name_with_helm_chart>

What did you expect to happen?

Expect Helm chart to install and update helm chart successfully. Alternatively be able to suppress or ignore this warning (which is sent to stderr).

What actually happened?

cdk deploy fails and I'm not able to deploy helm on EKS.

Environment

• CDK CLI Version : 1.93
• **Framework Version: **
• Node.js Version: v10.24.0
• OS : OSX (via Docker container, image: python:3.8.5-slim-buster)
• Language (Version): Python (3.8.5)

Other

Can be related to this change: helm/helm#8779 where warning messages are being sent to stderr rather than stdout.

---

This is 🐛 Bug Report

[iliapolo]

@mrsiejas I don't think those warnings are whats causing the failure. I deployed the dashboard using helm:

cluster.addHelmChart('dashboard', {
  chart: 'kubernetes-dashboard',
  repository: 'https://kubernetes.github.io/dashboard/',
});

And it successfully despite those warnings:

[INFO]	2021-05-05T10:25:13.417Z	a5811ffd-8493-4ea7-b3ed-eaabebc456e6	b'WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /tmp/kubeconfig\nWARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /tmp/kubeconfig\nRelease "eksclusterchartdashboarde22e314d" does not exist. Installing it now.\nNAME: eksclusterchartdashboarde22e314d\nLAST DEPLOYED: Wed May  5 10:25:12 2021\nNAMESPACE: default\nSTATUS: deployed\nREVISION: 1\nTEST SUITE: None\nNOTES:\n*********************************************************************************\n*** PLEASE BE PATIENT: kubernetes-dashboard may take a few minutes to install ***\n*********************************************************************************\n\nGet the Kubernetes Dashboard URL by running:\n  export POD_NAME=$(kubectl get pods -n default -l "app.kubernetes.io/name=kubernetes-dashboard,app.kubernetes.io/instance=eksclusterchartdashboarde22e314d" -o jsonpath="{.items[0].metadata.name}")\n  echo https://127.0.0.1:8443/\n  kubectl -n default port-forward $POD_NAME 8443:8443\n'

Note that the log you posted also contains:

UPGRADE FAILED: an
other operation (install/upgrade/rollback) is in progress\n'

Which I'm fairly certain is the real culprit. Looks like the previous installation must not have finished yet? You'll have to debug that a bit further.

[mrsiejas]

I've destroyed the stack. Rerun cdk deploy and still getting the error

Received response status [FAILED] from custom resource. Message returned: Error: b'WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /tmp/kubeconfig\nWARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /tmp/kubeconfig\nError: UPGRADE FAILED: an
other operation (install/upgrade/rollback) is in progress\n'

6:27:53 AM | CREATE_FAILED        | Custom::AWSCDK-EKS-HelmChart          | FluentBit532C5947
Received response status [FAILED] from custom resource. Message returned: Error: b'WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /tmp/kubeconfig\nWARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /tmp/kubeconfig\nError: UPGRADE FAILED: an
other operation (install/upgrade/rollback) is in progress\n'

Logs: /aws/lambda/integration-eks-cluster-awscdkawse-Handler886CB40B-1BNX6NZIHY5GW

at invokeUserFunction (/var/task/framework.js:95:19)
at process._tickCallback (internal/process/next_tick.js:68:7) (RequestId: b547c9f3-8f30-4ead-9306-7726784727d3)

        new CustomResource (/tmp/jsii-kernel-f6qOc1/node_modules/@aws-cdk/core/lib/custom-resource.js:30:25)
        \_ new HelmChart (/tmp/jsii-kernel-f6qOc1/node_modules/@aws-cdk/aws-eks/lib/helm-chart.js:35:9)
        \_ obj._wrapSandboxCode (/tmp/tmpy0rhunmt/lib/program.js:8154:58)

Maybe this is a race condition somewhere in the stack?

other operation (install/upgrade/rollback) is in progress\n'

The odd thing is that the same stack was deploying fine until recently, the changes made were bumping CDK version from 1.74 -> 1.93 and upgrading EKS to 1.19.

[mrsiejas]

I think you were right @iliapolo. The warning was not causing the chart to fail, it was a problem with Fluentbit immutable selector and the fact I was trying to update it via cdk deploy. Surprisingly cdk destroy didn't remove the old Helm chart.

Still, I do believe the kube config permissions warning should be addressed and fixed for clarity. Having multiple unrelated errors in the output makes it more difficult to troubleshoot the actual issue. Thanks for looking into it.

[iliapolo]

Opened #14560 to follow up on the warnings. Resolving this one.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.