Fluentd-Elasticsearch helm chart

[jicowan]

I stumbled across this post as I was troubleshooting an issue I was having with the fluentd-elasticsearch daemonset. I used Helm to install it onto my cluster. I updated the chart with a kube2iam annotation that assigns the pod an IAM role that allows it to post to my ES cluster. However, when I try to specify the role in the ES access policy I see the following in the fluentd logs:

2018-12-21 05:41:18 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=12 next_retry_seconds=2018-12-21 05:41:51 +0000 chunk="57d81ab28c5dc9c86e0e22f1c7e44525" error_class=Elasticsearch::Transport::Transport::Errors::Forbidden error="[403] {\"Message\":\"User: anonymous is not authorized to perform: es:ESHttpPost\"}"

When I exec into the container and curl the metadata endpoint, I can see that the pod is getting the role I've assigned to it, but I still can't seem to post to ES.

My access policy looks like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "role-arn"
      },
      "Action": [
        "es:ESHttpPost",
        "es:ESHttpPut"
      ],
      "Resource": "resource-arn/domain/<domain-name>/*"
    }
  ]
}

Do I need to add something to the configmap that the chart create in order for fluentd to use the role assigned to the pod?

[cosmo0920]

Do I need to add something to the configmap that the chart create in order for fluentd to use the role assigned to the pod?

fluentd-elasticsearch daemonset does not sign its request.
This is because unable to push events into AWS Elasticsearch Serivce via fluentd-elasticsearch.

[nukepuppy]

@cosmo0920 what steps would one need to do to get THIS plugin to work with the helm chart (stable helm charts) located here: https://github.com/kiwigrid/helm-charts/tree/master/charts/fluentd-elasticsearch ?

do you have suggestions on this ?

[cosmo0920]

Again, fluent-plugin-elasticsearch cannot handle AWS required sigv4 requests.
If users want to send events into AWS Elasticsearch Service, we should create fluent-plugin-aws-elasticsearch-service deamonset support first.
Because fluent-plugin-elasticsearch and fluent-plugin-aws-elasticsearch-service plugins are different plugin.
(@type elasticsearch calls fluent-plugin-elasticsearch not fluent-plugin-aws-elasticsearch-service)
Then, request AWS Elasticsearch service support in helm chart.

[nukepuppy]

@cosmo0920 its clear it cannot handle that.. thanks again for clarification.

My question is how to resolve. As this is not exactly a rare use case.

I'm asking for the next steps to get AWS support .. in that helm chart..

you're saying we need:

1. "we should create fluent-plugin-aws-elasticsearch-service deamonset support first."
   --- who is we?
   --- what is level of effort?
2. request AWS Elasticsearch service support in helm chart.

are there other steps to do this? I appreciate your response

[cosmo0920]

1. "we should create fluent-plugin-aws-elasticsearch-service deamonset support first."
   --- who is we?

Someone who wants to create it. Perhaps, including you.

   --- what is level of effort?

I have no idea. I'm not familiar with creating helm chart from scratch.

2. request AWS Elasticsearch service support in helm chart.

helm chart is maintained in the other repository. Someone who wants to add AWS ES service charts should send a PR to support it.

[cosmo0920]

Check this fluent/fluentd-kubernetes-daemonset#317

[cosmo0920]

Check:

• https://hub.helm.sh/charts/kiwigrid/fluentd-elasticsearch

fluentd-elasticsearch helm chart now support this with sidecar AWS signer proxy container.