Skip to content

Latest commit

 

History

History
234 lines (198 loc) · 10.3 KB

README.md

File metadata and controls

234 lines (198 loc) · 10.3 KB
Raw

Default configurations for Windows images in Amazon ECS

Using Fluent Bit on Amazon ECS Windows in a Daemon configuration using forward input plugin

In order to allow Amazon ECS Windows customers to get started promptly with Fluent Bit without performing significant configuration, we have crafted default configurations for the following output destinations-

These default configurations are baked in the Windows container images with image versions 2.28.4 and above. These can be used while deploying Fluent Bit using forward input plugin as an Amazon ECS service with Daemon scheduling strategy.

In order to use the same out of the box with Amazon ECS, you need to ensure-

  • Your task IAM role has sufficient permissions as mentioned in the Fluent Bit output plugin documentation.

  • You have overridden the entrypoint of the Windows Fluent Bit image with the specific configuration which you want to use.

  • You have set the environment variables required for the specific configuration as documented later in this guide.

  • You have configured your Amazon ECS task definition for your application tasks to use fluentd logging driver and the following labels are configured in labels option of fluentd docker logging driver-

    • com.amazonaws.ecs.cluster
    • com.amazonaws.ecs.container-name
    • com.amazonaws.ecs.task-arn
    • com.amazonaws.ecs.task-definition-family
    • com.amazonaws.ecs.task-definition-version

These labels are added by default to each container by Amazon ECS agent.

Architecture

In order to setup Fluent Bit on Amazon ECS as daemon, we would use the following approach-

  • Run Fluent Bit as a daemon container on the instance such that:
    • It would listen on port 24224 using the forward input plug-in.
    • We would expose the port 24224 to the host so that docker runtime can send logs to Fluent Bit using this exposed port.
    • Fluent Bit would have it’s own configuration which would enable it to send the logs records to specified destinations.
    • For production usage, we suggest running Fluent Bit task as an Amazon ECS Service with Daemon scheduling strategy. That would ensure that a single instance of Fluent Bit always runs on your container instances in the cluster.
  • All other Amazon ECS task containers will be launched using the fluentd docker logging driver.
    • Docker would connect to the TCP socket 24224 on localhost inside the host namespace.
    • Amazon ECS agent adds labels to the containers which includes cluster name, task definition family name, task definition revision number, task ARN, and the container name. We will add the same to the log record using labels option of the fluentd docker logging driver.
    • Because we are setting async option of the fluentd logging driver as true, even if the Fluent Bit container is restarted, docker would buffer the logs until Fluent Bit container is restarted. You can increase the buffer limit by setting the fluentd-buffer-limit option.

Therefore, in the end to end workflow:

  • Fluent Bit container will be started and would start listening on port 24224 which is exposed to the host.
  • Fluent Bit would use the Task IAM role credentials as specified in its task definition.
  • Other Amazon ECS tasks would be launched on the same instance which would use fluentd docker logging driver to connect to the Fluent Bit container on port 24224.
  • When logs are generated by application containers, docker runtime would tag those records, add additional metadata specified in labels, and then would forward them on port 24224 in the host namespace.
  • Fluent Bit would receive the log record on port 24224 since it is exposed to the host namespace.
  • Fluent Bit would perform its internal processing and would finally route the logs as specified.

Overriding the entrypoint for the Windows image

Windows Fluent Bit images can be configured to use different inbuilt configuration file by overriding the entrypoint of the container. For Amazon ECS customers, you need to add the same in Amazon ECS task container definition of Fluent Bit task.

{
  ...
  "containerDefinitions": [
    {
      ...
      
      "image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:windowsservercore-latest",
      
      "entryPoint": [
        "Powershell",
        "-Command"
      ],
      
      "command": [
        "C:\\entrypoint.ps1 -ConfigFile C:\\ecs_windows_forward_daemon\\kinesis.conf"
      ],
       
      "environment": [
        {
          "name": "AWS_REGION",
          "value": "us-west-2"
        }
      ],
      ...
    }
  ],
  ...
}

The new format of the entrypoint should be-

Powershell -Command C:\\entrypoint.ps1 -ConfigFile C:\\ecs_windows_forward_daemon\\xxxx.conf

The xxxx.conf can be among-

  • cloudwatch.conf
  • s3.conf
  • kinesis.conf
  • firehose.conf

Required environment variables

In order to use the default configurations, you need to set the destination specific environment variables in the container. These would be used to initialise your Fluent Bit configuration.

Amazon CloudWatch

AWS_REGION -> AWS region to be used for Amazon CloudWatch.

Amazon S3

BUCKET -> Amazon S3 bucket where the logs need to be stored.
AWS_REGION -> AWS region to be used for Amazon CloudWatch.

Amazon Kinesis Data Stream

STREAM -> The name of the Amazon Kinesis Delivery stream that you want log records sent to.
AWS_REGION -> AWS region to be used for Amazon Kinesis Streams.

Amazon Kinesis Data Firehose

DELIVERY_STREAM -> The name of the Amazon Kinesis Data Firehose Delivery stream that you want log records sent to.
AWS_REGION -> AWS region to be used for Amazon Kinesis Data Firehose.

Sample Fluent Bit task definition with overridden configuration

Change the following in this task definition-

  • task-iam-role: The IAM role to be used by the task
  • region: Region in which the CloudWatch logs need to be sent
{
  "family": "ecs-windows-fluent-bit",
  "taskRoleArn": "task-iam-role",
  "containerDefinitions": [
    {
      "name": "fluent-bit",
      "image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:windowsservercore-latest",
      "cpu": 512,
      "portMappings": [
        {
          "hostPort": 24224,
          "containerPort": 24224,
          "protocol": "tcp"
        }
      ],
      "entryPoint": [
        "Powershell",
        "-Command"
      ],
      "command": [
        "C:\\entrypoint.ps1 -ConfigFile C:\\ecs_windows_forward_daemon\\kinesis.conf"
      ],
      "environment": [
        {
          "name": "AWS_REGION",
          "value": "region"
        }
      ],
      "memory": 512,
      "essential": true,
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/fluent-bit-logs",
          "awslogs-region": "region",
          "awslogs-stream-prefix": "flb",
          "awslogs-create-group": "true"
        }
      }
    }
  ],
  "memory": "512",
  "cpu": "512"
}

Configuring Amazon ECS task definition to send the logs to Fluent Bit

In order to configure your Amazon ECS tasks to send the logs to Fluent Bit, you need to configure them using fluentd docker logging driver. A sample task definition which configures the same is-

{
  "family": "windows-iis-task",
  "containerDefinitions": [
    {
      "name": "sample-container",
      "image": "mcr.microsoft.com/windows/servercore/iis:windowsservercore",
      "cpu": 512,
      "memory": 512,
      "essential": true,
      "portMappings": [ {"protocol": "tcp", "containerPort": 80 } ],
      "logConfiguration": {
        "logDriver": "fluentd",
        "options": {
          "fluentd-address": "localhost:24224",
          "tag": "{{ index .ContainerLabels \"com.amazonaws.ecs.task-definition-family\" }}",
          "fluentd-async": "true",
          "labels": "com.amazonaws.ecs.cluster,com.amazonaws.ecs.container-name,com.amazonaws.ecs.task-arn,com.amazonaws.ecs.task-definition-family,com.amazonaws.ecs.task-definition-version"
        }
      }
    }
  ],
  "memory": "512",
  "cpu": "512"
}

Note: Amazon ECS agent adds labels to each container which can be used to distinguish them. These labels include-

  • com.amazonaws.ecs.cluster
  • com.amazonaws.ecs.container-name
  • com.amazonaws.ecs.task-arn
  • com.amazonaws.ecs.task-definition-family
  • com.amazonaws.ecs.task-definition-version

In our task definition, we configure fluentd logging driver to add these key-value pairs to the log records.

CloudWatch default configuration explanation

The default configuration of Amazon CloudWatch would-

  • Create a new log group for each Amazon ECS cluster and Amazon ECS task definition family
  • Create a new log stream for each task container in above generated log group whenever a new task is launched. Each stream will be marked with the task id to which the container belongs.
  • Add additional metadata such as Amazon ECS cluster name, Amazon ECS task ARN, Amazon ECS task container name, Amazon ECS task definition family, and the Amazon ECS task definition revision number in each log entry.

For example, if you have the following task definitions with the containers as:

ecs_task_1 -> container_1 and container_2
ecs_task_2 -> container_3

Assuming that these task definitions each have a single task running in an Amazon ECS cluster named windows, then we would get the following Amazon CloudWatch log groups and streams as below-

/aws/ecs/windows.ecs_task_1
    task-out.<TASK_ID>.container_1
    task-out.<TASK_ID>.container_2

/aws/ecs/windows.ecs_task_2
    task-out.<TASK_ID>.container_3