-
Notifications
You must be signed in to change notification settings - Fork 7
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feature(terraform): init terraform module
- Loading branch information
Showing
18 changed files
with
263 additions
and
69 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| * @schehata |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,6 @@ | ||
| /build | ||
| terraform.tfstate | ||
| terraform.tfstate.backup | ||
| .terraform.lock.hcl | ||
| .terraform | ||
| *.zip |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,65 +1,8 @@ | ||
| # Axiom CloudWatch Forwarder [](https://github.com/axiomhq/axiom-cloudwatch-forwarder/actions/workflows/ci.yaml) | ||
|
|
||
| Axiom CloudWatch Forwarder is a set of easy-to-use AWS CloudFormation stacks designed to forward logs from Amazon CloudWatch to [Axiom](https://axiom.co). It includes a Lambda function to handle the forwarding, as well as stacks to create CloudWatch log group subscription filters for both existing and future log groups. | ||
| Axiom CloudWatch Forwarder is a set of easy-to-use tools designed to forward logs from Amazon CloudWatch to [Axiom](https://axiom.co). It mainly includes a Lambda function to handle the forwarding. | ||
|
|
||
| Axiom CloudWatch Forwarder includes templates for the following CloudFormation stacks: | ||
| There are two ways to set up the CloudWatch log forwarding: | ||
|
|
||
| - **Forwarder** creates a Lambda function that forwards logs from CloudWatch to Axiom. | ||
| - **Subscriber** runs once to create subscription filters on _Forwarder_ for CloudWatch log groups specified by a combination of names, prefix, and regular expression. | ||
| - **Listener** creates a Lambda function that listens for new log groups and creates subscription filters for them on _Forwarder_. This way, you don't have to create subscription filters manually for new log groups. | ||
| - **Unsubscriber**: runs once to remove subscription filters on _Forwarder_ for CloudWatch log groups specified by a combination of names, prefix, and regular expression. | ||
|
|
||
| ## Setting up CloudWatch log group forwarding | ||
|
|
||
| 1. Create an Axiom organization at [app.axiom.co](https://app.axiom.co?ref=axiom-cloudwatch-forwarder). | ||
| 2. Create a dataset in Axiom. | ||
| 3. Create an API token in Axiom with permissions to ingest data to the dataset you created. | ||
| 4. [Click this link](https://console.aws.amazon.com/cloudformation/home?#/stacks/new?stackName=axiom-cloudwatch-forwarder&templateURL=https://axiom-cloudformation.s3.amazonaws.com/stacks/axiom-cloudwatch-forwarder-v1.1.1-cloudformation-stack.yaml) to open the _Forwarder_ stack template. | ||
| 5. Copy the _Forwarder_ Lambda ARN from the previous step, as it will be referenced in the _Subscriber_ stack. | ||
| 6. [Click this link](https://console.aws.amazon.com/cloudformation/home?#/stacks/new?stackName=axiom-cloudwatch-subscriber&templateURL=https://axiom-cloudformation.s3.amazonaws.com/stacks/axiom-cloudwatch-subscriber-v1.1.1-cloudformation-stack.yaml) to open the _Subscriber_ stack template. | ||
| 7. [Click this link](https://console.aws.amazon.com/cloudformation/home?#/stacks/new?stackName=axiom-cloudwatch-listener&templateURL=https://axiom-cloudformation.s3.amazonaws.com/stacks/axiom-cloudwatch-listener-v1.1.1-cloudformation-stack.yaml) to open the _Listener_ stack template. | ||
|
|
||
| For guidance on unsubscribing from log groups, please see [Removing subscription filters](#Removing-subscription-filters) section below. | ||
|
|
||
| ## Filtering CloudWatch log groups | ||
|
|
||
| The _Subscriber_ and _Unsubscriber_ stacks allow you to filter the log groups by a combination of names, prefix, and regular expression. | ||
|
|
||
| If no filters are specified, the stacks will subscribe to or unsubscribe from all log groups. You can also whitelist a specific set of log groups using filters in the CloudFormation stack parameters. | ||
|
|
||
| The log group names, prefix, and regular expression filters included are additive, meaning the union of all provided inputs will be matched. | ||
|
|
||
| ### Example | ||
|
|
||
| Assume we have the following list of log groups: | ||
|
|
||
| ``` | ||
| /aws/lambda/function-foo | ||
| /aws/lambda/function-bar | ||
| /aws/eks/cluster/cluster-1 | ||
| /aws/rds/instance-baz | ||
| ``` | ||
|
|
||
| - To subscribe to the Lambda log groups exclusively, a prefix filter with the value of `/aws/lambda` is a good choice. | ||
| - To subscribe to EKS and RDS log groups, a list of names with the value of `/aws/eks/cluster/cluster-1,/aws/rds/instance-baz` would work well. | ||
| - To subscribe to the EKS log group and all Lambda log groups, a combination of prefix and names list would select them. | ||
| - To use the regex filter, you can use a regular expression to match the log group names. For example, `\/aws\/lambda\/.*` would match all Lambda log groups. | ||
| - To subscribe to all log groups, leave the filters empty. | ||
|
|
||
| ## Listener architecture | ||
|
|
||
| The optional _Listener_ stack does the following: | ||
|
|
||
| - Creates an Amazon S3 bucket for AWS CloudTrail. | ||
| - Creates a trail to capture the creation of new log groups. | ||
| - Creates an event rule to pass those creation events to an Amazon EventBridge event bus. | ||
| - Sends an event via EventBridge to a Lambda function when a new log group is created. | ||
| - Creates a subscription filter for each new log group. | ||
|
|
||
| ## Removing subscription filters | ||
|
|
||
| If later on you want to remove subscription filters for one or more log groups: | ||
|
|
||
| - [Click this link](https://console.aws.amazon.com/cloudformation/home?#/stacks/new?stackName=axiom-cloudwatch-subscriber&templateURL=https://axiom-cloudformation.s3.amazonaws.com/stacks/axiom-cloudwatch-unsubscriber-v1.1.1-cloudformation-stack.yaml) to open the _Unsubscriber_ stack template. | ||
|
|
||
| The log group filtering works the same way as the _Subscriber_ stack. You can filter the log groups by a combination of names, prefix and regular expression. | ||
| - [A Terraform module](./module) that sets up the forwarding infrastructure and deploys the Lambda function. | ||
| - [Cloudformation Stacks](./cloudformation-stacks) that sets up the forwarding infrastructure and deploys the Lambda function, along with a Subscriber and Listener stacks. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| # Axiom CloudWatch Forwarder [](https://github.com/axiomhq/axiom-cloudwatch-forwarder/actions/workflows/ci.yaml) | ||
|
|
||
| Axiom CloudWatch Forwarder is a set of easy-to-use AWS CloudFormation stacks designed to forward logs from Amazon CloudWatch to [Axiom](https://axiom.co). It includes a Lambda function to handle the forwarding, as well as stacks to create CloudWatch log group subscription filters for both existing and future log groups. | ||
|
|
||
| Axiom CloudWatch Forwarder includes templates for the following CloudFormation stacks: | ||
|
|
||
| - **Forwarder** creates a Lambda function that forwards logs from CloudWatch to Axiom. | ||
| - **Subscriber** runs once to create subscription filters on _Forwarder_ for CloudWatch log groups specified by a combination of names, prefix, and regular expression. | ||
| - **Listener** creates a Lambda function that listens for new log groups and creates subscription filters for them on _Forwarder_. This way, you don't have to create subscription filters manually for new log groups. | ||
| - **Unsubscriber**: runs once to remove subscription filters on _Forwarder_ for CloudWatch log groups specified by a combination of names, prefix, and regular expression. | ||
|
|
||
| ## Setting up CloudWatch log group forwarding | ||
|
|
||
| 1. Create an Axiom organization at [app.axiom.co](https://app.axiom.co?ref=axiom-cloudwatch-forwarder). | ||
| 2. Create a dataset in Axiom. | ||
| 3. Create an API token in Axiom with permissions to ingest data to the dataset you created. | ||
| 4. [Click this link](https://console.aws.amazon.com/cloudformation/home?#/stacks/new?stackName=axiom-cloudwatch-forwarder&templateURL=https://axiom-cloudformation.s3.amazonaws.com/stacks/axiom-cloudwatch-forwarder-v1.1.1-cloudformation-stack.yaml) to open the _Forwarder_ stack template. | ||
| 5. Copy the _Forwarder_ Lambda ARN from the previous step, as it will be referenced in the _Subscriber_ stack. | ||
| 6. [Click this link](https://console.aws.amazon.com/cloudformation/home?#/stacks/new?stackName=axiom-cloudwatch-subscriber&templateURL=https://axiom-cloudformation.s3.amazonaws.com/stacks/axiom-cloudwatch-subscriber-v1.1.1-cloudformation-stack.yaml) to open the _Subscriber_ stack template. | ||
| 7. [Click this link](https://console.aws.amazon.com/cloudformation/home?#/stacks/new?stackName=axiom-cloudwatch-listener&templateURL=https://axiom-cloudformation.s3.amazonaws.com/stacks/axiom-cloudwatch-listener-v1.1.1-cloudformation-stack.yaml) to open the _Listener_ stack template. | ||
|
|
||
| For guidance on unsubscribing from log groups, please see [Removing subscription filters](#Removing-subscription-filters) section below. | ||
|
|
||
| ## Filtering CloudWatch log groups | ||
|
|
||
| The _Subscriber_ and _Unsubscriber_ stacks allow you to filter the log groups by a combination of names, prefix, and regular expression. | ||
|
|
||
| If no filters are specified, the stacks will subscribe to or unsubscribe from all log groups. You can also whitelist a specific set of log groups using filters in the CloudFormation stack parameters. | ||
|
|
||
| The log group names, prefix, and regular expression filters included are additive, meaning the union of all provided inputs will be matched. | ||
|
|
||
| ### Example | ||
|
|
||
| Assume we have the following list of log groups: | ||
|
|
||
| ``` | ||
| /aws/lambda/function-foo | ||
| /aws/lambda/function-bar | ||
| /aws/eks/cluster/cluster-1 | ||
| /aws/rds/instance-baz | ||
| ``` | ||
|
|
||
| - To subscribe to the Lambda log groups exclusively, a prefix filter with the value of `/aws/lambda` is a good choice. | ||
| - To subscribe to EKS and RDS log groups, a list of names with the value of `/aws/eks/cluster/cluster-1,/aws/rds/instance-baz` would work well. | ||
| - To subscribe to the EKS log group and all Lambda log groups, a combination of prefix and names list would select them. | ||
| - To use the regex filter, you can use a regular expression to match the log group names. For example, `\/aws\/lambda\/.*` would match all Lambda log groups. | ||
| - To subscribe to all log groups, leave the filters empty. | ||
|
|
||
| ## Listener architecture | ||
|
|
||
| The optional _Listener_ stack does the following: | ||
|
|
||
| - Creates an Amazon S3 bucket for AWS CloudTrail. | ||
| - Creates a trail to capture the creation of new log groups. | ||
| - Creates an event rule to pass those creation events to an Amazon EventBridge event bus. | ||
| - Sends an event via EventBridge to a Lambda function when a new log group is created. | ||
| - Creates a subscription filter for each new log group. | ||
|
|
||
| ## Removing subscription filters | ||
|
|
||
| If later on you want to remove subscription filters for one or more log groups: | ||
|
|
||
| - [Click this link](https://console.aws.amazon.com/cloudformation/home?#/stacks/new?stackName=axiom-cloudwatch-subscriber&templateURL=https://axiom-cloudformation.s3.amazonaws.com/stacks/axiom-cloudwatch-unsubscriber-v1.1.1-cloudformation-stack.yaml) to open the _Unsubscriber_ stack template. | ||
|
|
||
| The log group filtering works the same way as the _Subscriber_ stack. You can filter the log groups by a combination of names, prefix and regular expression. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| data "aws_cloudwatch_log_groups" "lambda_groups" { | ||
| log_group_name_prefix = "/aws/lambda/" | ||
| } | ||
|
|
||
| resource "axiom_dataset" "lambda_forwarder" { | ||
| name = "cloudwatch-lambda" | ||
| description = "[islam] test" | ||
| } | ||
|
|
||
| module "forwarder" { | ||
| source = "../../module" | ||
| axiom_dataset = axiom_dataset.lambda_forwarder.name | ||
| axiom_token = "xaat-46c6366c-47fe-487d-95fc-f35af3b55d20" | ||
| prefix = "axiom-cloudwatch-tf-test" | ||
| log_group_names = data.aws_cloudwatch_log_groups.lambda_groups.log_group_names | ||
| } | ||
|
|
||
|
|
||
| output "log_group_names" { | ||
| value = module.forwarder.log_group_names | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| terraform { | ||
| required_providers { | ||
| axiom = { | ||
| source = "axiomhq/axiom" | ||
| version = "1.1.2" | ||
| } | ||
| } | ||
| } | ||
|
|
||
| provider "axiom" { | ||
| api_token = "xaat-" | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| # terraform-aws-axiom | ||
|
|
||
| Setup required infrastructure and install Axiom Enterprise on AWS using Terraform. | ||
|
|
||
| ## AXiom CloudWatch Forwarder Terraform Module | ||
|
|
||
|
|
||
| ## Installation | ||
|
|
||
| Add the module as a required module in your terraform configuration file. | ||
|
|
||
| ```hcl | ||
| ``` | ||
|
|
||
|
|
||
| ```hcl | ||
| data "aws_cloudwatch_log_groups" "lambda_groups" { | ||
| # select the log group names to forward logs from | ||
| log_group_name_prefix = "/aws/lambda/" | ||
| } | ||
| resource "axiom_dataset" "lambda_forwarder" { | ||
| # create a dataset in Axiom | ||
| name = "cloudwatch-lambda" | ||
| description = "[islam] test" | ||
| } | ||
| module "cloudwatch_lambda_logs_forwarder" { | ||
| source = "https://github.com/axiomhq/axiom-cloudwatch-forwarder-tf.git" | ||
| axiom_dataset = axiom_dataset.lambda_forwarder.name | ||
| axiom_token = "xaat-*" | ||
| prefix = "axiom-cloudwatch" | ||
| log_group_names = data.aws_cloudwatch_log_groups.lambda_groups.log_group_names | ||
| } | ||
| output "log_group_names" { | ||
| value = module.forwarder.log_group_names | ||
| } | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| data "aws_region" "current" {} | ||
|
|
||
| data "aws_caller_identity" "current" {} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| data "archive_file" "forwarder" { | ||
| type = "zip" | ||
| source_file = "${path.module}/../src/forwarder.py" | ||
| output_path = "forwarder.zip" | ||
| } | ||
|
|
||
| resource "aws_lambda_function" "forwarder" { | ||
| filename = "forwarder.zip" | ||
| function_name = format("%s-forwarder", var.prefix) | ||
| logging_config { | ||
| log_format = "JSON" | ||
| log_group = aws_cloudwatch_log_group.forwarder.name | ||
| } | ||
| handler = "forwarder.handler" | ||
| runtime = "python3.9" | ||
| role = aws_iam_role.forwarder.arn | ||
|
|
||
| environment { | ||
| variables = { | ||
| AXIOM_TOKEN = var.axiom_token | ||
| AXIOM_DATASET = var.axiom_dataset | ||
| } | ||
| } | ||
| } | ||
|
|
||
| resource "aws_iam_role" "forwarder" { | ||
| name = format("%s-forwarder-role", var.prefix) | ||
| assume_role_policy = jsonencode({ | ||
| Version = "2012-10-17" | ||
| Statement = [ | ||
| { | ||
| Effect = "Allow" | ||
| Principal = { | ||
| Service = "lambda.amazonaws.com" | ||
| } | ||
| Action = "sts:AssumeRole" | ||
| }, | ||
| ] | ||
| }) | ||
| } | ||
|
|
||
| resource "aws_cloudwatch_log_group" "forwarder" { | ||
| name = format("/aws/axiom/%s-forwarder", var.prefix) | ||
| retention_in_days = 1 | ||
| } | ||
|
|
||
|
|
||
| resource "aws_lambda_permission" "allow_cloudwatch" { | ||
| statement_id = "AllowExecutionFromCloudWatch" | ||
| action = "lambda:InvokeFunction" | ||
| function_name = aws_lambda_function.forwarder.function_name | ||
| principal = "logs.amazonaws.com" | ||
| source_arn = format("arn:aws:logs:%s:%s:log-group:*:*", data.aws_region.current.name, data.aws_caller_identity.current.account_id) | ||
| } | ||
|
|
||
|
|
||
| resource "aws_cloudwatch_log_subscription_filter" "forwarder" { | ||
| for_each = { for index, name in var.log_group_names : index => name } | ||
| name = format("%s-forwarder-%s", var.prefix, element(split("/", each.value), length(split("/", each.value)) - 1)) | ||
| log_group_name = each.value | ||
| filter_pattern = "" | ||
| destination_arn = aws_lambda_function.forwarder.arn | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| output "forwarder_lambda" { | ||
| value = aws_lambda_function.forwarder.arn | ||
| } | ||
|
|
||
| output "log_group_names" { | ||
| value = var.log_group_names | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| terraform { | ||
| required_providers { | ||
| aws = { | ||
| source = "hashicorp/aws" | ||
| version = "~> 5.0" | ||
| } | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| variable "axiom_dataset" { | ||
| type = string | ||
| description = "Axiom dataset to forward logs to" | ||
| } | ||
|
|
||
| variable "axiom_token" { | ||
| type = string | ||
| description = "Axiom token for the dataset" | ||
| } | ||
|
|
||
| variable "prefix" { | ||
| type = string | ||
| default = "axiom-cloudwatch" | ||
| description = "prefix for resources, defaults to axiom-cloudwatch" | ||
| } | ||
|
|
||
|
|
||
| variable "log_group_names" { | ||
| type = list(string) | ||
| description = "list of log group names to forward to Axiom" | ||
| } |
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.