From d62a233caebe04e2caf0d9bcabc0926668bbaa34 Mon Sep 17 00:00:00 2001 From: Ra'Jiska Date: Sat, 11 Nov 2023 20:35:09 +0800 Subject: [PATCH 1/5] Cloudwatch Monitoring --- docs/configuration.md | 16 +++++++++++ docs/features.md | 61 ++++++++++++++++++++++++++++++++++-------- mkdocs.yml | 1 + packer/fck-nat.pkr.hcl | 3 ++- service/fck-nat.sh | 6 +++++ 5 files changed, 75 insertions(+), 12 deletions(-) create mode 100644 docs/configuration.md diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 0000000..c84eb4e --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,16 @@ +# fck-nat Configuration + +## Configuration file + +Upon starting, fck-nat evaluates a configuration file describing how the instance should behave as well as what features +shall be enabled. To configure fck-nat, ensure a file `/etc/fck-nat.conf` exists with your configuration. fck-nat +requires the service to be restarted by running `service fck-nat-resart`. In most implementations this configuration is +passed only once via EC2's user data. + +The following describes available options: +| name | description +---------------------- +| `eni_id` | The ID of the Elastic Network Interface to attach to the instance and use as a consistent endpoint to send traffic to fck nat. This is required when using high-availability mode. | +| `eip_id` | The ID of an Elastic IP to be attached to the public network interface. This ensures the NAT gateway public traffic is always routed through the same public IP address. | +| `cwagent_enabled` | If set, enables Cloudwatch agent and forward instance metrics to Cloudwatch. Requires `cwagent_cfg_param_arn` to be set. | +| `cwagent_cfg_param_arn` | The ARN of the SSM Parameter holding the Cloudwatch agent configuration and which the agent shall pull from. Requires `cwagent_enabled` to be set. | diff --git a/docs/features.md b/docs/features.md index 8038a21..92e28fd 100644 --- a/docs/features.md +++ b/docs/features.md @@ -7,19 +7,58 @@ autoscaling group, fck-nat can be configured to always attach a specific ENI at a consistent internal-facing IP address. Additionally, it is also possible to configure an already allocated EIP address that would be carried through instance refreshs. -To enable these features, you'll need to create a config file at `/etc/fck-nat.conf` like this: +Those features are controlled by `eni_id` and `eip_id` directive in the configuration file. -``` -eni_id= -eip_id= -``` +## Metrics -Once the fck-nat configuration is created, be sure to restart the service by running `service fck-nat restart`. +One of the objectives of fck-nat is to offer as close as possible metric parity with Managed NAT Gateway. While the +project supports various metrics similar to the managed NAT Gateway via Cloudwatch agent, each provider is responsible +for passing their configuration to the agent via fck-nat's `cwagent_enabled`, and `cwagent_cfg_param_arn` directives +within its configuration file. -In the official fck-nat CDK construct, we configure this via UserData on the autoscaling group. +As an example, you might use the following configuration file which have Cloudwatch agent report most of metrics +provided in the managed NAT Gateway: -## Metrics +``` json +{ + "agent": { + "metrics_collection_interval": 60, + "run_as_user": "root", + "usage_data": false + }, + "metrics": { + "namespace": "fck-nat", + "metrics_collected": { + "net": { + "resources": ["eth0", "eth1"], + "measurement": [ + { "name": "bytes_recv", "rename": "BytesIn", "unit": "Bytes" }, + { "name": "bytes_sent", "rename": "BytesOut", "unit": "Bytes" }, + { "name": "packets_sent", "rename": "PacketsOutCount", "unit": "Count" }, + { "name": "packets_recv", "rename": "PacketsInCount", "unit": "Count" }, + { "name": "drop_in", "rename": "PacketsDropInCount", "unit": "Count" }, + { "name": "drop_out", "rename": "PacketsDropOutCount", "unit": "Count" } + ] + }, + "netstat": { + "measurement": [ + { "name": "tcp_syn_sent", "rename": "ConnectionAttemptOutCount", "unit": "Count" }, + { "name": "tcp_syn_recv", "rename": "ConnectionAttemptInCount", "unit": "Count" }, + { "name": "tcp_established", "rename": "ConnectionEstablishedCount", "unit": "Count" } + ] + }, + "mem": { + "measurement": [ + { "name": "used_percent", "rename": "MemoryUsed", "unit": "Percent" } + ] + } + }, + "append_dimensions": { + "InstanceId": "$${aws:InstanceId}" + } + } +} +``` -One of the objectives of fck-nat is to offer as close as possible metric parity with Managed NAT Gateway. If this -feature is important to you, help us prioritize it by +1'ing the following issue: [Report additional metrics from -fck-nat](https://github.com/AndrewGuenther/fck-nat/issues/16) +If this feature is important to you, help us prioritize it by +1'ing the following issue: [Report additional metrics +from fck-nat](https://github.com/AndrewGuenther/fck-nat/issues/16) diff --git a/mkdocs.yml b/mkdocs.yml index e42d608..e8e30e3 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -18,6 +18,7 @@ theme: nav: - Introduction: index.md - Deploying: deploying.md + - Configuration: configuration.md - Features: features.md repo_url: https://github.com/AndrewGuenther/fck-nat diff --git a/packer/fck-nat.pkr.hcl b/packer/fck-nat.pkr.hcl index bae55ab..8fbff3b 100644 --- a/packer/fck-nat.pkr.hcl +++ b/packer/fck-nat.pkr.hcl @@ -96,7 +96,8 @@ build { provisioner "shell" { inline = [ - "sudo yum --nogpgcheck -y localinstall /tmp/fck-nat-${var.version}-any.rpm" + "sudo yum --nogpgcheck -y localinstall /tmp/fck-nat-${var.version}-any.rpm", + "sudo yum install amazon-cloudwatch-agent -y" ] } } diff --git a/service/fck-nat.sh b/service/fck-nat.sh index 930d0b7..ce15d4f 100755 --- a/service/fck-nat.sh +++ b/service/fck-nat.sh @@ -70,4 +70,10 @@ iptables -t nat -F echo "Adding NAT rule..." iptables -t nat -A POSTROUTING -o "$nat_interface" -j MASQUERADE -m comment --comment "NAT routing rule installed by fck-nat" +if test -n "$cwagent_enabled" && test -n "$cwagent_cfg_param_name"; then + echo "Found cwagent_enabled and cwagent_cfg_param_arn configuration, starting CloudWatch agent..." + systemctl enable amazon-cloudwatch-agent + /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c "ssm:$cwagent_cfg_param_name" +fi + echo "Done!" From 58eaa8695222e6abd128ed35397181d495b48355 Mon Sep 17 00:00:00 2001 From: Ra'Jiska Date: Sat, 11 Nov 2023 20:45:24 +0800 Subject: [PATCH 2/5] Fix Configuration.md Format --- docs/configuration.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index c84eb4e..4423232 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -8,9 +8,9 @@ requires the service to be restarted by running `service fck-nat-resart`. In mos passed only once via EC2's user data. The following describes available options: -| name | description ----------------------- -| `eni_id` | The ID of the Elastic Network Interface to attach to the instance and use as a consistent endpoint to send traffic to fck nat. This is required when using high-availability mode. | -| `eip_id` | The ID of an Elastic IP to be attached to the public network interface. This ensures the NAT gateway public traffic is always routed through the same public IP address. | -| `cwagent_enabled` | If set, enables Cloudwatch agent and forward instance metrics to Cloudwatch. Requires `cwagent_cfg_param_arn` to be set. | +| name | description | +| ----------------------- | ----------- | +| `eni_id` | The ID of the Elastic Network Interface to attach to the instance and use as a consistent endpoint to send traffic to fck nat. This is required when using high-availability mode. | +| `eip_id` | The ID of an Elastic IP to be attached to the public network interface. This ensures the NAT gateway public traffic is always routed through the same public IP address. | +| `cwagent_enabled` | If set, enables Cloudwatch agent and forward instance metrics to Cloudwatch. Requires `cwagent_cfg_param_arn` to be set. | | `cwagent_cfg_param_arn` | The ARN of the SSM Parameter holding the Cloudwatch agent configuration and which the agent shall pull from. Requires `cwagent_enabled` to be set. | From 3433913b6050c031ac7d4e0d46aa8866e7d1e0fd Mon Sep 17 00:00:00 2001 From: Ra'Jiska Date: Tue, 14 Nov 2023 14:07:07 +0800 Subject: [PATCH 3/5] Add ethtool metrics + required IAM documentation --- docs/configuration.md | 2 +- docs/features.md | 20 +++++++++++++++++--- 2 files changed, 18 insertions(+), 4 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index 4423232..4353a0b 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -13,4 +13,4 @@ The following describes available options: | `eni_id` | The ID of the Elastic Network Interface to attach to the instance and use as a consistent endpoint to send traffic to fck nat. This is required when using high-availability mode. | | `eip_id` | The ID of an Elastic IP to be attached to the public network interface. This ensures the NAT gateway public traffic is always routed through the same public IP address. | | `cwagent_enabled` | If set, enables Cloudwatch agent and forward instance metrics to Cloudwatch. Requires `cwagent_cfg_param_arn` to be set. | -| `cwagent_cfg_param_arn` | The ARN of the SSM Parameter holding the Cloudwatch agent configuration and which the agent shall pull from. Requires `cwagent_enabled` to be set. | +| `cwagent_cfg_param_name` | The name of the SSM Parameter holding the Cloudwatch agent configuration and which the agent shall pull from. Requires `cwagent_enabled` to be set. | diff --git a/docs/features.md b/docs/features.md index 92e28fd..f68e0f6 100644 --- a/docs/features.md +++ b/docs/features.md @@ -7,7 +7,10 @@ autoscaling group, fck-nat can be configured to always attach a specific ENI at a consistent internal-facing IP address. Additionally, it is also possible to configure an already allocated EIP address that would be carried through instance refreshs. -Those features are controlled by `eni_id` and `eip_id` directive in the configuration file. +Those features are controlled by `eni_id` and `eip_id` directive in the configuration file. + +**IAM requirements**: `ec2:AttachNetworkInterface`, `ec2:ModifyNetworkInterfaceAttribute` on `*` for ha-mode, plus +`ec2:AssociateAddress`, `ec2:DisassociateAddress` on `*` when using a static EIP. ## Metrics @@ -47,6 +50,15 @@ provided in the managed NAT Gateway: { "name": "tcp_established", "rename": "ConnectionEstablishedCount", "unit": "Count" } ] }, + "ethtool": { + "interface_include": ["eth0", "eth1"], + "metrics_include": [ + "bw_in_allowance_exceeded", + "bw_out_allowance_exceeded", + "conntrack_allowance_exceeded", + "pps_allowance_exceeded" + ] + }, "mem": { "measurement": [ { "name": "used_percent", "rename": "MemoryUsed", "unit": "Percent" } @@ -60,5 +72,7 @@ provided in the managed NAT Gateway: } ``` -If this feature is important to you, help us prioritize it by +1'ing the following issue: [Report additional metrics -from fck-nat](https://github.com/AndrewGuenther/fck-nat/issues/16) +Ensure you are aware of Cloudwatch metrics costs before enabling the Cloudwatch agent. The above configuration would +cost you about $17/monthly, excluding free tier. + +**IAM requirements**: `ssm:GetParameter` on the SSM Parameter ARN, and `cloudwatch:PutMetricData` on `*`. \ No newline at end of file From e4a95ff94d496a5f8989efc5dca6e8c40b511626 Mon Sep 17 00:00:00 2001 From: Ra'Jiska Date: Tue, 14 Nov 2023 14:13:45 +0800 Subject: [PATCH 4/5] Fix cwagent_cfg_param_arn to cwagent_cfg_param_name --- docs/configuration.md | 2 +- docs/features.md | 2 +- service/fck-nat.sh | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index 4353a0b..cf188db 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -12,5 +12,5 @@ The following describes available options: | ----------------------- | ----------- | | `eni_id` | The ID of the Elastic Network Interface to attach to the instance and use as a consistent endpoint to send traffic to fck nat. This is required when using high-availability mode. | | `eip_id` | The ID of an Elastic IP to be attached to the public network interface. This ensures the NAT gateway public traffic is always routed through the same public IP address. | -| `cwagent_enabled` | If set, enables Cloudwatch agent and forward instance metrics to Cloudwatch. Requires `cwagent_cfg_param_arn` to be set. | +| `cwagent_enabled` | If set, enables Cloudwatch agent and forward instance metrics to Cloudwatch. Requires `cwagent_cfg_param_name` to be set. | | `cwagent_cfg_param_name` | The name of the SSM Parameter holding the Cloudwatch agent configuration and which the agent shall pull from. Requires `cwagent_enabled` to be set. | diff --git a/docs/features.md b/docs/features.md index f68e0f6..cb473d4 100644 --- a/docs/features.md +++ b/docs/features.md @@ -16,7 +16,7 @@ Those features are controlled by `eni_id` and `eip_id` directive in the configur One of the objectives of fck-nat is to offer as close as possible metric parity with Managed NAT Gateway. While the project supports various metrics similar to the managed NAT Gateway via Cloudwatch agent, each provider is responsible -for passing their configuration to the agent via fck-nat's `cwagent_enabled`, and `cwagent_cfg_param_arn` directives +for passing their configuration to the agent via fck-nat's `cwagent_enabled`, and `cwagent_cfg_param_name` directives within its configuration file. As an example, you might use the following configuration file which have Cloudwatch agent report most of metrics diff --git a/service/fck-nat.sh b/service/fck-nat.sh index ce15d4f..eed440b 100755 --- a/service/fck-nat.sh +++ b/service/fck-nat.sh @@ -71,7 +71,7 @@ echo "Adding NAT rule..." iptables -t nat -A POSTROUTING -o "$nat_interface" -j MASQUERADE -m comment --comment "NAT routing rule installed by fck-nat" if test -n "$cwagent_enabled" && test -n "$cwagent_cfg_param_name"; then - echo "Found cwagent_enabled and cwagent_cfg_param_arn configuration, starting CloudWatch agent..." + echo "Found cwagent_enabled and cwagent_cfg_param_name configuration, starting CloudWatch agent..." systemctl enable amazon-cloudwatch-agent /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c "ssm:$cwagent_cfg_param_name" fi From 0fff5e15c789f2d8831a0aa8daebe57e57ade734 Mon Sep 17 00:00:00 2001 From: Ra'Jiska Date: Tue, 14 Nov 2023 14:14:52 +0800 Subject: [PATCH 5/5] English is hard --- docs/features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/features.md b/docs/features.md index cb473d4..05fae1d 100644 --- a/docs/features.md +++ b/docs/features.md @@ -72,7 +72,7 @@ provided in the managed NAT Gateway: } ``` -Ensure you are aware of Cloudwatch metrics costs before enabling the Cloudwatch agent. The above configuration would +Ensure you are aware of Cloudwatch metrics costs before enabling Cloudwatch agent. The above configuration would cost you about $17/monthly, excluding free tier. **IAM requirements**: `ssm:GetParameter` on the SSM Parameter ARN, and `cloudwatch:PutMetricData` on `*`. \ No newline at end of file