Experiment: Up and Running EKS Clusters serving Kafka Clusters with Strimzi

This is a experimental environment to learn, test and automate Strimzi Operator features.

Project Diagram

Deploy

terraform init
terraform apply --auto-approve

Cleanup

terraform destroy --auto-approve

Requirements

Name	Version
aws	>= 5.0
helm	~> 2.0
kubectl	~> 1.14
kubernetes	~> 2.0
tls	~> 3.1.0

Providers

Name	Version
aws	5.35.0
helm	2.12.1
kubectl	1.14.0
kubernetes	2.25.2
time	0.10.0
tls	3.1.0

Modules

No modules.

Resources

Name	Type
aws_eip.vpc_iep	resource
aws_eks_addon.cni	resource
aws_eks_addon.coredns	resource
aws_eks_addon.csi_driver	resource
aws_eks_addon.kubeproxy	resource
aws_eks_cluster.main	resource
aws_eks_node_group.general	resource
aws_eks_node_group.kafka	resource
aws_eks_node_group.observability	resource
aws_eks_node_group.zookeeper	resource
aws_iam_instance_profile.nodes	resource
aws_iam_openid_connect_provider.eks	resource
aws_iam_policy.aws_load_balancer_controller_policy	resource
aws_iam_policy.csi_driver	resource
aws_iam_policy_attachment.aws_load_balancer_controller_policy	resource
aws_iam_policy_attachment.csi_driver	resource
aws_iam_role.alb_controller	resource
aws_iam_role.eks_cluster_role	resource
aws_iam_role.eks_nodes_roles	resource
aws_iam_role_policy_attachment.cloudwatch	resource
aws_iam_role_policy_attachment.cni	resource
aws_iam_role_policy_attachment.ecr	resource
aws_iam_role_policy_attachment.eks-cluster-cluster	resource
aws_iam_role_policy_attachment.eks-cluster-service	resource
aws_iam_role_policy_attachment.node	resource
aws_iam_role_policy_attachment.ssm	resource
aws_internet_gateway.gw	resource
aws_kms_alias.eks	resource
aws_kms_key.eks	resource
aws_lb.kafka	resource
aws_lb.prometheus	resource
aws_lb_listener.grafana	resource
aws_lb_listener.plaintext	resource
aws_lb_target_group.grafana	resource
aws_lb_target_group.plaintext	resource
aws_nat_gateway.nat	resource
aws_route.nat_access	resource
aws_route.public_internet_access	resource
aws_route_table.igw_route_table	resource
aws_route_table.nat	resource
aws_route_table_association.private1a	resource
aws_route_table_association.private1b	resource
aws_route_table_association.private1c	resource
aws_route_table_association.public_1a	resource
aws_route_table_association.public_1b	resource
aws_route_table_association.public_1c	resource
aws_security_group.cluster_nodes_sg	resource
aws_security_group_rule.nodeport	resource
aws_security_group_rule.nodeports	resource
aws_subnet.private_subnet_1a	resource
aws_subnet.private_subnet_1b	resource
aws_subnet.private_subnet_1c	resource
aws_subnet.public_subnet_1a	resource
aws_subnet.public_subnet_1b	resource
aws_subnet.public_subnet_1c	resource
aws_vpc.cluster_vpc	resource
aws_vpc_ipv4_cidr_block_association.pods	resource
helm_release.alb_ingress_controller	resource
helm_release.prometheus	resource
helm_release.strimzi	resource
kubectl_manifest.configmap	resource
kubectl_manifest.grafana	resource
kubectl_manifest.kafka	resource
kubectl_manifest.kafka_plaintext	resource
kubectl_manifest.podmonitor	resource
kubernetes_config_map.aws-auth	resource
time_sleep.wait_operator	resource
aws_caller_identity.current	data source
aws_eks_cluster_auth.default	data source
aws_iam_policy_document.aws_load_balancer_controller_assume_role	data source
aws_iam_policy_document.aws_load_balancer_controller_policy	data source
aws_iam_policy_document.csi_driver	data source
aws_iam_policy_document.eks_cluster_role	data source
aws_iam_policy_document.eks_nodes_role	data source
aws_ssm_parameter.eks	data source
tls_certificate.eks	data source

Inputs

Name	Description	Type	Default	Required
addon_cni_version	Specifies the version of the AWS VPC CNI (Container Network Interface) plugin to use, which manages the network interfaces for pod networking.	string	"v1.14.1-eksbuild.1"	no
addon_coredns_version	Defines the version of CoreDNS to use, a DNS server/forwarder that is integral to internal Kubernetes DNS resolution.	string	"v1.11.1-eksbuild.4"	no
addon_csi_version	Indicates the version of the Container Storage Interface (CSI) driver to use for managing storage volumes in Kubernetes.	string	"v1.26.1-eksbuild.1"	no
addon_kubeproxy_version	Sets the version of Kubeproxy to be used, which handles Kubernetes network services like forwarding the requests to correct containers.	string	"v1.29.0-eksbuild.1"	no
aws_region	AWS region where the EKS cluster will be deployed. This should be set to the region where you want your Kubernetes resources to reside.	string	"us-east-1"	no
cluster_name	The name of the Amazon EKS cluster. This is a unique identifier for your EKS cluster within the AWS region.	string	"eks-kafka"	no
cluster_private_zone	The private DNS zone name for the EKS cluster in AWS Route53. This zone is used for internal DNS resolution within the cluster.	string	"k8s.cluster"	no
default_tags	A map of default tags to apply to all resources. These tags can help with identifying and organizing resources within the AWS environment.	map(string)	{ "Environment": "prod", "Foo": "Bar", "Ping": "Pong" }	no
general_instances_sizes	A list of EC2 instance types to use for the EKS worker nodes. These instance types should balance between cost, performance, and resource requirements for your workload.	list	[ "t3.large" ]	no
general_scale_options	Configuration for the EKS cluster auto-scaling. It includes the minimum (min), maximum (max), and desired (desired) number of worker nodes.	map	{ "desired": 2, "max": 2, "min": 2 }	no
k8s_version	The version of Kubernetes to use for the EKS cluster. This version should be compatible with the AWS EKS service and other infrastructure components.	string	"1.29"	no
kafka_desired_replicas	n/a	number	4	no
kafka_enable_cross_zone_load_balancing	Controls whether cross-zone load balancing is enabled for the Network Load Balancer (NLB) associated with the Kafka brokers, allowing even traffic distribution across all availability zones.	bool	false	no
kafka_instances_sizes	A list of EC2 instance types to use for the EKS worker nodes. These instance types should balance between cost, performance, and resource requirements for your workload.	list	[ "c6a.2xlarge" ]	no
kafka_limit_cpu	n/a	string	"2"	no
kafka_limit_memory	n/a	string	"16Gi"	no
kafka_min_insync_replicas	n/a	number	2	no
kafka_nlb_ingress_enable_termination_protection	Determines if termination protection is enabled for the Network Load Balancer (NLB) associated with the Kafka brokers, preventing accidental deletion.	bool	false	no
kafka_nlb_ingress_internal	Indicates whether the Network Load Balancer (NLB) for the Kafka brokers should be internal ('true') or external ('false'), controlling the scope of access to within the AWS network or from the internet, respectively.	string	"true"	no
kafka_nlb_ingress_type	Specifies the type of ingress to be used for Kafka, such as 'network', determining how the NLB (Network Load Balancer) handles incoming traffic to the Kafka brokers.	string	"network"	no
kafka_replication_factor	n/a	number	3	no
kafka_request_cpu	n/a	string	"2"	no
kafka_request_max_bytes	n/a	number	2147483647	no
kafka_request_memory	n/a	string	"8Gi"	no
kafka_scale_options	Configuration for the EKS cluster auto-scaling. It includes the minimum (min), maximum (max), and desired (desired) number of worker nodes.	map	{ "desired": 4, "max": 4, "min": 4 }	no
kafka_storage_class	n/a	string	"gp2"	no
kafka_storage_size	n/a	string	"20Gi"	no
kafka_xms	n/a	string	"4g"	no
kafka_xmx	n/a	string	"6g"	no
observability_instances_sizes	A list of EC2 instance types to use for the EKS worker nodes. These instance types should balance between cost, performance, and resource requirements for your workload.	list	[ "t3.large" ]	no
observability_scale_options	Configuration for the EKS cluster auto-scaling. It includes the minimum (min), maximum (max), and desired (desired) number of worker nodes.	map	{ "desired": 2, "max": 2, "min": 2 }	no
prometheus_enable_cross_zone_load_balancing	Controls whether cross-zone load balancing is enabled for the Network Load Balancer (NLB) associated with the Prometheus server, allowing even traffic distribution across all availability zones.	bool	false	no
prometheus_nlb_ingress_enable_termination_protection	Determines if termination protection is enabled for the Network Load Balancer (NLB) associated with the Prometheus server, preventing accidental deletion.	bool	false	no
prometheus_nlb_ingress_internal	Indicates whether the Network Load Balancer (NLB) for the Prometheus server should be internal ('true') or external ('false'), controlling the scope of access to within the AWS network or from the internet, respectively.	string	"false"	no
prometheus_nlb_ingress_type	Defines the type of ingress to be used for Prometheus, such as 'network', determining how the NLB (Network Load Balancer) handles incoming traffic to the Prometheus server.	string	"network"	no
zookeeper_desired_replicas	Specifies the desired number of Zookeeper replicas for handling distributed coordination tasks efficiently and reliably.	number	3	no
zookeeper_instances_sizes	Specifies the sizes of EC2 instances for Zookeeper nodes within the EKS cluster. Optimal instance types are chosen based on the balance between cost, performance, and resource requirements for Zookeeper.	list	[ "c5.large" ]	no
zookeeper_limit_cpu	Specifies the maximum CPU limit for each Zookeeper pod, preventing the pod from consuming more CPU resources than this limit.	string	"2"	no
zookeeper_limit_memory	Sets the maximum memory limit for each Zookeeper pod, ensuring that the pod does not exceed this amount to maintain cluster stability.	string	"16Gi"	no
zookeeper_request_cpu	Determines the CPU request for each Zookeeper pod, reserving the specified amount of CPU resources for optimal performance.	string	"2"	no
zookeeper_request_memory	Defines the memory request for each Zookeeper pod, which guarantees the specified amount of memory for proper operation.	string	"8Gi"	no
zookeeper_scale_options	Defines the scaling options for the Zookeeper nodes in the EKS cluster, including the minimum, maximum, and desired number of nodes to ensure consistent performance and fault tolerance.	map	{ "desired": 3, "max": 3, "min": 3 }	no
zookeeper_storage_class	Defines the storage class to be used for Zookeeper persistent storage, impacting the performance and availability of the storage.	string	"gp2"	no
zookeeper_storage_size	Allocates the size of the persistent storage for each Zookeeper node, ensuring sufficient space for data storage and log retention.	string	"20Gi"	no

Outputs

Name	Description
bootstrap_servers_plaintext	n/a
grafana_default_pass	n/a
grafana_default_user	n/a
grafana_url	n/a

Field Notes

List Kafka Nodes

kubectl get nodes -l NodeGroupType=kafka -o wide

List Observability Nodes

kubectl get nodes -l NodeGroupType=observability -o wide

List General Nodes

kubectl get nodes -l NodeGroupType=general -o wide

List Zookeeper Nodes

kubectl get nodes -l NodeGroupType=zookeeper -o wide

List Strimzipodsets

kubectl get strimzipodsets -n strimzi

List Kafka Configs

kubectl get kafka -n strimzi

Kafka Stress - Produce Messages

/usr/local/bin/kafka-stress --bootstrap-servers eks-kafka-kafka-1dfc54296e6bfac8.elb.us-east-1.amazonaws.com:9092 --events 300000 --topic kafka-stress

Kafka Stress - Consume Messages

/usr/local/bin/kafka-stress --bootstrap-servers eks-kafka-kafka-1dfc54296e6bfac8.elb.us-east-1.amazonaws.com:9092 --test-mode consumer --topic kafka-stress --consumer-group teste

Deploy "kafka-cli" to manage resources

kubectl -n strimzi run --restart=Never --image=quay.io/strimzi/kafka:0.38.0-kafka-3.6.0 kafka-cli -- /bin/sh -c "exec tail -f /dev/null"

kubectl -n strimzi exec -it kafka-cli -- bin/kafka-topics.sh \
  --describe \
  --topic kafka-stress \
  --bootstrap-server cluster-kafka-plain-bootstrap:9092

References

• Github - Strimzi Kafka Operator
• Deploying and scaling Apache Kafka on Amazon EKS
• Using Strimzi with Amazon Network load balancers
• Apache Kafka on Kubernetes with Strimzi