AUTOBOT is an automated deployment solution for Elasticsearch clusters combining Anisible for Elastic installation & configuration, and CloudFormation for AWS infrastructure provisioning.
The Ansible playbook is designed to be used independently from CloudFormation: you are welcome to ignore the CloudFormation aspects of this project without any consequence. Additionally, the Ansible playbook is written for Debian/Ubuntu linux, but the "Configuration: Ansible" section of the readme below explains what to modify to support other linux flavors, as well as environments with restricted network access (local package installs).
AUTOBOT automates the deployment of a complete Elasticsearch cluster with:
- dedicated master nodes
- hot nodes for active data
- frozen nodes for historical data
- Helper node (also Ansible controller) running:
- Kibana for visualization
- (TO DO) MinIO for S3-compatible storage
- Load balancer for cluster access (CloudFormation only)
- Private DNS zone for service discovery (CloudFormation only)
Note: The elastic user password is dropped into ~/.elasticsearch/elastic_password.txt on the Ansible controller
- Automated Elasticsearch installation and configuration
- The playbook is designed for Debian/Ubuntu with internet connectivity to enable adding the Elastic repo & 'apt install'
- The 8.x version of Elastic/Kibana that is installed can be modified in the elasticsearch & kibana task main.ymls
- In order to install a non-8.x version, you'll need to modify the common task's main.yml (search for "8.x")
- The common/elasticsearch/kibana tasks have commented-out sections for local package installation in restricted environments
- If you are using a non-Debian/Ubuntu flavor, search for "apt" and "systemd" and update those functions are per your non-debian OS
- roles/common/tasks/main.yml
- roles/elasticsearch/tasks/main.yml
- roles/kibana/tasks/main.yml
- The playbook is designed for Debian/Ubuntu with internet connectivity to enable adding the Elastic repo & 'apt install'
- HTTP & Transport certificate management (Self-Signed by the initial master node)
- Cluster bootstrap and node enrollment
- Kibana setup and integration
- MinIO deployment and configuration (TO DO)
- VPC with public subnets
- EC2 instances:
- Master/Data nodes: m6i.16xlarge
- Helper node: t3.xlarge (runs Ansible, Kibana, MinIO)
- Application Load Balancer
- Route 53 private hosted zone
- S3 bucket for frozen indices
- Security groups and IAM roles
- playbook/utils/remove-es.yml can be used to remove all elastic components from the cluster. It does not undo OS changes made by tuning.yml
- remove-es.yml assumes debian/ubuntu and installation/removal via apt, as well as systemd. If your environment doesn't match this, you'll need to update this file
- playbook/utils/tuning.yml is not executed as part of the default playbook.yml - it should be run AFTER playbook.xml is complted
- tuning.yml should also be updated as per your specific environment - it has guidance in the comments at the top of the file, like:
- Make sure to set your JVM heap size appropriately based on Elastic's official guidance: https://www.elastic.co/guide/en/elasticsearch/reference/current/advanced-configuration.html#set-jvm-heap-size
- Search for "-Xms128g" and "-Xmx128g" in this file and replace with values appropriate for your environment.
- Additionally, this assumes systemd is being used. If not, search for "systemd" here and update as per your environment.
- tuning.yml should also be updated as per your specific environment - it has guidance in the comments at the top of the file, like:
- Python 3.x
- Ansible 2.9+
- SSH Keypair authentication support (See Step 6 in Quick Start below)
- For CloudFormation automation, you'll also need:
- A valid EC2 key pair
- AWS CLI configured with appropriate credentials
- Your IP address for security group configuration
- Clone the repository:
git clone https://github.com/yourusername/autobot.git
cd autobot- (Optional - if using CloudFormation) Customize the CloudFormation template:
- Find/Replace all
$username$ $project$ values as per your environment. - Update/Remove all Tag Keys as per your environment
UserData sections of EC2 Instance Configurations:
--filters "Name=tag:Name,Values=$username$-$project$-kibana-node" <---- UPDATE THIS
Tags:
- Key: Name
Value: $username$-$project$-vpc <---- UPDATE THIS
- Key: division
Value: field
- Key: org
Value: sa
- Key: team
Value: amer-strat
- Key: project
Value: $username$-$project$ <---- UPDATE THIS- (Optional - if using CloudFormation) Deploy the CloudFormation stack:
aws cloudformation create-stack \
--stack-name autobot-elastic \
--template-body file://cloudformation.yaml \
--parameters ParameterKey=KeyName,ParameterValue=YOUR_KEY_NAME \
ParameterKey=AdminIP,ParameterValue=YOUR_IP_CIDR \
--capabilities CAPABILITY_IAM- (Optional) Update playbook/roles/elasticsearch/tasks/main.yml with custom values as needed
- Find the stanza "Set certificate password fact" and change es_cert_pass if you want a custom value
- Note if you change es_cert_pass here, you also must change it in playbook/roles/kibana/tasks/main.yml with the same value
- Find the stanza "Set s3 secret key fact" and change s3_client_secret_key to match your s3 provider (minIO, others) spec
- Find the stanza "Set s3 access key fact" and change s3_client_access_key to match your s3 provider (minIO, others) spec
- name: Set certificate password fact
set_fact:
es_cert_pass: "elastic2024"
- name: Set s3 secret key fact
set_fact:
s3_client_secret_key: "sample_secret_key" <---- UPDATE THIS
- name: Set s3 access key fact
set_fact:
s3_client_access_key: "sample_access_key" <---- UPDATE THIS- Generate or modify inventory.ini:
- If you created your cluster manually, you should customize the sample playbook/inventory.ini with your cluster hostnames & private key file at the bottom
- You must include the "ansible_host=" prefix as specified in the sample playbook/inventory.ini
- If your cluster was built with Cloudformation in steps 2&3 above, run the provided script utils/gen-inventory.sh to generate inventory.ini based on EC2 tags
- You'll need to replace "your-project" in gen-inventory.sh with the project name you used in the CloudFormation template
- You may need to chmod it to be executable
Modify this line in inventory.ini with the full path to your private key file:
ansible_ssh_private_key_file=/home/ubuntu/.ssh/your-key.pem <---- UPDATE THIS- Set up SSH access on helper node:
- Update the utils/config with your private key file name and inventory hostnames/IP-ranges
- Copy the config file and priate key to the Ansible (helper) node, and set permissions
# ~/.ssh/config
# This is a ssh config file that's used to allow Ansible to perform automation tasks to the cluster
Host *.elastic.internal
IdentityFile ~/.ssh/your-key.pem <---- UPDATE THIS
User ubuntu
StrictHostKeyChecking no
Host *.amazonaws.com *.compute.internal ec2-* 10.* 3.* 18.* <---- UPDATE THIS
IdentityFile ~/.ssh/your-key.pem <---- UPDATE THIS
User ubuntu
# Copy SSH config and key to helper node
# From the "utils" subdirectory of the playbook:
scp -rp config your-key.pem ubuntu@<HELPER_NODE_IP>:/home/ubuntu/.ssh/ <---- replace "your-key" and <HELPER_NODE_IP>
chmod 600 /home/ubuntu/.ssh/config /home/ubuntu/.ssh/your-key.pem <---- replace "your-key"
# Test SSH Keypair authentication from your client machine to the Anisble helper node:
ssh -i your-key.pem ubuntu@<HELPER_NODE_IP> <---- replace "your-key" and <HELPER_NODE_IP>
ssh ubuntu@<MASTER_NODE_INTERNAL_IP> <---- replace <MASTER_NODE__IP>- (Optional) Customize the location where Elastic will store its data
- By default, and as specified in /etc/elasticsearch/elasticsearch.yml, Elastic will store data in: /var/lib/elasticsearch/
- If you have a different location in mind (like a dedicated data array), you should modify playbook/roles/elasticsearch/tasks/main.yml:
# playbook/roles/elasticsearch/tasks/main.yml:
- name: Create initial elasticsearch settings
copy:
dest: /etc/elasticsearch/elasticsearch.yml
content: |
# Elasticsearch configuration
path.data: /var/lib/elasticsearch <------ Change this value to the folder you want Elastic to store its data in
path.logs: /var/log/elasticsearch- Deploy with Ansible:
cd playbook
ansible-playbook -i inventory.ini playbook.yml- (Recommended) Environment Tuning:
- utils/tuning.yml was written for Linux OS with systemd usage, and defines very large (128GB) Heap Size values
- You should verify and modify references to "heap", "apt", and "systemd" in this file, as per your environment
- Heap size guidance can be found here: https://www.elastic.co/guide/en/elasticsearch/reference/current/advanced-configuration.html#set-jvm-heap-size
cd playbook
ansible-playbook -i inventory.ini utils/tuning.yml- (Optional) Remove Elastic Components:
- utils/remove-es.yml was written for Debian/Ubuntu OS with apt & systemd usage
- You may need to modify as per your environment
cd playbook
ansible-playbook -i inventory.ini utils/remove-es.yml- Acts as Ansible control node
- Runs MinIO server for S3-compatible storage
- Runs Kibana for visualization
- Handles certificate management
- Dedicated cluster coordination
- Certificate authority
- Cluster state management
- Active data storage
- Search and indexing
- Historical data storage
- Integrated with MinIO for cost-effective storage
- Private VPC network
- Security groups limiting access to specified IP
- Internal TLS/SSL encryption
- Authentication enabled by default
- Private DNS for internal service discovery
After deployment, you can access:
- Elasticsearch:
https://<load-balancer-dns>:9200 - Kibana:
https://<helper-node-dns>:5601 - MinIO Server:
http://<helper-node-dns>:9000 - MinIO Console:
http://<helper-node-dns>:9001
View stack status:
aws cloudformation describe-stacks --stack-name autobot-elasticView stack events (troubleshooting):
aws cloudformation describe-stack-events --stack-name autobot-elasticClean up resources:
# Remove the CloudFormation stack
aws cloudformation delete-stack --stack-name autobot-elastic
# Wait for stack deletion to complete
aws cloudformation wait stack-delete-complete --stack-name autobot-elasticautobot/
├── cloudformation.yaml # AWS infrastructure template
├── generate-inventory.sh # Script to generate Ansible inventory
├── playbook/
│ ├── inventory.ini # Ansible inventory
│ ├── playbook.yml # Main playbook
│ └── roles/
│ ├── common/ # Common configurations
│ ├── elasticsearch/ # Elasticsearch setup
│ └── kibana/ # Kibana setup
├── utils/
│ ├── config # .ssh/config example
│ ├── remove-es.yml # Cleanup playbook
│ ├── tuning.yml # OS Tuning playbook
Feel free to submit issues, fork the repository, and create pull requests for any improvements.