Cloud security is the top priority at Amazon Web Services (AWS) and the security partner ecosystem plays a critical role in building and executing security capabilities. Infopercept Consulting is an AWS Partner which leverages open-source and building security solutions for customers.
In this post, we will share how Infopercept is leveraging Shuffle, an open-source general purpose security automation platform that can be used for building security playbooks.
The key elements of Shuffle are ease of integration with AWS services, as well as open source-like integration with Yara malware analysis. We’ll build a response playbook for malware detection and isolation.
Analyzing Objects in Amazon S3 Amazon Simple Storage Service (Amazon S3) allows you to store vast amounts of data, and objects uploaded to S3 need to be monitored for any malicious objects that can end up infecting your critical systems.
Shuffle is an open-source security orchestration, automation, and response (SOAR) implementation that makes automation accessible to anyone. A workflow will be created when a file gets placed in S3, and it will be sent to Shuffle via a webhook where it will be further analyzed with the help of automation.
Solution Overview Shuffle fetches objects from Amazon S3 that will be scanned by the YARA rules, which are used to classify and identify malware samples by creating descriptions of malware families based on textual or binary patterns. Once any malicious indicator is found, an action will be defined to either delete or quarantine it for further analysis.
The IP address of the uploader can be blocked with an additional Shuffle workflow. The solution detailed in this post leverages serverless components for fast and cost-effective execution.
Figure 1 – Malware detection on Amazon S3 workflow.
Prerequisites AWS account Amazon S3 bucket Amazon EC2 instance with 8 core, CPU 32 GB, and RAM 500 GB AWS Lambda function AWS Lambda Configuration The AWS Lambda configuration needs to be set so that whenever any object is uploaded to S3, the Lambda function will get triggered by sending the data to the Shuffle’s webhook. The Webhook will then receive the object data and further analysis will take place in Shuffle.
User flow: User > upload object to S3 > S3 200 OK triggers AWS Lambda function > triggers webhook > file is downloaded from within Shuffle.
Figure 2 – AWS Lambda function for forwarding the object data to Shuffle for analysis.
After logging into your AWS console, go to Lambda functions and click Create function in the top right corner. Use Author from scratch and type in a name like “Shuffle-forwarding.” Make sure to choose Runtime as Python 3.8. Click Create function in the bottom right corner.
Figure 3 – Creating the AWS Lambda function.
Click Add trigger in the window left of the function. In the next menu, find Amazon S3 and before choosing the bucket you want and the Event type, click Add trigger. Note that the bucket and Lambda function have to be in the same AWS region.
Figure 4 – Configuration for the Lambda function trigger.
Under Configuration > Environment variables, click Edit. Add variable with key “SHUFFLE_WEBHOOK” and the value from earlier, and then click Save.
Figure 5 – Setting up the environment variables for the Webhook.
Next, it’s time to add some code. Go to the Code tab and paste in the code below:
import urllib3 import json import urllib.parse import urllib3
import os print('Loading function') def lambda_handler(event, context): # Get the object from the event and show its content type bucket = event['Records'][0]['s3']['bucket']['name'] webhook = os.environ.get("SHUFFLE_WEBHOOK") if not webhook: return "No webhook environment defined: SHUFFLE_WEBHOOK" http = urllib3.PoolManager() ret = http.request('POST', webhook, body=json.dumps(event["Records"][0]).encode("utf-8")) if ret.status != 200: return "Bad status code for webhook: %d" % ret.status_code print("Status code: %d\nData: %s" % (ret.status, ret.data)) Click Deploy and this should now forward the request to Shuffle.
Amazon S3 Bucket Configuration for Scanning Create an Amazon S3 bucket where objects will be monitored with YARA rules. If a malicious object is detected, the object will be quarantined and the IP address can be captured to block and investigate further.
Enable the server access logging and provide the target bucket where you want to save the logs.
Figure 6 – Logging configurations for the monitored S3 bucket.
Setting Up Shuffle SOAR Shuffle will be installed in a container environment using Docker:
Make sure you have Docker and docker-compose installed in your Amazon Elastic Compute Cloud (Amazon EC2) instance. Download Shuffle: git clone https://github.com/frikky/Shuffle cd Shuffle Fix prerequisites for the Opensearch database (Elasticsearch): mkdir shuffle-database sudo chown -R 1000:1000 shuffle-database Run docker-compose: docker-compose up -d When done, follow the below steps:
After installation, go to http://localhost:3001 (or your server name – https is on port 3443). Set up your admin account (username and password). Shuffle doesn’t have a default username and password. Sign in with the same username and password. Go to /apps and see if you have any apps yet. If not, you may need to configure proxies. Check out https://shuffler.io/docs/configuration as it has lots of useful information to get started. Creating the Workflow in Shuffle Workflows are the backbone of Shuffle, empowering you to automate your daily tasks by with a simple interface. Workflows use apps, triggers, conditions, and variables to make powerful automations in no time.
Clone the workflow from the following link: https://github.com/Infopercept/shuffle-workflows
Once the file is downloaded, you can upload the file from Shuffle console.
Figure 7 – Shuffle workflow for the S3 malware scanning.
The icons placed here depicts our workflow, and the S3 events will be collected and sent to Shuffle via webhook. This will notify Shuffle of any new object.
Breakdown of the Workflow A webhook from the Triggers panel will bring the events for correlation, when the object is placed in your S3 bucket. Shuffle tools will be used for setting various conditions and will help in correlation. Here, it will be used for parsing the URL of S3 object.
Figure 8 – Connecting webhook to shuffle tools.
Once the URL is parsed, the objects have to be fetched from the parsed URL.
Figure 9 – Fetching the S3 object for analysis.
To fetch objects from S3, it needs to be authenticated. Provide a name, access key, secret key, and region. Once the objects start to fetch from S3, YARA rules will start scanning if any object is placed in the S3 bucket.
Figure 10 – Object scanned by the YARA rules.
Any malicious object detected will be deleted automatically; malicious files can also be quarantined for further analysis.
Let’s take it a step further and explore how the Shuffle orchestrator can be leveraged for different flows. Consider that not only the malicious file needs to be detected, but the uploader IP address should be captured and blocked. Also, when an IP address is blocked put an alert in a slack channel.
Check out this reference link to configure Slack authentication application.
Add the malicious IP address to Amazon GuardDuty threat list to build threat intel.
Figure 11 – Adding the malicious IP address to GuardDuty.
It will be downloading the threat list add the IP address detected and upload the threat list again to the S3 bucket.
Figure 12 – Sending the alert notification to the Slack channel.
That’s it! Now save the workflow and you are good to go.
Conclusion
Through the setup detailed in this post, you can secure Amazon S3 from getting infecting through malicious threat actors using this serverless solution. The workflow scans files from YARA rules and sends malicious IP addresses to Amazon GuardDuty so adversaries do not affect the environment in the future. You can delete or even quarantine files, along with blocking the suspicious IP addresses.
Shuffle provides flexibility to build custom response automation flows, enabling you to build your own playbooks. Infopercept provides implementation and consultation services to help you build your own playbooks.