Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add threat intelligence for 2.12 #6273

Merged
merged 9 commits into from
Feb 8, 2024
59 changes: 34 additions & 25 deletions _security-analytics/sec-analytics-config/detectors-config.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,9 @@
To quickly select one or more known rules and dismiss others, first deselect all rules by turning off the **Rule name** toggle, then search for your target rule names and select each individually by turning its toggle on.
{: .tip }

1. Review the field mappings. Field mappings allow the system to accurately pass event data from the log to the detector and then use the data to trigger alerts. For more information about field mappings, see the **About field mappings** section later in this topic.
1. Review the field mappings. Field mappings allow the system to accurately pass event data from the log to the detector and then use the data to trigger alerts. For more information about field mappings, see the [A note on field names](#a-note-on-field-names) section.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

1. Choose whether to enable [threat intelligence]({{site.url}}{{site.baseurl}}/security-analytics/usage/detectors#threat-intelligence-feeds) feeds. Threat intelligence feeds only work with **standard** log types.

1. In the **Detector schedule** section, create a schedule for how often to run the detector. Specify a unit of time and a corresponding number to set the interval. The following image shows that the detector runs every 3 minutes.

Expand Down Expand Up @@ -94,38 +96,16 @@

---

## About field mappings
The data source (log index), log type, and detection rules specified in the first step determine which fields are available for mapping. For example, when "Windows logs" is selected as the log type, this parameter, along with the specific detection rules, determines the list of detection field names available for the mapping. Similarly, the selected data source determines the list of log source field names that are available for the mapping.

The system uses prepackaged Sigma rules for detector creation. It can automatically map important fields for a specific log type to the corresponding fields in the Sigma rules. The field mapping step presents a view of automatically mapped fields while also providing the option to customize, change, or add new field mappings. When a detector includes customized rules, you can follow this step to manually map detector rule field names to log source field names.

Because the system has the ability to automatically map field names, this step is optional. However, the more fields that can be mapped between detector fields and log source fields, the greater the accuracy of generated findings.

### A note on field names
If you choose to perform manual field mapping, you should be familiar with the field names in the log index and have an understanding of the data contained in those fields. If you have an understanding of the log source fields in the index, the mapping is typically a straightforward process.

Security Analytics takes advantage of prepackaged Sigma rules for security event detection. Therefore, the field names are derived from a Sigma rule field standard. To make them easier to identify, however, we have created aliases for the Sigma rule fields based on the open-source Elastic Common Schema (ECS) specification. These alias rule field names are the field names used in these steps. They appear in the **Detector field name** column of the mapping tables.

Although the ECS rule field names are largely self-explanatory, you can find predefined mappings of the Sigma rule field names to ECS rule field names, for all supported log types, in the GitHub Security Analytics repository. Navigate to the [OSMappings](https://github.com/opensearch-project/security-analytics/tree/main/src/main/resources/OSMapping) folder and select the file for the specific log type. For example, to see the Sigma rule fields that correspond to ECS rule fields for the Windows log type, select the [`windows_logtype.json` file](https://github.com/opensearch-project/security-analytics/blob/main/src/main/resources/OSMapping/windows_logtype.json). The `raw_field` value in the file represents the Sigma rule field name in the mapping.

### Amazon Security Lake logs

[Amazon Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) converts security log and event data to the [Open Cybersecurity Schema Framework](https://docs.aws.amazon.com/security-lake/latest/userguide/open-cybersecurity-schema-framework.html) (OCSF) to normalize combined data and facilitate its management. OpenSearch supports ingestion of log data from Security Lake in the OCSF format, and Security Analytics can automatically map fields from OCSF to ECS (the default field-mapping schema).

The Security Lake log types that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **dns** when [defining a detector](#step-1-define-a-detector). Furthermore, because logs such as CloudTrail logs can conceivably be captured in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.

To reveal fields for a log index in either raw format or OCSF, use the [Get Mappings View]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/mappings-api/#get-mappings-view) API and specify the index in the `index_name` field of the request.
{: .tip }

### Automatically mapped fields
## Automatically mapped fields

Once you select a data source and log type, the system attempts to automatically map fields between the log and rule fields. Switch to the **Mapped fields** tab to show the list of these mappings. When the field names are similar to one another, the system can successfully match the two, as shown in the following image.

<img src="{{site.url}}{{site.baseurl}}/images/Security/automatic-mappings.png" alt="Field mapping example for automatic mappings" width="85%">

Although these automatic matches are normally dependable, it's still a good idea to review the mappings in the **Mapped fields** table and verify that they are correct and matched as expected. If you find a mapping that doesn't appear to be accurate, you can use the dropdown list to search for and select the correct field name. For more information about matching field names, see the following section.

### Available fields
## Available fields

The field names that are not automatically mapped appear in the **Available fields** table. In this table you can manually map detection rule fields to data source fields, as shown in the following image.

Expand All @@ -139,6 +119,35 @@
* Once the log source field name is selected and mapped to the detector field name, the icon in the **Status** column to the right changes from the alert icon to a check mark.
* Make as many matches between field names as possible to complete an accurate mapping for the detector and log source fields.

### A note on field names
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

If you choose to perform manual field mapping, you should be familiar with the field names in the log index and have an understanding of the data contained in those fields. If you have an understanding of the log source fields in the index, the mapping is typically a straightforward process.

Security Analytics takes advantage of prepackaged Sigma rules for security event detection. Therefore, the field names are derived from a Sigma rule field standard. To make them easier to identify, however, we have created aliases for the Sigma rule fields based on the following specifications:

- For all log types, the open-source Elastic Common Schema (ECS) specification.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Include a link to the spec?

Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
- For AWS CloudTrail, Domain Name System (DNS) log types, and virtual private network (VPC) flow logs, go to the [Open Cybersecurity Framework](https://github.com/ocsf/ocsf-schema).

These alias rule field names are the field names used in these steps. They appear in the **Detector field name** column of the mapping tables.

You can find predefined mappings of the Sigma rule field names to ECS rule field names for all supported log types in the following locations:
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

- The [Supported log types]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/log-types/) reference documentation.

- The [GitHub Security Analytics](https://github.com/opensearch-project/security-analytics) repository. To find the field mappings:
1. Navigate to the [OSMappings](https://github.com/opensearch-project/security-analytics/tree/main/src/main/resources/OSMapping) folder.

Check failure on line 138 in _security-analytics/sec-analytics-config/detectors-config.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _security-analytics/sec-analytics-config/detectors-config.md#L138

[OpenSearch.Spelling] Error: OSMappings. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.
Raw output
{"message": "[OpenSearch.Spelling] Error: OSMappings. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_security-analytics/sec-analytics-config/detectors-config.md", "range": {"start": {"line": 138, "column": 24}}}, "severity": "ERROR"}
2. Select the file for the specific log type. For example, to view the Sigma rule fields that correspond to ECS rule fields for the Windows log type, select the [`windows_logtype.json` file](https://github.com/opensearch-project/security-analytics/blob/main/src/main/resources/OSMapping/windows_logtype.json). The `raw_field` value in the file represents the Sigma rule field name in the mapping.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"specified" instead of "specific"?


## Amazon Security Lake logs

Check failure on line 141 in _security-analytics/sec-analytics-config/detectors-config.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _security-analytics/sec-analytics-config/detectors-config.md#L141

[OpenSearch.HeadingCapitalization] 'Amazon Security Lake logs' is a heading and should be in sentence case.
Raw output
{"message": "[OpenSearch.HeadingCapitalization] 'Amazon Security Lake logs' is a heading and should be in sentence case.", "location": {"path": "_security-analytics/sec-analytics-config/detectors-config.md", "range": {"start": {"line": 141, "column": 4}}}, "severity": "ERROR"}

[Amazon Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) converts security log and event data to the [Open Cybersecurity Schema Framework](https://docs.aws.amazon.com/security-lake/latest/userguide/open-cybersecurity-schema-framework.html) (OCSF) to normalize combined data and facilitate its management. OpenSearch supports ingestion of log data from Security Lake in the OCSF format, and Security Analytics can automatically map fields from OCSF to ECS (the default field-mapping schema).
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

The Security Lake log types that can be used as log sources for detector creation include AWS CloudTrail, Amazon Route 53, and Amazon VPC Flow Logs. Given that Amazon Route 53 is a log that captures DNS activity, its log type should be specified as **dns** when [defining a detector](#step-1-define-a-detector). Furthermore, because logs such as AWS CloudTrail logs can conceivably be captured in both raw format and OCSF, it is good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.

Check failure on line 145 in _security-analytics/sec-analytics-config/detectors-config.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _security-analytics/sec-analytics-config/detectors-config.md#L145

[OpenSearch.Spelling] Error: dns. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.
Raw output
{"message": "[OpenSearch.Spelling] Error: dns. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_security-analytics/sec-analytics-config/detectors-config.md", "range": {"start": {"line": 145, "column": 252}}}, "severity": "ERROR"}
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazon Route 53 is a service, not a log.


To reveal fields for a log index in either raw format or OCSF, use the [Get Mappings View]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/mappings-api/#get-mappings-view) API and specify the index in the `index_name` field of the request.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
{: .tip }


Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
## What's next

If you are ready to view findings generated by the new detector, see the [Working with findings]({{site.url}}{{site.baseurl}}/security-analytics/usage/findings/) section. If you would like to import rules or set up custom rules before working with findings, see the [Working with detection rules]({{site.url}}{{site.baseurl}}/security-analytics/usage/rules/) section.
Expand Down
10 changes: 10 additions & 0 deletions _security-analytics/usage/detectors.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,16 @@ To edit a detector, begin by selecting the link to the detector in the Detector
After you select the **Alert triggers** tab, you also have the option to add additional alerts for the detector by selecting **Add another alert condition** at the bottom of the page.
{: .tip }

### Threat intelligence feeds

A threat intelligence feed is a real-time, continuous data stream that gathers information related to risks or threats. The critical information in the tactical threat intelligence feed is called an “indicator of compromise” (IoC).
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a fan of the last sentence here (or my rewrite), as it's a little ambiguous. Can we be more specific about what we mean by "critical information"? In other words, what information exactly is called an IoC (what makes it critical)?


As of OpenSearch 2.12, you can enable threat intelligence for Sigma rules related to malicious IP addresses.

To enable threat intelligence feeds, select the **Enable threat intelligence-based detection** option.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know we can't change this at the moment, but just FYI, it should be "threat-intelligence-based detection".


Threat intelligence feeds only work with **standard** log types.

---
## Detector actions

Expand Down
Loading