[Meta][Amazon Security Lake] Supporting OCSF v1.1

[jamiehynds]

Our current Amazon Security Lake supports OCSF v1.0, which was the latest version of the schema when we initially shipped the integration. The OCSF schema has evolved since, and is now at v1.1.

Our Security Lake pipelines need to be adjusted to ensure we're inline with v1.1, including new event classes, objects and categories. Backward compatibility is not a significant concern in this case, as an example our security findings pipeline can be deprecated, as security findings were deprecated in OCSF v1.1. Related dashboards will also need to be removed, and new ones added to account for new classes introduced in v1.1

Once we have updated pipelines to support the latest OCSF version, we'll create an issue to build a generic OCSF to ECS package, for users who would like to ingest OCSF formatted data outside of Security Lake.

For a full list of v1.1 changes we need to adhere to, please see here: https://github.com/ocsf/ocsf-schema/blob/main/CHANGELOG.md#v110---january-25th-2024

Update: OCSF v1.2 is now available - https://github.com/ocsf/ocsf-schema/blob/main/CHANGELOG.md#v120---april-23rd-2024 - will let @ShourieG decide if we make the necessary changes for both 1.1 and 1.2 in this issue, or create a separate isseu for v1.2

Tasks

Beta Give feedback

1. Add support for system tests
2. Add 3 new events types in the same data stream (with separate mappings)
3. [Amazon Security Lake]Re-work ingestion pipelines to support dynamic mappings #10219
   Integration:amazon_security_lake Team:Security-Service Integrations enhancement +3
   
4. [Amazon Security Lake]Perform a pass over respective dashboards and update/add any dashboard element or tables if required #10266
   Integration:amazon_security_lake Team:Security-Service Integrations enhancement integration +4
   
5. [Fleet] - Support dynamic templates for "date" types and array of objects containing same type of attributes kibana#187951
   Team:Fleet enhancement +2
6. Allow data stream overrides in system tests for specific integration packages elastic-package#1917
   Team:Ecosystem enhancement +2
7. [Amazon Security Lake] Add support for new objects and event classes, profiles and update schemas accordingly #10740
   Integration:amazon_security_lake Team:Security-Service Integrations enhancement integration +4
   

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[ShourieG]

@jamiehynds, as discussed we are initially starting the upgrade to OCSF v1.1 and targeting the release by 8.15. Post the upgrade to v1.1 we will start with the upgrades for supporting v1.2.

[ShourieG]

We are hitting the upper limit of 2048 fields per data stream while performing the ocsf v1.1 upgrade due to addition of new object types to base objects. I've discussed with the ecosystem team and an issue has been opened for addressing this limitation in future. Right now we might have to keep these new objects in a flattened state to bypass these limitations. cc: @jamiehynds @andrewkroh

[ShourieG]

@narph @jamiehynds, PR is now merged. The issue could still be kept open since there are couple of related issues which are yet to be solved.