[Tuning] SDH - Removing timestamp_overide for Threat Match Rules

[imays11]

Pull Request

Issue link(s):
https://github.com/elastic/sdh-protections/issues/539

Summary - What I changed

Removing timestamp_override = "event.ingested" from threat match rules as the event.ingested field is no longer ingested as part of the APM ingest pipeline causing rule failure for customers as of 8.15. This change should fix the issue and back port to the necessary stacks with our next rules release. The use of event.ingested is a best practice we use at the team level but is not a requirement at the rule level so this change should not cause any issues as the default @timestamp field will still work.

[protectionsmachine]

Rule: Tuning - Guidelines

These guidelines serve as a reminder set of considerations when tuning an existing rule.

Documentation and Context

• Detailed description of the suggested changes.
• Provide example JSON data or screenshots.
• Provide evidence of reducing benign events mistakenly identified as threats (False Positives).
• Provide evidence of enhancing detection of true threats that were previously missed (False Negatives).
• Provide evidence of optimizing resource consumption and execution time of detection rules (Performance).
• Provide evidence of specific environment factors influencing customized rule tuning (Contextual Tuning).
• Provide evidence of improvements made by modifying sensitivity by changing alert triggering thresholds (Threshold Adjustments).
• Provide evidence of refining rules to better detect deviations from typical behavior (Behavioral Tuning).
• Provide evidence of improvements of adjusting rules based on time-based patterns (Temporal Tuning).
• Provide reasoning of adjusting priority or severity levels of alerts (Severity Tuning).
• Provide evidence of improving quality integrity of our data used by detection rules (Data Quality).
• Ensure the tuning includes necessary updates to the release documentation and versioning.

Rule Metadata Checks

• updated_date matches the date of tuning PR merged.
• min_stack_version should support the widest stack versions.
• name and description should be descriptive and not include typos.
• query should be inclusive, not overly exclusive. Review to ensure the original intent of the rule is maintained.

Testing and Validation

• Validate that the tuned rule's performance is satisfactory and does not negatively impact the stack.
• Ensure that the tuned rule has a low false positive rate.

[Mikaayenson]

@imays11 it may be good to go back to APM in the original slack thread and ask why they removed support and if we really need to do this.

[imays11]

I'm closing this PR, we will not need to remove the event.ingested requirement for now as APM is going to reintroduce the field at least temporarily. They are still considering removing it in 9.x which would not be ideal. I left some context here in the SDH issue on why we actually use that field in place of @timestamp and hope this will be considered by the APM team.