[Winlogbeat] Add support for FileDelete events (event id 23) to sysmon module

[marc-gr]

What does this PR do?

FileDelete events were added in Sysmon v11, this PR adds support for FileDelete events (event id 23) to sysmon module.

Why is it important?

Prior to this change processing such events lead to an 'unexpected sysmon event_id' error message.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

TODO

• Diff the schema XML from v10 to v11 and make updates.
• Empty strings are replaced with “-“ to work around a WEF bug
• Handle Event ID 23
• Add .evtx test files from v11.

Related issues

Closes #18094

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request [Winlogbeat] Add support for FileDelete events (event id 23) to sysmon module #18340 updated, Pull request [Winlogbeat] Add support for FileDelete events (event id 23) to sysmon module #18340 updated]

• Start Time: 2020-05-07T15:49:28.910+0000

• Duration: 44 min 49 sec (2628887)

• Commit: 69510ad

Test stats 🧪

Test	Results
Failed	0
Passed	263
Skipped	1
Total	264

[andrewkroh]

The process namespace is "managed" by ECS. If archived and is_executable are not part of ECS then we should not add them to process.

Either leave them as-is or copy them over to a sysmon namespace like sysmon.is_archived. Then add an entry to this file for the field. https://github.com/elastic/beats/blob/master/x-pack/winlogbeat/module/sysmon/_meta/fields.yml The benefit of copying them is that we can set the correct data type. Leaving them in winlog.event_data means they will be a keyword in Elasticsearch rather than boolean.

[marc-gr]

Added them to sysmon.file.archived and sysmon.file.is_executable 👍

[andrewkroh]

LGTM. Congrats on your first Beats PR 🍾 .

As a follow-up pull request, I noticed a few issues with the module as a whole that we should address.

• Set related.hash.
• Set file.extension/name/directory.
• hash.* is not part of ECS. It should be used as file.hash.* or process.hash.*. We can't delete the existing hash.* fields until 8.0, so for 7.x we could populate them both. And then do a breaking change for 8.0 where we drop hash.*.

[andrewkroh]

@marc-gr I opened #18364 to track those additional changes. And I disucssed the rule.name: - with @andrewstucki and we think the field should be dropped so I added that to the issue.

Normally you will merge the PRs after they are approved and green on CI, but I wanted to get this into the 7.8 branch asap so that it can be included in 7.8.0. This still needs to be backported into the 7.x branch. Can you please use the dev-tools/cherrypick_pr script to do that (./dev-tools/cherrypick_pr --create_pr 7.x 18340 0a327bb). Read the script b/c you need to setup a github auth token first.