[ML] Adding v3 modules for Security_Linux and Security_Windows and Deprecating v1 + v2

[bfilar]

Summary

• Follow up to a closed PR in 8.1 [ML] Adding V3 modules for Security_Linux and Security_Windows #123274
• This PR seeks to consolidate the current set of Security ML Modules: https://github.com/elastic/security-team/issues/2960
• Added files for new + refactored v3 jobs for both security_linux and security_windows - for use within the Security app.
• Deprecated v1 and v2 versions of those jobs
• Deprecated Auditbeat Host Process module due to a duplicative job in the Linux Security module
• Consolidated auth modules for Linux and Windows will be consolidated into their respective OS modules

Files/Job Artifacts:

---

• 2 updated manifest .json files - for both linux and windows

• Updated/new ML Job configurations for 26 jobs - each with associated datafeed configuration files:

  • security_linux: 14 jobs

    • v3_linux_anomalous_network_activity
    • v3_linux_anomalous_network_port_activity_ecs
    • v3_linux_anomalous_process_all_hosts_ecs
    • v3_linux_anomalous_user_name_ecs
    • v3_linux_network_configuration_discovery
    • v3_linux_network_connection_discovery
    • v3_linux_rare_metadata_process
    • v3_linux_rare_metadata_user
    • v3_linux_rare_sudo_user
    • v3_linux_rare_user_compiler
    • v3_linux_system_information_discovery
    • v3_linux_system_process_discovery
    • v3_linux_system_user_discovery
    • v3_rare_process_by_host_linux_ecs

  • security_windows: 12 jobs

    • v3_rare_process_by_host_windows_ecs
    • v3_windows_anomalous_network_activity_ecs
    • v3_windows_anomalous_path_activity_ecs
    • v3_windows_anomalous_process_all_hosts_ecs
    • v3_windows_anomalous_process_creation
    • v3_windows_anomalous_script
    • v3_windows_anomalous_service
    • v3_windows_anomalous_user_name_ecs
    • v3_windows_rare_metadata_process
    • v3_windows_rare_metadata_user
    • v3_windows_rare_user_runas_event
    • v3_windows_rare_user_type10_remote_login

Tests:

---

Individual job test tracking stats available here: https://docs.google.com/spreadsheets/d/1JOUIVsitaMdEdhM3WT2Eag4ELI-rI2Jec7bXildJsdQ/edit#gid=0

@randomuserid to also post more updates as needed to this issue + regarding tests, thanks

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[randomuserid]

@elasticmachine merge upstream

[randomuserid]

@elasticmachine merge upstream

[peteharverson]

One of the ML API integration tests in x-pack/test/api_integration/apis/ml/modules/setup_module.ts is failing here as it is trying to run against the siem_auditbeat_auth module which has been deleted here. I can't tell if the module_siem_auditbeat data set used in this test works against the security_auth module. If so, you could just switch the siem_auditbeat_auth module referenced in this test. If not, this test will need removing. Unless we have another data set we could use here @pheyos ?

[pheyos]

Fixing the tests requires a few more adjustments, I'm currently looking into the details.

[randomuserid]

@elasticmachine merge upstream

[randomuserid]

@elasticmachine merge upstream

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 574d41c
• Storybooks Preview
• Webpack Bundle Analyzer

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	4.8MB	4.8MB	-74.0B

History

• 💔 Build #32896 failed 3a5ee00
• 💔 Build #32858 failed d159f08
• 💔 Build #32830 failed 3206df9
• 💔 Build #32731 failed e93ab8a
• 💔 Build #32654 failed b59d8b9

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @randomuserid @bfilar

[randomuserid]

Many of the tests need to be updated with the deprecation of the v1 modules. New tests may be needed for the v3 modules. This work will need to be costed and planned so we're moving this forward to 8.3 for now.