[Filebeat][New Module] Zoom webhook module

[P1llus]

What does this PR do?

This PR creates a module and fileset for Zoom webhooks, allowing filebeat to ingest webhooks initiated from Zoom and map it to ECS.

Why is it important?

Adds a new supported product to Filebeat.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• Need to check if the size of fields.yml is still okay
• Needs a second iteration to look over the different mappings and compare them to https://marketplace.zoom.us/docs/api-reference/webhook-reference/
• Need to add test data
• Need to add support for Zoom secret (in header). Will be done in a separate PR: [Filebeat][http_endpoint input] Adds support for custom auth header names and secret #20435

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request #20414 updated]

• Start Time: 2020-09-14T09:21:23.224+0000

• Duration: 51 min 53 sec

Test stats 🧪

Test	Results
Failed	0
Passed	5592
Skipped	825
Total	6417

[andrewkroh]

CI is failing due to:

Error generating Index Pattern: field <zoom.meeting.settings.auto_recording> is duplicated, remove it or set 'overwrite: true', {Name:meeting.settings.auto_recording Type:keyword Description:Settings related to the meeting

[P1llus]

Need to fix a few more outputs from the test logs, but except that it should be pretty complete now.

[P1llus]

@andrewstucki didn't expect you to be this quick, was thinking about doing one final run through fields.yml and fix up some descriptions since plenty of them had been changed a bit. :)

For the zoomroom bit, the fieldname is related to the webhook endpoint. So meeting endpoint ends up under zoom.meeting.* while the zoomroom endpoint then ends up with the unfortunate name of zoom.zoomroom.* . It does have a special relation to something called zoomroom though, its conference rooms with dedicated equipment connected to zoom, which has its on API endpoint.

[P1llus]

Okay now I think the fields.yml is ready for a review @andrewstucki :) Updated types and added better descriptions

[andrewstucki]

initial comments

[webmat]

I've given the fields a quick look, and entered comments for some of the things I've seen. This is not a thorough review :-)

One thing of note, in ECS a community member has recently opened an ECS RFC (RFC 0003) to capture audit logs on random things in accounts like Zoom meetings, and other. The RFC is still in the very early stages, so no action needed for now. But just putting the idea out there, that there's a place coming in ECS for audit logs on "everything else" that doesn't have a clean place in ECS (as opposed to audit logs on files, users, etc) :-)

[P1llus]

Will add in the last two comments tomorrow as they require a bit more work, but the rest should now have been marked resolved

[P1llus]

This should also wait for #20435 to be merged first, so that the new config items is added to the manifest, else the builds will fail.

[P1llus]

There has been a lot of changes into how similar fields are merged and how dates, timestamps, event.start/end and duration is calculated, so might need a fresh review. All the PR comments have been incorporated as well.

[P1llus]

Running some last tests on this and then it should be ready for one last review

[P1llus]

jenkins test this please

[P1llus]

@andrewkroh Added the SSL option, though the documentation for that is a bit more tricky, I don't think it would be sufficient to add in the docs for the common ssl options or? The input itself is using "tls.common", so it does work the same, but wanted to hear your opinion if we wanted a custom doc for it or not.

Anything else needed to be reviewed for this module before we find it complete?

[andrewkroh]

Linking to the shared SSL docs should be fine, similar to what Okta does. The inputs that accept TLS connections use those docs.

[P1llus]

Jenkins test this please

[P1llus]

jenkins, test this

[marc-gr]

In case CI issues persist for a while, locally the zoom tests are passing green:

>> python test: Integration Testing
============================================== test session starts ===============================================
platform linux -- Python 3.7.3, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /go/src/github.com/elastic/beats, configfile: pytest.ini
plugins: timeout-1.3.4
timeout: 90.0s
timeout method: signal
timeout func_only: True
collected 22 items                                                                                               

tests/system/test_filebeat_xpack.py ....                                                                   [ 18%]
tests/system/test_http_endpoint.py .........                                                               [ 59%]
tests/system/test_xpack_modules.py .........                                                               [100%]

================================================ warnings summary ================================================
x-pack/filebeat/tests/system/test_filebeat_xpack.py::FilebeatXPackTest::test_export_config
  /go/src/github.com/elastic/beats/filebeat/tests/system/../../../libbeat/tests/system/beat/common_tests.py:70: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
    yml = yaml.load(output)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
----- generated xml file: /go/src/github.com/elastic/beats/x-pack/filebeat/build/TEST-python-integration.xml -----
============================================== slowest 20 durations ==============================================
13.37s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_0_zoom
3.51s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_3_zoom
3.35s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_5_zoom
3.28s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_7_zoom
3.13s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_8_zoom
2.97s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_1_zoom
2.91s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_4_zoom
2.86s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_2_zoom
2.86s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_6_zoom
2.28s call     x-pack/filebeat/tests/system/test_filebeat_xpack.py::FilebeatXPackTest::test_export_index_pattern
2.03s call     x-pack/filebeat/tests/system/test_filebeat_xpack.py::FilebeatXPackTest::test_export_template
1.68s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_request
1.40s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_wrong_content_header
1.39s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_correct_auth_header
1.39s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_malformed_json
1.38s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_get_request
1.37s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_wrong_auth_header
1.36s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_wrong_auth_value
1.35s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_empty_body
1.29s call     x-pack/filebeat/tests/system/test_filebeat_xpack.py::FilebeatXPackTest::test_export_config
==================================== 22 passed, 1 warning in 61.50s (0:01:01) ====================================
>> python test: Integration Testing Complete