Rewrite Zeek plugin to convert to/from STIX-2 Indicators and Sightings

[0snap]

📔 Description

Following up on the STIX-2 rewrite of Threat Bus: this PR updates the Zeek plugin.

• Convert STIX-2 Indicators to Zeek/Broker events
• Update Zeek script to consume Zeek/Broker events & ingest in Intel framework
• Convert sightings / matches from Zeek (Broker events) to STIX-2 valid Sightings
• Updated unit tests
• Integration-test the Zeek-plugin's Broker interface in isolation
• Integration-test with a full Zeek instance that runs the Threat Bus app script
• Update / separate from ZMQ message-passing integration test

📝 Checklist

• All user-facing changes have changelog entries.
• The changes are reflected on docs.tenzir.com/threatbus, if necessary.
• The PR description contains instructions for the reviewer, if necessary.

🎯 Review Instructions

Do at least a code-review & run the unit tests. If you want to try out more, here is how:

Integration testing:
You require a local Broker installation with Python bindings for integration testing

• Follow the instructions from the Zeek plugin's README
• Run the integration tests

Interactive usage:
You require local Zeek and Broker installations with Python bindings for interactive usage

• Make sure your local Zeek and Broker installations have matching versions (i.e., Zeek 4 -> Broker 2.x, Zeek 3.2 -> Broker 1.4)
• Start Threat Bus with the Zeek plugin, in-mem backbone plugin, and the Zmq-app plugin
• Start a local Zeek instance using the Threat Bus app script and monitor your local network interface (sudo zeek -i wlp0s20f3 -C apps/zeek/threatbus.zeek)
• Use the test utils to send STIX-2 indicators to the Zmq-endpoint -> that should ultimately be mapped via Broker and arrive in Zeek. Send something like example.com.
• Use curl or similar to invoke example.com -> Zeek should match that immediately, report a Sighting to Threat Bus and that should be logged as valid STIX-2 (debug logging)

[lgtm-com]

This pull request introduces 2 alerts when merging 5ac2124 into 1ba54e7 - view on LGTM.com

new alerts:

• 2 for Unused import

[lgtm-com]

This pull request introduces 2 alerts when merging 5cba369 into bf33351 - view on LGTM.com

new alerts:

• 2 for Unused import

[lgtm-com]

This pull request introduces 2 alerts when merging 90b452f into a082f2d - view on LGTM.com

new alerts:

• 2 for Unused import

[lgtm-com]

This pull request introduces 2 alerts when merging 73b3762 into a082f2d - view on LGTM.com

new alerts:

• 2 for Unused import

[lgtm-com]

This pull request introduces 1 alert when merging b4d40a5 into a082f2d - view on LGTM.com

new alerts:

• 1 for Except block handles 'BaseException'

[lgtm-com]

This pull request introduces 1 alert when merging b4d40a5 into 5ad3665 - view on LGTM.com

new alerts:

• 1 for Except block handles 'BaseException'

[lgtm-com]

This pull request introduces 1 alert when merging 67c46e0 into 5ad3665 - view on LGTM.com

new alerts:

• 1 for Except block handles 'BaseException'

[lgtm-com]

This pull request introduces 1 alert when merging ce57817 into 5ad3665 - view on LGTM.com

new alerts:

• 1 for Except block handles 'BaseException'

[tobim]

I tested this locally and can vouch that sightings are logged from threatbus when zeek detects them.

I didn't do a very thorough code review, most of it is in tests which are repetitive by nature. I only have a few suggestions and questions which can be resolved by adding comments.

[tobim]

Tested again with the latest changes. Everything seems to work as expected.