Changelog

This changelog documents all notable user-facing changes of Threat Bus.

Every entry has a category for which we use the following visual abbreviations:

2021.07.29

🐞 Threatbus now only attempts to load plugins that are explicitly listed in the config file. #150
🎁 Many configuration options for threatbus and pyvast-threatbus now have default values. See the example configs for a detailed list. #150
🐞 The content and format of the threatbus-zmq-app plugin's subscription success response has changed. Prior to this change, the plugin used to respond with an endpoint in the host:port format, which contained a wrong hostname (e.g., 0.0.0.0 instead of a publicly reachable topic). From now on, the plugin returns only the ports for pub and sub communication and leaves it to the subscribing app to connect with the right host/IP. #140
⚠️ Threat Bus now uses Dynaconf for configuration management. Configuration via a config file works exactly as it has worked before. Users can provide a path to the config file using the -c option. Threat Bus now considers files named config.yaml and config.yml as default configs if located in the same directory. Additionally, Threat Bus now supports configration via environment variables and .dotenv. Env vars need to be prefixed with THREATBUS_ to be respected and always take precedence over values in config files. #133

⚠️ The official tenzir/threatbus Docker image now uses Debian:Bullseye as base image and uses the Debian-maintained Broker libraries to enable communication with Zeek. #132
⚠️ The Threat Bus community chat moved from Gitter to Slack. Join us in the #threatbus channel for interactive discussions. #123

🎁 Threat Bus now supports subscriptions for multiple topics. The zmq-app-plugin implements those multi-topic subscriptions in a backwards-compatible way. Subscribers benefit from this change, as they only get assigned a single point-to-point topic for their subscription, instead of one point-to-point topic for every subscribed Threat Bus topic. #120
⚠️ The -c / --config parameter is now explicitly required to start Threat Bus. Starting without it will print a helpful error message. #119
🎁 We now provide a simple asyncio template for writing applications that connect to Threat Bus via ZeroMQ. #118
⚠️ The threatbus-zeek plugin now uses the timestamp of Zeek intel matches to set the last_seen property of resulting STIX-2 Sightings, instead of setting the created timestamp. The created timestamp now always refers to the actual creation time of the sightings. #117

🐞 We fixed a bug in the ZeroMQ app plugin that threw an exception upon receiving SnapshotRequests. #116
⚠️ The Threat Bus community chat moved from Element to Gitter. Join us at gitter.im/tenzir/threatbus or via Matrix at #tenzir_threatbus:gitter.im. #113

🎁 The CIFv3 plugin now supports the STIX-2 (version 2.1) standard for Indicators. The plugin converts STIX-2 Indicators on best-effort basis to CIFv3 indicators before forwarding them to the configured CIF endpoint. #106
🐞 We fixed a bug in the routing logic for SnapshotRequests. Apps can now request snapshots as expected for all stix2-prefixed topics. #103
🎁 The Zeek plugin now supports the STIX-2 (version 2.1) standard for Indicators and Sightings. The plugin converts STIX-2 Indicators on best-effort basis to Zeek Intel items before forwarding them to Zeek. Likewise, the plugin converts Zeek sightings to valid STIX-2 Sightings before publishing them on Threat Bus topics. #103
🎁 The MISP plugin now supports the STIX-2 (version 2.1) standard for Indicators and Sightings. The plugin converts MISP attributes to valid STIX-2 Indicators on best-effort basis before publishing them on Threat Bus topics. Likewise, the plugin converts STIX-2 Sightings to MISP sightings before sending them the MISP. #102
🐞 We fixed a bug in the JSON (de-)serialization logic for SnapshotEnvelopes and SnapshotRequests that lead to a malformed type field in the JSON representations of both types. #102

🎁 The MISP plugin now uses extra dependencies. Users can now chose the wanted dependencies during installation by running pip install threatbus-misp[zmq] to install the ZeroMQ dependency, or pip install threatbus-misp[kafka] to install the Kafka dependency. The plugin throws a fatal error if none of these dependencies is installed and exits immediately. #99
🎁 The RabbitMQ backbone plugin, the In-memory backbone plugins, and the Zmq-app plugin now support the STIX-2 (version 2.1) standard for Indicators and Sightings. #97
⚡️ Threat Bus now uses STIX-2 (version 2.1) as internal transport format for Indicators and Sightings. App-plugins now have to provide required mappings between the STIX-2 format and app-specific formats (e.g., the Zeek plugin needs to map STIX-2 to the Zeek intel format). The home-made types threatbus.data.Sighting and threatbus.data.Indicator are removed from the codebase. Plugins with versions earlier than 2021.02.24 are incompatible to the new Threat Bus version. #97

🎁 New systemd unit files are now available in the Threat Bus repository to run both Threat Bus and pyvast-threatbus as system services. #77

🐞 The RabbitMQ backbone plugin ignored user-defined queue parameters, such as durable or lazy queues. It now respects such parameters again. #76

🐞 The Zeek app did not perform an outbound connection to Threat Bus in cluster mode. Now the master peers with Threat Bus to establish a connection. #68
🎁 The zmq-app and zeek plugins now use the Unix select system call for improved performance during message passing. The previous approach impacted the performance with a constant delay for every message and did not scale. The new approach saves at least that constant factor per message. For ZeroMQ publishing we observed a speedup of approximately factor 183 for 100k events. #61
🎁 The rabbitmq backbone plugin now uses an asynchronous SelectConnection instead of a blocking one. We measured a speedup of approximately factor 1.2 for 100k events. #61
🎁 Threat Bus now has a controlled shutdown. Pressing ctrl+c first shuts down backbone plugins, then app plugins, and lastly Threat Bus itself. #61
⚠️ There exists a new base class for implementing plugin-threads. Plugin developers should extend the new StoppableWorker for every plugin. Threat Bus and all plugins in this repository now implement that class. #61
⚠️ Threat Bus and all plugins now use multiprocessing.JoinableQueue for message passing. #61
🎁 The zmq-app plugin now supports synchronous heartbeats. With heartbeats, both Threat Bus and the connected apps can mutually ensure that the connected party is still alive. #58

🎁 The MISP plugin now works without a valid PyMISP API connection. If omitted in the configuration, the plugin can still receive indicators via ZeroMQ or Kafka, but it cannot report back sightings or request snapshots. #55
🎁 The MISP plugin now supports a whitelist-filtering mechanism. Users can specify required properties of IoCs (MISP attributes) in the configuration file. The filter is implemented for IoCs that are received via ZeroMQ or Kafka as well as IoCs that are requested as part of a snapshot. #49
🎁 The generic Threat Bus ZeroMQ application plugin has replaced the former VAST plugin. Any app that communicates via ZeroMQ can implement this plugin's protocol to connect with Threat Bus effortlessly. #46