feat: simplified components

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "u-siem"
-version = "0.7.0"
+version = "0.8.0"
 authors = ["Samuel Garcés <samuel.garces@protonmail.com>"]
 license = "MIT"
 description = "A framework for building custom SIEMs"

README.md

@@ -1,6 +1,6 @@
 # uSIEM
 
-[![crates.io](https://img.shields.io/crates/v/u-siem.svg?style=for-the-badge&logo=rust)](https://crates.io/crates/u-siem-core) [![documentation](https://img.shields.io/badge/read%20the-docs-9cf.svg?style=for-the-badge&logo=docs.rs)](https://docs.rs/u-siem-core) [![MIT License](https://img.shields.io/crates/l/u-siem?style=for-the-badge)](https://github.com/u-siem/u-siem-core/blob/main/LICENSE) [![Rust](https://img.shields.io/github/actions/workflow/status/u-siem/u-siem-core/rust.yml?style=for-the-badge)](https://github.com/u-siem/u-siem-core/workflows/Rust/badge.svg?branch=main)
+[![crates.io](https://img.shields.io/crates/v/u-siem.svg?style=for-the-badge&logo=rust)](https://crates.io/crates/u-siem) [![documentation](https://img.shields.io/badge/read%20the-docs-9cf.svg?style=for-the-badge&logo=docs.rs)](https://docs.rs/u-siem) [![MIT License](https://img.shields.io/crates/l/u-siem?style=for-the-badge)](https://github.com/u-siem/u-siem-core/blob/main/LICENSE) [![Rust](https://img.shields.io/github/actions/workflow/status/u-siem/u-siem-core/rust.yml?style=for-the-badge)](https://github.com/u-siem/u-siem-core/workflows/Rust/badge.svg?branch=main&style=for-the-badge)
 
 Framework definitions that allow building a custom SIEM. Code you own SIEM like you code your own web page.
 
@@ -37,46 +37,121 @@ You can see a more updated document here: https://github.com/u-siem/parser-bench
 ```
 Note: It has changed quite a lot and is still changing.
 
-## Node Types
-
-### Kernel
-The kernel will be executed in a single thread with high priority and will be responsible of scaling the number of nodes of each type if it detects a congestion in a element. It will alse route messages between nodes.
-
-### Commander
-This component will accept commands from a user and sent it to be routed by the kernel to specific nodes.
-
-### InputNode
-It ingests logs and process them.
-We can support elasticsearch type (Like API-REST) or syslog.
-
-### ParsingNode
-This node will be the most important and will be able to parse the logs sent to him by the input node.
-There will be a special node still being designed to process logs like MySQL or Linux logs that are multiline.
-
-### EnrichmentNode
-It adds information about the IP, if its in a blocklist, if its a AmazonWebServices/Azure/GoogleCloud IP, if the IP has never been seen it then it contacts the GatheringNode to find information about that IP. It adds the tag "never_seen_ip" in that cases. It uses datasets to access information in a non-blocking form. See https://1drv.ms/p/s!AvHbai5ZA14wjV9J4rbBlSWyIw0t?e=AgBWNf
-
-### GatheringNode
-Consults feeds or databases like AbuseIP/Virus total to know if the IP is malicios or not, the same with domains and Hashes. It then updates the appropiated Dataset with the info to enrich future logs.
-
-### IndexingNode
-Send logs to index in the database (elasticsearch/SQLite/Postgres...) and queries them when asked.
-
-### RuleNode
-Set conditions for logs and fires alerts when the conditions matches.
-If a Rule is battletested, then we can tell the SOAR node to do things.
-https://github.com/Neo23x0/sigma/tree/master/rules/windows
-
-### AlertNode
-Creates alerts and sents them to another SIEM or stores them locally to use the native UI. Uses templates for the alerts.
+## Components
+Each node that receives messages is called a component, and they are managed by the Kernel component. uSIEM does not impose restrictions over how the kernel is designed/developed, but it does for the rest of the components. 
+A component is best defined as a piece of code that listens for events, runs in its own thread and it managed by the Kernel. For a component to be managed by the Kernel it must implement the SiemComponent or the SimplifiedComponent trait.
+This last one lets us design components focused on functionality, the simplified components are later wrapped inside a SiemComponent like the [SimpleComponent](./src/components/simplified.rs).
+```rust
+pub trait SiemComponent: Send {
+    fn name(&self) -> &'static str {
+        "SiemComponent"
+    }
+    /// Get the channel to this component
+    fn local_channel(&self) -> Sender<SiemMessage>;
+    /// Sets the channel used to receive/send logs. It's the kernel who sets the channel
+    fn set_log_channel(&mut self, sender: Sender<SiemLog>, receiver: Receiver<SiemLog>);
+
+    /// Execute the logic of this component in an infinite loop. Must be stopped using Commands sent using the channel.
+    fn run(&mut self) -> SiemResult<()>;
+
+    /// Allow to store information about this component like the state or configurations.
+    fn set_storage(&mut self, conn: Box<dyn SiemComponentStateStorage>);
+
+    /// Capabilities and actions that can be performed by this component
+    fn capabilities(&self) -> SiemComponentCapabilities;
+
+    /// Allows the Kernel to duplicate this component
+    fn duplicate(&self) -> Box<dyn SiemComponent>;
+
+    /// Initialize the component with the datasets before executing run
+    fn set_datasets(&mut self, datasets: DatasetHolder);
+}
+```
 
-### SoarNode
-Do actions automatically, like blocking IPs, domains...
-OpnSense supports blocking IPs with a simple API-REST call, the same for Cortex XDR.
-For PaloAlto: https://panos.pan.dev/docs/apis/panos_dag_qs
-Work in progress: define a custom trait that can be used with a common component as to simplify design. So we only need to import a library that defines the actions to be done (like an API call) and works in any custom SOAR component.
+The simplified version makes responding to external events a simple task thanks to the on_xxxx methods.
+```rust
+/// Simplified SIEM component desing. All returned errors are logged to debug except the tick() function that ends the execution
+#[allow(unused_variables)]
+pub trait SimplifiedComponent : Send {
+    /// Allow to store information about this component like the state or configurations.
+    fn set_storage(&mut self, conn: Box<dyn SiemComponentStateStorage>) {}
+
+    /// Capabilities and actions that can be performed by this component
+    fn capabilities(&self) -> SiemComponentCapabilities;
+
+    /// Allows the Kernel to duplicate this component
+    fn duplicate(&self) -> Box<dyn SimplifiedComponent>;
+
+    /// Initialize the component with the datasets before executing run
+    fn set_datasets(&mut self, datasets: DatasetHolder) {}
+
+    /// Executed when the component receives a command to execute
+    fn on_command(&mut self, header : SiemCommandHeader, action : SiemCommandCall) -> SiemResult<()> {
+        Ok(())
+    }
+    fn on_response(&mut self, header : SiemCommandHeader, action : SiemCommandResponse) -> SiemResult<()> {
+        Ok(())
+    }
+    /// Executed when the component receives a log. Return Ok(None) to filter and remove the log
+    fn on_log(&mut self, log : SiemLog) -> SiemResult<Option<SiemLog>> {
+        Ok(Some(log))
+    }
+    /// Called when the component updates i
+    fn on_dataset(&mut self, dataset : SiemDataset) -> SiemResult<()> {
+        Ok(())
+    }
+    fn on_alert(&mut self, alert : SiemAlert) -> SiemResult<()> {
+        Ok(())
+    }
+    /// The component received a task to execute
+    fn on_task(&mut self, header : SiemCommandHeader, task : SiemTask)-> SiemResult<()> {
+        Ok(())
+    }
+    fn on_task_result(&mut self, header : SiemCommandHeader, result : SiemTaskResult)-> SiemResult<()> {
+        Ok(())
+    }
+    /// Errors are roported as debug
+    fn on_notification(&mut self, notifiation : Notification)-> SiemResult<()> {
+        Ok(())
+    }
+    /// Perform actions. Return the number of millisecons to call the tick function again, or None for not (Default). If ab error is returned the component will end its execution.
+    fn tick(&mut self) -> SiemResult<Option<u64>> {
+        Ok(None)
+    }
+}
+```
 
-An idea: Apply multiple simple rules (like DarkTrace) does to calculate the threat score associated with the event. That score is added to the total score of a user in a period of time (Slicing Window). It will be implemented in redis wit a ScoreSet of users-scores in periods of 15 min with each removed after 24 hours by default.
+```rust
+#[derive(Clone)]
+struct BasicComponent {}
+impl BasicComponent {
+    pub fn new() -> Self {
+        Self {}
+    }
+}
+impl SimplifiedComponent for BasicComponent {
+    fn on_log(&mut self, mut log : SiemLog) -> SiemResult<Option<SiemLog>> {
+        log.add_field("PROCESSED", "PROCESSED".into());
+        Ok(Some(log))
+    }
+
+    fn capabilities(&self) -> SiemComponentCapabilities {
+        SiemComponentCapabilities::new(
+            LogString::Borrowed("SimpleComponent"),
+            LogString::Borrowed("SimpleComponent"),
+            LogString::Borrowed(""),
+            vec![],
+            vec![],
+            vec![],
+            vec![],
+        )
+    }
+
+    fn duplicate(&self) -> Box<dyn SimplifiedComponent> {
+        Box::new(self.clone())
+    }
+}
+```
 
 ## [Datasets](./src/components/dataset/README.md)
 

src/components/dataset/holder.rs

@@ -1,4 +1,4 @@
-use super::{SiemDataset, SiemDatasetType};
+use super::{SiemDataset, SiemDatasetType, text_map::TextMapSynDataset, geo_ip::GeoIpSynDataset, i18n::I18nSynDataset, ip_map::IpMapSynDataset, ip_map_list::IpMapListSynDataset, ip_set::IpSetSynDataset, text_set::TextSetSynDataset, text_map_list::TextMapListSynDataset, ip_net::IpNetSynDataset};
 use std::collections::BTreeMap;
 
 /// The dataset holder allows access to the latest version of a dataset almost instantly without the need to check if there is an update of a dataset using a channel as was done previously.
@@ -39,4 +39,59 @@ impl DatasetHolder {
 
         Self { datasets }
     }
+
+    pub fn geoip(&self) -> Option<&GeoIpSynDataset> {
+        self.datasets.get(&SiemDatasetType::GeoIp)?.try_into().ok()
+    }
+    pub fn i18n(&self) -> Option<&I18nSynDataset> {
+        self.datasets.get(&SiemDatasetType::I18n)?.try_into().ok()
+    }
+    pub fn ip_mac(&self) -> Option<&IpMapSynDataset> {
+        self.datasets.get(&SiemDatasetType::IpMac)?.try_into().ok()
+    }
+    pub fn ip_dns(&self) -> Option<&IpMapListSynDataset> {
+        self.datasets.get(&SiemDatasetType::IpDNS)?.try_into().ok()
+    }
+    pub fn mac_host(&self) -> Option<&TextMapSynDataset> {
+        self.datasets.get(&SiemDatasetType::MacHost)?.try_into().ok()
+    }
+    pub fn host_user(&self) -> Option<&TextMapSynDataset> {
+        self.datasets.get(&SiemDatasetType::HostUser)?.try_into().ok()
+    }
+    pub fn block_ip(&self) -> Option<&IpSetSynDataset> {
+        self.datasets.get(&SiemDatasetType::BlockIp)?.try_into().ok()
+    }
+    pub fn block_domain(&self) -> Option<&TextSetSynDataset> {
+        self.datasets.get(&SiemDatasetType::BlockDomain)?.try_into().ok()
+    }
+    pub fn block_email(&self) -> Option<&TextSetSynDataset> {
+        self.datasets.get(&SiemDatasetType::BlockEmailSender)?.try_into().ok()
+    }
+    pub fn block_country(&self) -> Option<&TextSetSynDataset> {
+        self.datasets.get(&SiemDatasetType::BlockCountry)?.try_into().ok()
+    }
+    pub fn host_vulnerable(&self) -> Option<&TextMapListSynDataset> {
+        self.datasets.get(&SiemDatasetType::HostVulnerable)?.try_into().ok()
+    }
+    pub fn user_tag(&self) -> Option<&TextMapListSynDataset> {
+        self.datasets.get(&SiemDatasetType::UserTag)?.try_into().ok()
+    }
+    pub fn asset_tag(&self) -> Option<&TextMapListSynDataset> {
+        self.datasets.get(&SiemDatasetType::AssetTag)?.try_into().ok()
+    }
+    pub fn cloud_service(&self) -> Option<&IpNetSynDataset> {
+        self.datasets.get(&SiemDatasetType::IpCloudService)?.try_into().ok()
+    }
+    pub fn cloud_provider(&self) -> Option<&IpNetSynDataset> {
+        self.datasets.get(&SiemDatasetType::IpCloudProvider)?.try_into().ok()
+    }
+    pub fn user_headquarters(&self) -> Option<&TextMapSynDataset> {
+        self.datasets.get(&SiemDatasetType::UserHeadquarters)?.try_into().ok()
+    }
+    pub fn ip_headquarters(&self) -> Option<&IpNetSynDataset> {
+        self.datasets.get(&SiemDatasetType::IpHeadquarters)?.try_into().ok()
+    }
+    pub fn configuration(&self) -> Option<&TextMapSynDataset> {
+        self.datasets.get(&SiemDatasetType::Configuration)?.try_into().ok()
+    }
 }

src/components/dataset/mod.rs

@@ -443,6 +443,29 @@ impl<'a> TryFrom<&'a SiemDataset> for &'a GeoIpSynDataset {
     }
 }
 
+impl TryFrom<SiemDataset> for I18nSynDataset {
+    type Error = &'static str;
+
+    fn try_from(value: SiemDataset) -> Result<Self, Self::Error> {
+        if let SiemDataset::I18n(v) = value {
+            Ok(v)
+        } else {
+            Err("I18nSynDataset is only valid for I18n dataset!")
+        }
+    }
+}
+impl<'a> TryFrom<&'a SiemDataset> for &'a I18nSynDataset {
+    type Error = &'static str;
+
+    fn try_from(value: &'a SiemDataset) -> Result<Self, Self::Error> {
+        if let SiemDataset::I18n(v) = value {
+            Ok(v)
+        } else {
+            Err("I18nSynDataset is only valid for I18n dataset!")
+        }
+    }
+}
+
 impl TryFrom<(SiemDatasetType, IpMapSynDataset)> for SiemDataset {
     type Error = &'static str;
 
@@ -929,4 +952,4 @@ pub enum UpdateDataset {
     HostVulnerable(UpdateTextMapList),
     CorrelationRules(UpdateGeoIp),
     I18n(UpdateI18n),
-}
+}

src/components/mod.rs

@@ -27,7 +27,7 @@ pub mod rule;
 pub mod storage;
 pub mod task;
 pub mod use_case;
-
+pub mod simplified;
 pub trait SiemComponent: Send {
     fn name(&self) -> &'static str {
         "SiemComponent"