next/508/20240628/v1 #11394

Add support for 'by_flow' track option. This allows using the various threshold options in the context of a single flow. Example: alert tcp ... stream-event:pkt_broken_ack; \ threshold:type limit, track by_flow, count 1, seconds 3600; The example would limit the number of alerts to once per hour for packets triggering the 'pkt_broken_ack' stream event. Implemented as a special "flowvar" holding the threshold entries. This means no synchronization is required, making this a cheaper option compared to the other trackers. Ticket: OISF#6822.

Allow rate_filter and thresholds from the global config to specify tracking "by_flow".

Traffic variables (flowvars, flowbits, xbits, etc) use a smaller int for their type than detection types. As a workaround make sure the values fit in a uint8_t.

Limits propegation checked for DETECT_DEPTH as a content flag, which appears to have worked by chance. After reshuffling the keyword id's it no longer worked. This patch uses the proper flag DETECT_CONTENT_DEPTH.

Thresholding often has 2 stages: 1. recording matches 2. appling an action, like suppress E.g. with something like: threshold:type limit, count 10, seconds 3600, track by_src; the recording state is about counting 10 first hits for an IP, then followed by the "suppress" state that might last an hour. By_src/by_dst are expensive, as they do a host table lookup and lock the host. If many threads require this access, lock contention becomes a serious problem. This patch adds a thread local cache to avoid the synchronization overhead. When the threshold for a host enters the "apply" stage, a thread local hash entry is added. This entry knows the expiry time and the action to apply. This way the action can be applied w/o the synchronization overhead. A rbtree is used to handle expiration. Implemented for IPv4.

Packet pointer is not used during allocation.

Add a callback and helper function to handle data expiration. Update datasets to explicitly not use expiration.

Instead of a Host and IPPair table thresholding layer, use a dedicated THash to store both. This allows hashing on host+sid+tracker or ippair+sid+tracker, to create more unique hash keys. This allows for fewer hash collisions. The per rule tracking also uses this, so that the single big lock is no longer a single point of contention. Reimplement storage for flow thresholds to reuse as much logic as possible from the host/ippair/rule thresholds. Ticket: OISF#426.

Ticket: OISF#6967.

Use the same hash key as for the regular threshold storage, so include gid, rev, tentant id.

Implement new `type backoff` for thresholding. This allows alerts to be limited. A count of 1 with a multiplier of 10 would generate alerts for matching packets: 1, 10, 100, 1000, 10000, 100000, etc. A count of 1 with a multiplier of 2 would generate alerts for matching packets: 1, 2, 4, 8, 16, 32, etc. Like with other thresholds, rule actions like drop and setting of flowbits will still be performed for each matching packet. Current implementation is only for the by_flow tracker and for per rule threshold statements. Tracking is done using uint32_t. When it reaches this value, the rest of the packets in the tracker will use the silent match. Ticket: OISF#7120.

Enable backoff for most rules. The rules looking at the session start up use a count of 1 and a multiplier of 2. Post-3whs rules use a count of 1 and a multiplier of 10.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

next/508/20240628/v1 #11394

next/508/20240628/v1 #11394

Commits on Jun 28, 2024