Skip to content

Commit

Permalink
userguide: document flow_id, with examples
Browse files Browse the repository at this point in the history 
Flow_id explanation expanded from version shared by Peter Manev.

Task OISF#6445
  • Loading branch information
@jufajardini @victorjulien
jufajardini authored and victorjulien committed Nov 17, 2023
1 parent bec1d8c commit 58fb559
Show file tree
Hide file tree
Showing 2 changed files with 311 additions and 1 deletion.
310 changes: 309 additions & 1 deletion doc/userguide/output/eve/eve-json-format.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,307 @@ All the JSON log types share a common structure:
::


{"timestamp":"2009-11-24T21:27:09.534255","event_type":"TYPE", ...tuple... ,"TYPE":{ ... type specific content ... }}
{"timestamp":"2009-11-24T21:27:09.534255","flow_id":ID_NUMBER, "event_type":"TYPE", ...tuple... ,"TYPE":{ ... type specific content ... }}

Field: flow_id
~~~~~~~~~~~~~~

Correlates the network protocol, flow logs EVE data and any evidence that
Suricata has logged to an ``alert`` event and that alert's metadata, as well as
to ``fileinfo``/file transaction and anomaly logs, if available. The same correlation
and logs are produced regardless if there is an alert, for any session/flow.

The ability to correlate EVE logs belonging to a specific session/flow was
introduced in 2014 (see `commit f1185d051c21 <https://github.com/OISF/suricata/
commit/f1185d051c210ca0daacdddbe865a51af24f4ea3>`_).

Further below, you can see several examples of events logged by Suricata: an
:ref:`alert<eve-format-alert>` for an ``HTTP`` rule, ``fileinfo``, :ref:`http<eve-format-http>`,
:ref:`anomaly<eve-format-anomaly>`, and :ref:`flow<eve-format-flow>` events, all
easily correlated using the ``flow_id`` EVE field::

$ jq 'select(.flow_id==1676750115612680)' eve.json

Event type: ``alert``::

{
"timestamp": "2023-09-18T06:13:41.532140+0000",
"flow_id": 1676750115612680,
"pcap_cnt": 130,
"event_type": "alert",
"src_ip": "142.11.240.191",
"src_port": 35361,
"dest_ip": "192.168.100.237",
"dest_port": 49175,
"proto": "TCP",
"pkt_src": "wire/pcap",
"ether": {
"src_mac": "52:54:00:36:3e:ff",
"dest_mac": "12:a9:86:6c:77:de"
},
"tx_id": 1,
"alert": {
"action": "allowed",
"gid": 1,
"signature_id": 2045001,
"rev": 1,
"signature": "ET ATTACK_RESPONSE Win32/LeftHook Stealer Browser Extension Config Inbound",
"category": "A Network Trojan was detected",
"severity": 1,
"metadata": {
"affected_product": [
"Windows_XP_Vista_7_8_10_Server_32_64_Bit"
],
"attack_target": [
"Client_Endpoint"
],
"created_at": [
"2023_04_17"
],
"deployment": [
"Perimeter"
],
"former_category": [
"ATTACK_RESPONSE"
],
"signature_severity": [
"Major"
],
"updated_at": [
"2023_04_18"
]
}
},
"http": {
"hostname": "142.11.240.191",
"http_port": 35361,
"url": "/",
"http_content_type": "text/xml",
"http_method": "POST",
"protocol": "HTTP/1.1",
"status": 200,
"length": 5362
},
"files": [
{
"filename": "/",
"gaps": false,
"state": "CLOSED",
"stored": false,
"size": 5362,
"tx_id": 1
}
],
"app_proto": "http",
"direction": "to_client",
"flow": {
"pkts_toserver": 13,
"pkts_toclient": 12,
"bytes_toserver": 1616,
"bytes_toclient": 8044,
"start": "2023-09-18T06:13:33.324862+0000",
"src_ip": "192.168.100.237",
"dest_ip": "142.11.240.191",
"src_port": 49175,
"dest_port": 35361
}
}

Event type: ``fileinfo``::

{
"timestamp": "2023-09-18T06:13:33.903924+0000",
"flow_id": 1676750115612680,
"pcap_cnt": 70,
"event_type": "fileinfo",
"src_ip": "192.168.100.237",
"src_port": 49175,
"dest_ip": "142.11.240.191",
"dest_port": 35361,
"proto": "TCP",
"pkt_src": "wire/pcap",
"ether": {
"src_mac": "12:a9:86:6c:77:de",
"dest_mac": "52:54:00:36:3e:ff"
},
"http": {
"hostname": "142.11.240.191",
"http_port": 35361,
"url": "/",
"http_content_type": "text/xml",
"http_method": "POST",
"protocol": "HTTP/1.1",
"status": 200,
"length": 212
},
"app_proto": "http",
"fileinfo": {
"filename": "/",
"gaps": false,
"state": "CLOSED",
"stored": false,
"size": 137,
"tx_id": 0
}
}

Event type: ``HTTP``::

{
"timestamp": "2023-09-18T06:13:33.903924+0000",
"flow_id": 1676750115612680,
"pcap_cnt": 70,
"event_type": "http",
"src_ip": "192.168.100.237",
"src_port": 49175,
"dest_ip": "142.11.240.191",
"dest_port": 35361,
"proto": "TCP",
"pkt_src": "wire/pcap",
"ether": {
"src_mac": "12:a9:86:6c:77:de",
"dest_mac": "52:54:00:36:3e:ff"
},
"tx_id": 0,
"http": {
"hostname": "142.11.240.191",
"http_port": 35361,
"url": "/",
"http_content_type": "text/xml",
"http_method": "POST",
"protocol": "HTTP/1.1",
"status": 200,
"length": 212,
"request_headers": [
{
"name": "Content-Type",
"value": "text/xml; charset=utf-8"
},
{
"name": "SOAPAction",
"value": "\"http://tempuri.org/Endpoint/CheckConnect\""
},
{
"name": "Host",
"value": "142.11.240.191:35361"
},
{
"name": "Content-Length",
"value": "137"
},
{
"name": "Expect",
"value": "100-continue"
},
{
"name": "Accept-Encoding",
"value": "gzip, deflate"
},
{
"name": "Connection",
"value": "Keep-Alive"
}
],
"response_headers": [
{
"name": "Content-Length",
"value": "212"
},
{
"name": "Content-Type",
"value": "text/xml; charset=utf-8"
},
{
"name": "Server",
"value": "Microsoft-HTTPAPI/2.0"
},
{
"name": "Date",
"value": "Mon, 18 Sep 2023 06:13:33 GMT"
}
]
}
}

Event type: ``anomaly``::

{
"timestamp": "2023-09-18T06:13:58.882971+0000",
"flow_id": 1676750115612680,
"pcap_cnt": 2878,
"event_type": "anomaly",
"src_ip": "192.168.100.237",
"src_port": 49175,
"dest_ip": "142.11.240.191",
"dest_port": 35361,
"proto": "TCP",
"pkt_src": "wire/pcap",
"ether": {
"src_mac": "12:a9:86:6c:77:de",
"dest_mac": "52:54:00:36:3e:ff"
},
"tx_id": 3,
"anomaly": {
"app_proto": "http",
"type": "applayer",
"event": "UNABLE_TO_MATCH_RESPONSE_TO_REQUEST",
"layer": "proto_parser"
}
}


Event type: ``flow``::

{
"timestamp": "2023-09-18T06:13:21.216460+0000",
"flow_id": 1676750115612680,
"event_type": "flow",
"src_ip": "192.168.100.237",
"src_port": 49175,
"dest_ip": "142.11.240.191",
"dest_port": 35361,
"proto": "TCP",
"app_proto": "http",
"flow": {
"pkts_toserver": 3869,
"pkts_toclient": 1523,
"bytes_toserver": 3536402,
"bytes_toclient": 94102,
"start": "2023-09-18T06:13:33.324862+0000",
"end": "2023-09-18T06:14:13.752399+0000",
"age": 40,
"state": "closed",
"reason": "shutdown",
"alerted": true
},
"ether": {
"dest_macs": [
"52:54:00:36:3e:ff"
],
"src_macs": [
"12:a9:86:6c:77:de"
]
},
"tcp": {
"tcp_flags": "1e",
"tcp_flags_ts": "1e",
"tcp_flags_tc": "1a",
"syn": true,
"rst": true,
"psh": true,
"ack": true,
"state": "closed",
"ts_max_regions": 1,
"tc_max_regions": 1
}
}

.. note::
It is possible to have even more detailed alert records, by enabling for
instance logging http-body, or alert metadata (:ref:`alert output<eve-output-alert>`).

Examples come from pcap found at https://app.any.run/tasks/ce7ca983-9e4b-4251-a7c3-fefa3da02ebe/.


Event types
~~~~~~~~~~~
Expand Down Expand Up @@ -86,6 +386,8 @@ generated the event.
omitted from internal "pseudo" packets such as flow timeout
packets.

.. _eve-format-alert:

Event type: Alert
-----------------

Expand Down Expand Up @@ -191,6 +493,8 @@ If pcap log capture is active in `multi` mode, a `capture_file` key will be adde
with value being the full path of the pcap file where the corresponding packets
have been extracted.

.. _eve-format-anomaly:

Event type: Anomaly
-------------------

Expand Down Expand Up @@ -304,6 +608,8 @@ Examples
}
}

.. _eve-format-http:

Event type: HTTP
----------------

Expand Down Expand Up @@ -1345,6 +1651,8 @@ Example of SSH logging:
}
}

.. _eve-format-flow:

Event type: Flow
----------------

Expand Down
2 changes: 2 additions & 0 deletions doc/userguide/output/eve/eve-json-output.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,8 @@ Output types::
# enabled: yes ## set enable to yes to enable query pipelining
# batch-size: 10 ## number of entry to keep in buffer

.. _eve-output-alert:

Alerts
~~~~~~

Expand Down

0 comments on commit 58fb559

Please sign in to comment.