-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added hep protocol support as parser in telegraf #10039
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
# HEP | ||
|
||
The HEP data format parses a HEP packet into metric fields. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not familiar with HEP and I suspect many telegraf users aren't either. Could you add a link to the project here in the docs? https://github.com/sipcapture/HEP It might be worthwhile to provide a more comprehensive example of how this parser is meant to be used There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The parser provides compatibility with the HEP encapsulation protocol which is almost universally supported in Open-Source VoIP platforms such as Asterisk, Freeswitch, Kamailio, OpenSIPS and many more, alongside major vendors like Genesys, Sansay, and other. This parser is dedicated to provide a layer of compatibility for those platforms to form and send metrics to Telegraf without implementing new patches/protocols. |
||
|
||
**NOTE:** All HEP packets are stores as Tags unless provided specifically | ||
provided with `hep_header` array and body is parsed with JSON parser. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Does HEP only ever embed JSON formatted data? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @reimda as far as I read into this, newer versions of the protocol can embedd a JSON payload, so we then need an embedded JSON parsing... :-( There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @reimda HEP is a generic encapsulation protocol and per-se can carry any payload, including binary. The focus is on JSON in this specific usecase as all the integrating platforms are capable of producing and consuming it. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hmmm if this is the case, why not export the payload in a field |
||
All the JSON parser features were imported in Hep parser. Please check Json Parser for more details. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not sure what you mean by "all json parser features were imported". Telegraf has two JSON parsers, "json" and "json_v2". This PR uses the older one which doesn't work well for some common json object structures. Should you switch to v2? |
||
|
||
Any field/header can be ignored using already telegraf filters. | ||
|
||
### Configuration | ||
|
||
```toml | ||
[[inputs.socket_listener]] | ||
service_address = "udp://:8094" | ||
data_format = "hep" | ||
hep_header = ["protocol"] | ||
|
||
``` |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,152 @@ | ||
package hep | ||
|
||
import ( | ||
"encoding/binary" | ||
"fmt" | ||
"net" | ||
"strconv" | ||
"time" | ||
) | ||
|
||
// HEP chuncks | ||
const ( | ||
Version = 1 // Chunk 0x0001 IP protocol family (0x02=IPv4, 0x0a=IPv6) | ||
Protocol = 2 // Chunk 0x0002 IP protocol ID (0x06=TCP, 0x11=UDP) | ||
IP4SrcIP = 3 // Chunk 0x0003 IPv4 source address | ||
IP4DstIP = 4 // Chunk 0x0004 IPv4 destination address | ||
IP6SrcIP = 5 // Chunk 0x0005 IPv6 source address | ||
IP6DstIP = 6 // Chunk 0x0006 IPv6 destination address | ||
SrcPort = 7 // Chunk 0x0007 Protocol source port | ||
DstPort = 8 // Chunk 0x0008 Protocol destination port | ||
Tsec = 9 // Chunk 0x0009 Unix timestamp, seconds | ||
Tmsec = 10 // Chunk 0x000a Unix timestamp, microseconds | ||
ProtoType = 11 // Chunk 0x000b Protocol type (DNS, LOG, RTCP, SIP) | ||
NodeID = 12 // Chunk 0x000c Capture client ID | ||
NodePW = 14 // Chunk 0x000e Authentication key (plain text / TLS connection) | ||
Payload = 15 // Chunk 0x000f Captured packet payload | ||
CID = 17 // Chunk 0x0011 Correlation ID | ||
Vlan = 18 // Chunk 0x0012 VLAN | ||
NodeName = 19 // Chunk 0x0013 NodeName | ||
) | ||
|
||
// HEP represents HEP packet | ||
type HEP struct { | ||
Version uint32 | ||
Protocol uint32 | ||
SrcIP string | ||
DstIP string | ||
SrcPort uint32 | ||
DstPort uint32 | ||
Tsec uint32 | ||
Tmsec uint32 | ||
ProtoType uint32 | ||
NodeID uint32 | ||
NodePW string | ||
Payload string | ||
CID string | ||
Vlan uint32 | ||
ProtoString string | ||
Timestamp time.Time | ||
NodeName string | ||
} | ||
|
||
func (h *HEP) parseHEP(packet []byte) error { | ||
length := binary.BigEndian.Uint16(packet[4:6]) | ||
if int(length) != len(packet) { | ||
return fmt.Errorf("HEP packet length is %d but should be %d", len(packet), length) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please leave out the |
||
} | ||
currentByte := uint16(6) | ||
|
||
for currentByte < length { | ||
hepChunk := packet[currentByte:] | ||
if len(hepChunk) < 6 { | ||
return fmt.Errorf("HEP chunk must be >= 6 byte long but is %d", len(hepChunk)) | ||
} | ||
//chunkVendorId := binary.BigEndian.Uint16(hepChunk[:2]) | ||
chunkType := binary.BigEndian.Uint16(hepChunk[2:4]) | ||
chunkLength := binary.BigEndian.Uint16(hepChunk[4:6]) | ||
if len(hepChunk) < int(chunkLength) || int(chunkLength) < 6 { | ||
return fmt.Errorf("HEP chunk with %d byte < chunkLength %d or chunkLength < 6", len(hepChunk), chunkLength) | ||
} | ||
chunkBody := hepChunk[6:chunkLength] | ||
|
||
switch chunkType { | ||
case Version, Protocol, ProtoType: | ||
if len(chunkBody) != 1 { | ||
return fmt.Errorf("HEP chunkType %d should be 1 byte long but is %d", chunkType, len(chunkBody)) | ||
} | ||
case SrcPort, DstPort, Vlan: | ||
if len(chunkBody) != 2 { | ||
return fmt.Errorf("HEP chunkType %d should be 2 byte long but is %d", chunkType, len(chunkBody)) | ||
} | ||
case IP4SrcIP, IP4DstIP, Tsec, Tmsec, NodeID: | ||
if len(chunkBody) != 4 { | ||
return fmt.Errorf("HEP chunkType %d should be 4 byte long but is %d", chunkType, len(chunkBody)) | ||
} | ||
case IP6SrcIP, IP6DstIP: | ||
if len(chunkBody) != 16 { | ||
return fmt.Errorf("HEP chunkType %d should be 16 byte long but is %d", chunkType, len(chunkBody)) | ||
} | ||
} | ||
|
||
switch chunkType { | ||
case Version: | ||
h.Version = uint32(chunkBody[0]) | ||
case Protocol: | ||
h.Protocol = uint32(chunkBody[0]) | ||
case IP4SrcIP: | ||
h.SrcIP = net.IP(chunkBody).To4().String() | ||
case IP4DstIP: | ||
h.DstIP = net.IP(chunkBody).To4().String() | ||
case IP6SrcIP: | ||
h.SrcIP = net.IP(chunkBody).To16().String() | ||
case IP6DstIP: | ||
h.DstIP = net.IP(chunkBody).To16().String() | ||
case SrcPort: | ||
h.SrcPort = uint32(binary.BigEndian.Uint16(chunkBody)) | ||
case DstPort: | ||
h.DstPort = uint32(binary.BigEndian.Uint16(chunkBody)) | ||
case Tsec: | ||
h.Tsec = binary.BigEndian.Uint32(chunkBody) | ||
case Tmsec: | ||
h.Tmsec = binary.BigEndian.Uint32(chunkBody) | ||
case ProtoType: | ||
h.ProtoType = uint32(chunkBody[0]) | ||
switch h.ProtoType { | ||
case 1: | ||
h.ProtoString = "sip" | ||
case 5: | ||
h.ProtoString = "rtcp" | ||
case 34: | ||
h.ProtoString = "rtpagent" | ||
case 35: | ||
h.ProtoString = "rtcpxr" | ||
case 38: | ||
h.ProtoString = "horaclifix" | ||
case 53: | ||
h.ProtoString = "dns" | ||
case 100: | ||
h.ProtoString = "log" | ||
case 112: | ||
h.ProtoString = "alert" | ||
Comment on lines
+116
to
+131
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you please define those magic numbers similarly to the chunk-type etc? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Protocol strings are the equivalent of a There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't doubt any of your words, but it won't hurt to define those numbers as consts and use them as speaking names. You can the even implement a type with the |
||
default: | ||
h.ProtoString = strconv.Itoa(int(h.ProtoType)) | ||
} | ||
case NodeID: | ||
h.NodeID = binary.BigEndian.Uint32(chunkBody) | ||
case NodePW: | ||
h.NodePW = string(chunkBody) | ||
case Payload: | ||
h.Payload = string(chunkBody) | ||
case CID: | ||
h.CID = string(chunkBody) | ||
case Vlan: | ||
h.Vlan = uint32(binary.BigEndian.Uint16(chunkBody)) | ||
case NodeName: | ||
h.NodeName = string(chunkBody) | ||
default: | ||
} | ||
currentByte += chunkLength | ||
} | ||
return nil | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like it's going through a name change to EEP? If we do add a parser, shouldn't it use the new name?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Due to the large number of integrations using HEP, the name change did not go through as of yet and won't be for the foreseeable future to avoid confusion on an already obscure protocol.