Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added hep protocol support as parser in telegraf #10039

Closed
wants to merge 2 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -1368,6 +1368,10 @@ func (c *Config) getParserConfig(name string, tbl *ast.Table) (*parsers.Config,
c.getFieldBool(tbl, "csv_trim_space", &pc.CSVTrimSpace)
c.getFieldStringSlice(tbl, "csv_skip_values", &pc.CSVSkipValues)

//for hep parser
c.getFieldStringSlice(tbl, "hep_header", &pc.HEPHeader)
c.getFieldString(tbl, "hep_measurement_name", &pc.HepMeasurementName)

c.getFieldStringSlice(tbl, "form_urlencoded_tag_keys", &pc.FormUrlencodedTagKeys)

c.getFieldString(tbl, "value_field_name", &pc.ValueFieldName)
Expand Down Expand Up @@ -1598,7 +1602,7 @@ func (c *Config) missingTomlField(_ reflect.Type, key string) error {
"tagdrop", "tagexclude", "taginclude", "tagpass", "tags", "template", "templates",
"value_field_name", "wavefront_source_override", "wavefront_use_strict",
"xml", "xpath", "xpath_json", "xpath_msgpack", "xpath_protobuf", "xpath_print_document",
"xpath_protobuf_file", "xpath_protobuf_type":
"xpath_protobuf_file", "xpath_protobuf_type", "hep_header", "hep_measurement_name":

// ignore fields that are common to all plugins.
default:
Expand Down
19 changes: 19 additions & 0 deletions plugins/parsers/hep/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# HEP
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like it's going through a name change to EEP? If we do add a parser, shouldn't it use the new name?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Due to the large number of integrations using HEP, the name change did not go through as of yet and won't be for the foreseeable future to avoid confusion on an already obscure protocol.


The HEP data format parses a HEP packet into metric fields.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not familiar with HEP and I suspect many telegraf users aren't either. Could you add a link to the project here in the docs? https://github.com/sipcapture/HEP

It might be worthwhile to provide a more comprehensive example of how this parser is meant to be used

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parser provides compatibility with the HEP encapsulation protocol which is almost universally supported in Open-Source VoIP platforms such as Asterisk, Freeswitch, Kamailio, OpenSIPS and many more, alongside major vendors like Genesys, Sansay, and other. This parser is dedicated to provide a layer of compatibility for those platforms to form and send metrics to Telegraf without implementing new patches/protocols.


**NOTE:** All HEP packets are stores as Tags unless provided specifically
provided with `hep_header` array and body is parsed with JSON parser.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does HEP only ever embed JSON formatted data?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@reimda as far as I read into this, newer versions of the protocol can embedd a JSON payload, so we then need an embedded JSON parsing... :-(

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@reimda HEP is a generic encapsulation protocol and per-se can carry any payload, including binary. The focus is on JSON in this specific usecase as all the integrating platforms are capable of producing and consuming it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm if this is the case, why not export the payload in a field payload together with a payload_type and then use a parser processor to parse these? This way you can stay generic in this parser...

All the JSON parser features were imported in Hep parser. Please check Json Parser for more details.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what you mean by "all json parser features were imported". Telegraf has two JSON parsers, "json" and "json_v2". This PR uses the older one which doesn't work well for some common json object structures. Should you switch to v2?


Any field/header can be ignored using already telegraf filters.

### Configuration

```toml
[[inputs.socket_listener]]
service_address = "udp://:8094"
data_format = "hep"
hep_header = ["protocol"]

```
152 changes: 152 additions & 0 deletions plugins/parsers/hep/hep.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,152 @@
package hep

import (
"encoding/binary"
"fmt"
"net"
"strconv"
"time"
)

// HEP chuncks
const (
Version = 1 // Chunk 0x0001 IP protocol family (0x02=IPv4, 0x0a=IPv6)
Protocol = 2 // Chunk 0x0002 IP protocol ID (0x06=TCP, 0x11=UDP)
IP4SrcIP = 3 // Chunk 0x0003 IPv4 source address
IP4DstIP = 4 // Chunk 0x0004 IPv4 destination address
IP6SrcIP = 5 // Chunk 0x0005 IPv6 source address
IP6DstIP = 6 // Chunk 0x0006 IPv6 destination address
SrcPort = 7 // Chunk 0x0007 Protocol source port
DstPort = 8 // Chunk 0x0008 Protocol destination port
Tsec = 9 // Chunk 0x0009 Unix timestamp, seconds
Tmsec = 10 // Chunk 0x000a Unix timestamp, microseconds
ProtoType = 11 // Chunk 0x000b Protocol type (DNS, LOG, RTCP, SIP)
NodeID = 12 // Chunk 0x000c Capture client ID
NodePW = 14 // Chunk 0x000e Authentication key (plain text / TLS connection)
Payload = 15 // Chunk 0x000f Captured packet payload
CID = 17 // Chunk 0x0011 Correlation ID
Vlan = 18 // Chunk 0x0012 VLAN
NodeName = 19 // Chunk 0x0013 NodeName
)

// HEP represents HEP packet
type HEP struct {
Version uint32
Protocol uint32
SrcIP string
DstIP string
SrcPort uint32
DstPort uint32
Tsec uint32
Tmsec uint32
ProtoType uint32
NodeID uint32
NodePW string
Payload string
CID string
Vlan uint32
ProtoString string
Timestamp time.Time
NodeName string
}

func (h *HEP) parseHEP(packet []byte) error {
length := binary.BigEndian.Uint16(packet[4:6])
if int(length) != len(packet) {
return fmt.Errorf("HEP packet length is %d but should be %d", len(packet), length)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please leave out the "HEP" prefix in the error message.

}
currentByte := uint16(6)

for currentByte < length {
hepChunk := packet[currentByte:]
if len(hepChunk) < 6 {
return fmt.Errorf("HEP chunk must be >= 6 byte long but is %d", len(hepChunk))
}
//chunkVendorId := binary.BigEndian.Uint16(hepChunk[:2])
chunkType := binary.BigEndian.Uint16(hepChunk[2:4])
chunkLength := binary.BigEndian.Uint16(hepChunk[4:6])
if len(hepChunk) < int(chunkLength) || int(chunkLength) < 6 {
return fmt.Errorf("HEP chunk with %d byte < chunkLength %d or chunkLength < 6", len(hepChunk), chunkLength)
}
chunkBody := hepChunk[6:chunkLength]

switch chunkType {
case Version, Protocol, ProtoType:
if len(chunkBody) != 1 {
return fmt.Errorf("HEP chunkType %d should be 1 byte long but is %d", chunkType, len(chunkBody))
}
case SrcPort, DstPort, Vlan:
if len(chunkBody) != 2 {
return fmt.Errorf("HEP chunkType %d should be 2 byte long but is %d", chunkType, len(chunkBody))
}
case IP4SrcIP, IP4DstIP, Tsec, Tmsec, NodeID:
if len(chunkBody) != 4 {
return fmt.Errorf("HEP chunkType %d should be 4 byte long but is %d", chunkType, len(chunkBody))
}
case IP6SrcIP, IP6DstIP:
if len(chunkBody) != 16 {
return fmt.Errorf("HEP chunkType %d should be 16 byte long but is %d", chunkType, len(chunkBody))
}
}

switch chunkType {
case Version:
h.Version = uint32(chunkBody[0])
case Protocol:
h.Protocol = uint32(chunkBody[0])
case IP4SrcIP:
h.SrcIP = net.IP(chunkBody).To4().String()
case IP4DstIP:
h.DstIP = net.IP(chunkBody).To4().String()
case IP6SrcIP:
h.SrcIP = net.IP(chunkBody).To16().String()
case IP6DstIP:
h.DstIP = net.IP(chunkBody).To16().String()
case SrcPort:
h.SrcPort = uint32(binary.BigEndian.Uint16(chunkBody))
case DstPort:
h.DstPort = uint32(binary.BigEndian.Uint16(chunkBody))
case Tsec:
h.Tsec = binary.BigEndian.Uint32(chunkBody)
case Tmsec:
h.Tmsec = binary.BigEndian.Uint32(chunkBody)
case ProtoType:
h.ProtoType = uint32(chunkBody[0])
switch h.ProtoType {
case 1:
h.ProtoString = "sip"
case 5:
h.ProtoString = "rtcp"
case 34:
h.ProtoString = "rtpagent"
case 35:
h.ProtoString = "rtcpxr"
case 38:
h.ProtoString = "horaclifix"
case 53:
h.ProtoString = "dns"
case 100:
h.ProtoString = "log"
case 112:
h.ProtoString = "alert"
Comment on lines +116 to +131
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please define those magic numbers similarly to the chunk-type etc?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Protocol strings are the equivalent of a type tag and are injected by HEP senders to further identify the payload type.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't doubt any of your words, but it won't hurt to define those numbers as consts and use them as speaking names. You can the even implement a type with the Stringer interface to cover this conversion here.

default:
h.ProtoString = strconv.Itoa(int(h.ProtoType))
}
case NodeID:
h.NodeID = binary.BigEndian.Uint32(chunkBody)
case NodePW:
h.NodePW = string(chunkBody)
case Payload:
h.Payload = string(chunkBody)
case CID:
h.CID = string(chunkBody)
case Vlan:
h.Vlan = uint32(binary.BigEndian.Uint16(chunkBody))
case NodeName:
h.NodeName = string(chunkBody)
default:
}
currentByte += chunkLength
}
return nil
}
Loading