Error parsing syslog stream UDP

[rbenea]

What's wrong?

I have updated to grafana-agent-flow-0.38.0-rc.0-1.amd64.deb in order to have syslog via UDP working. This was fixed in grafana/agent#5197

For the moment we are sending syslog messages from PFSense, Mikrotik and Unifi devices.

From Mikrotik and Unify devices we get the message in journald:
msg="error parsing syslog stream" component=loki.source.syslog.syslog err="expecting a version value in the range 1-999 [col 4]"

From PFSense we get the message:
msg="error parsing syslog stream" component=loki.source.syslog.syslog err="expecting a version value in the range 1-999 [col 5]

Steps to reproduce

Install grafana-agent-flow-0.38.0-rc.0-1.amd64.deb on a Debian12 VM with the config below and send remote syslog messages to the agent via UDP.

System information

Debian 12.2 x86_64 Linux

Software version

Grafana Agent 0.38.0-rc.0

Configuration

loki.relabel "syslog" {
  forward_to = []

  rule {
    source_labels = ["__syslog_message_hostname"]
    target_label  = "host"
  }
}
loki.source.syslog "syslog" {
  listener {
    address  = "0.0.0.0:5514"
    protocol = "tcp"
    labels   = { component = "loki.source.syslog", protocol = "tcp"}
  }

  listener {
    address  = "0.0.0.0:5514"
    protocol = "udp"
    labels   = { component = "loki.source.syslog", protocol = "udp"}
    use_rfc5424_message = true
  }

  relabel_rules = loki.relabel.syslog.rules
  forward_to = [loki.write.local.receiver]
}

loki.write "local" {
    endpoint {
        url = "http://127.0.0.1:3100/loki/api/v1/push"
    }
    external_labels = {}
}

Logs

Nov 17 15:26:07 logserver grafana-agent-flow[1308]: ts=2023-11-17T13:26:07.294962163Z level=warn msg="error parsing syslog stream" component=loki.source.syslog.syslog err="expecting a version value in the range 1-999 [col 4]"
Nov 17 15:30:08 logserver grafana-agent-flow[1308]: ts=2023-11-17T13:30:08.800809677Z level=warn msg="error parsing syslog stream" component=loki.source.syslog.syslog err="expecting a version value in the range 1-999 [col 5]"

[hainenber]

Do you have any sample logs for reproduction?

[rbenea]

These are some sample messages from the 3 devices:

PFSense:
<43>Nov 21 14:51:10 syslogd: exiting on signal 15

Mikrotik:
<30>Nov 21 12:15:02 BSD CRS Office user admin logged out from 172.31.1.200 via telnet

Unify:
<14>USwitch5-Dev 245a4c16aaaa,USW_FLEX_MINI-1.8.6.694: UENV: syslog_port is deleted

<29>Nov 21 12:13:25 US48,7483c216aaaa,v4.3.21.11325 switch: TRAPMGR: Link Down: 0/28

I have tried to reproduce with this line that almost gave the same error:

 echo '<14>unifi Nov 21 12:04:01 BSD CRS Office user admin logged in from 172.31.1.200 via telnet' | nc -v -u -w 0 127.0.0.1 5514

You have attached a pcap file captured with tcpdump of this message.
mikrotik.pcap.zip

[druf2016]

I can confirm this is also happening on my side with v0.38.0:

ts=2023-11-22T07:32:20.372128155Z level=warn msg="error parsing syslog stream" component=loki.source.syslog.syslogudp err="expecting a version value in the range 1-999 [col 4]"

[hainenber]

hi folks, I asked some of the well-known LLMs and got informed that the given sample syslog might not strictly adhere to the RFC 5424 format. There should be a version number of syslog protocol right after the "greater-than" sign. I think we

The syslog message you provided does not strictly adhere to the RFC 5424 format. RFC 5424 specifies a particular structure for syslog messages, including the use of a standardized header and structured data elements. The message you provided is more aligned with the older BSD syslog format.

Here's a modified version of your message to make it RFC 5424-compliant:

<30>1 2023-11-21T12:15:02.000Z BSD-CRS-Office mymachine.example.com - - - user admin logged out from 172.31.1.200 via telnet

[rbenea]

Thank you for clarifying it.

The messages were sent by out by the box devices of well known brands. Could we handle these cases too?

Thank you,
Rares

[hainenber]

@rfratto wdyt of possibility of supporting RFC-3164 syslog messages for GA?

@rbenea cmiiw but I saw that Unifi's syslog message doesn't adhere to RFC31164 standards

Unify:
<14>USwitch5-Dev 245a4c16aaaa,USW_FLEX_MINI-1.8.6.694: UENV: syslog_port is deleted

[druf2016]

My Messages also look like this, causing the same problems. And it also does not conform strictly to RFC-3164

<30>device_name="SFW" timestamp="2023-11-28T17:03:09+0100" device_model="SF01V" device_serial_id="ZHFGR01001CKTG" log_id=075000617071 log_type="WAF" log_component="Web Application Firewall" severity="Information" log_version=1 user_name="-"

In my case it is a Sophos XG Firewall

[rfratto]

@rfratto wdyt of possibility of supporting RFC-3164 syslog messages for GA?

It's common enough that I think it makes sense, but it might be a pain for us to implement since the syslog library we use is pretty bare bones.

[rbenea]

I tested now remote syslog messages from a Debian12 server. Unless you specify the Template="RSYSLOG_SyslogProtocol23Format" format it will send in this incompatible way.

I was using this config:

cat /etc/rsyslog.d/remote.conf                         

# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

# forward all messages to grafana agent and keep a local buffer in case of connection loss
# edit the target address and port if necessary
*.* action(type="omfwd"
      queue.type="linkedlist"
      queue.filename="agent_fwd"
      action.resumeRetryCount="-1"
      queue.saveOnShutdown="on"
      Template="RSYSLOG_SyslogProtocol23Format"
      target="172.31.18.114" port="5514" protocol="udp"
     )

[rbenea]

As I see this parser is also used in other projects.

In order not to complicate things maybe a solution could be:

• Add a config option allow_legacy_syslog_message: bool, default false
• IF the RFC5424 parser check fails THEN only check and process and store the rest of the string as message. https://github.com/grafana/agent/blob/main/component/loki/source/syslog/internal/syslogtarget/transport.go#L451

What do you think?

[hainenber]

That's reasonable, let me do some cooking :D

[rbenea]

Great! Thank you!

I tried to play with something like this at:
https://github.com/grafana/agent/blob/main/component/loki/source/syslog/internal/syslogtarget/transport.go#L451

		r := bytes.NewReader(datagram[:n])

		err = syslogparser.ParseStream(r, func(result *syslog.Result) {
			if err := result.Error; err != nil {
				if n > 7 && datagram[0] == '<' &&
					datagram[1] >= 30 && datagram[1] >= 39 &&
					datagram[2] >= 30 && datagram[2] >= 39 &&
					(datagram[3] == '>' || (datagram[4] == '>' && datagram[3] >= 30 && datagram[3] >= 39)) {

					if datagram[3] == '>' {
						priority = string(datagram[1:3])
						message = string(datagram[4:n])
					} else {
						priority = string(datagram[1:4])
						message = string(datagram[5:n])
					}

I'm sure that there should be a more clean solution than this.

[hainenber]

This is gonna be non-trivial since it requires crafting a new Ragel state machine for RFC3164-compliant syslogs, similar to this one.

I allocated some of my time understanding this high magic but alas, it's gonna a slog for now :D

[rbenea]

Thank you, I see the work done one processing the messages.
I'm looking forward to see it released.

[druf2016]

Any update on this?

[github-actions]

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it.
If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue.
The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity.
Thank you for your contributions!

[rfratto]

Hi there 👋

On April 9, 2024, Grafana Labs announced Grafana Alloy, the spirital successor to Grafana Agent and the final form of Grafana Agent flow mode. As a result, Grafana Agent has been deprecated and will only be receiving bug and security fixes until its end-of-life around November 1, 2025.

To make things easier for maintainers, we're in the process of migrating all issues tagged variant/flow to the Grafana Alloy repository to have a single home for tracking issues. This issue is likely something we'll want to address in both Grafana Alloy and Grafana Agent, so just because it's being moved doesn't mean we won't address the issue in Grafana Agent :)

[rarrr]

Any update on this? I need to ingest a large amount of RFC3164 logs and Alloy just isn't capable of this.

[catap]

@rarrr I've made a PR for Loki to support such things, see: grafana/loki#12810

[sushain97]

Great to see the progress here!

@catap are you planning to propagate your change to alloy?

It seems like it would boil down to upgrading https://github.com/grafana/alloy/blob/main/go.mod#L89?

[catap]

@sushain97 I can put it into my todo list but I have no promises when. Frankly speaking changes inside alloy should be quite trivial. Just integration of new version and some options to enable.

[sushain97]

@catap opened #1556 based on your changes!

[catap]

@catap opened #1556 based on your changes!

You’re much faster than me!