Unable to Parse Timestamp in OTEL Collector Filelog Receiver

[vimalraj3]

Description

I am experiencing an issue with the OTEL Collector configuration where the filelog receiver is unable to parse timestamps from the log files. Below are the relevant configurations and the error message encountered:

OpenTelemetry Collector configuration

receivers:
  filelog:
    include: [ /var/logs/*.log ]
    operators:
      - type: regex_parser
        regex: '^(?P<timestamp>\d{4}-\d{2}-\d{2}-\d{2}-\d{2}-\d{2}:\d{3}) \[(?P<thread>[^\]]+)\] (?P<sev>[A-Z]*) (?P<msg>.*)$'
        timestamp:
          parse_from: attributes.timestamp
          layout_typ: strptime
          layout: '%Y-%m-%d-%H-%M-%S:%L'
        severity:
          parse_from: attributes.sev

exporters:
  debug:
    verbosity: detailed 

service:
  pipelines:
    logs:
      receivers: [filelog] 
      exporters: [debug]

Input Logs

2024-07-31-11-11-20:295 [http-nio-8300-exec-5] DEBUG com.txnws_watchlist.TxnWsWatchlist.eventing.RabbitPublisher:46 - Exit in rabbitMqEventPublisher
2024-07-31-11-11-20:295 [http-nio-8300-exec-5] DEBUG com.txnws_watchlist.TxnWsWatchlist.eventing.RabbitPublisher:46 - Exit in rabbitMqEventPublisher
2024-07-31-11-11-20:295 [http-nio-8300-exec-5] DEBUG com.txnws_watchlist.TxnWsWatchlist.eventing.RabbitPublisher:46 - Exit in rabbitMqEventPublisher
2024-07-31-11-11-20:295 [http-nio-8300-exec-5] DEBUG com.txnws_watchlist.TxnWsWatchlist.eventing.RabbitPublisher:46 - Exit in rabbitMqEventPublisher

Steps to Reproduce:

1. Configure the OTEL Collector with the above configuration.
2. Include log files with timestamps in the format YYYY-MM-DD-HH-MM-SS:SSS.
3. Run the collector and observe the error in the logs.

Expected Result

The timestamp in the logs should be parsed correctly using the specified regex pattern and layout format.

Actual Result

2024-08-02T11:56:06.423Z    error   helper/transformer.go:102       Failed to process entry {"kind": "receiver", "name": "filelog","data_type": "logs", "operator_id": "regex_parser", "operator_type": "regex_parser", "error": {"description": "time parser: parsing time \"2024-07-31-11-11-20:295\" as \"2006-01-02-15-04-05::999\": cannot parse \"295\" as \"::999\""}, "action": "send"}  

Collector version

v0.104.0

Environment information

OS: Amazon Linux

[github-actions]

Pinging code owners for receiver/filelog: @djaglowski. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[bacherfl]

Hi, I was looking into this out of curiosity and found that golang's time.Parse function seems to require a format where the milliseconds part of the timestamp is prefixed with either . or , - so apparently it is not possible to have a timestamp format where the milliseconds are prefixed with :, as indicated in this section of the time package: https://github.com/golang/go/blob/f428c7b729d3d9b37ed4dacddcd7ff88f4213f70/src/time/format.go#L316

Would it be an option to change the format of the time stamps in the logs to something like YYYY-MM-DD-HH-MM-SS.SSS?

[vimalraj3]

Thank you.

[crobert-1]

Hello @vimal-oneway, just want to check, have you been able to test to see if @bacherfl's suggestion works for you? If so, can we close this issue?

[vimalraj3]

Hi @crobert-1 ,

By using OTTL functions, I was able to concatenate the time and milliseconds with a period, which allowed me to parse it into a timestamp. Here’s the code snippet I used:

 - set(attributes["timestamp_ms"], Concat([attributes["timestamp"], attributes["ms"]], "."))

This resolved the problem I was experiencing. Thank you for your support!
Yes we can close this issue.

[ChrsMark]

Based on #34390 (comment), should the layout: '%Y-%m-%d-%H-%M-%S:%L' be validated at first place on receiver's startup to explicitly indicate that it's not a valid layout?

[bacherfl]

Based on #34390 (comment), should the layout: '%Y-%m-%d-%H-%M-%S:%L' be validated at first place on receiver's startup to explicitly indicate that it's not a valid layout?

I can look into adding such a check

[djaglowski]

Closing this based on the behavior being native to golang's time package, and because we've now added validation to detect it in #34459.