Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

glog parser's "glog_timestamp" field can't be used as the event's timestamp #35

Closed
seh opened this issue Apr 16, 2019 · 2 comments
Closed

Comments

@seh
Copy link

seh commented Apr 16, 2019

For the parsers like "json" and "keyval" that confront timestamp fields as text, it's possible to use the "timefield" and "drop_field" processors to "move" those fields to the primary timestamp of the posted log event. That is, say that there's a log record in "keyval" format with a field named "ts," with its values as RFC 3339 strings. You can write the following to make the "ts" field's time the event's time:

- dataset: "My Dataset"
  namespace: kube-system
  labelSelector: "app = prometheus-node-exporter"
  parser: keyval
  processors:
  - timefield:
      field: time
      format: "2006-01-02T15:04:05Z07:00" # RFC 3339
  - drop_field:
      field: time

However, with the "glog" format, it tries to help by parsing the expected time string into a time.Time value, and stashing it in the "glog_timestamp" field. It's not possible to later "move" this "glog_timestamp" field into the event's timestamp, because the "timefield" processor only consumes fields with values of type string or int.

It would be nice if the "timefield" processor could treat the "format" configuration field as optional (maybe it already does), and if it's absent, tolerate extracting a field that's already of type time.Time. If it could do that, we could promote the "glog_timestamp" field like this:

- dataset: "My Dataset"
  namespace: kube-system
  labelSelector: "k8s-app notin (calico-node, calico-typha)"
  parser: glog
  processors:
  - timefield:
      field: glog_timestamp
  - drop_field:
      field: glog_timestamp
tredman added a commit to honeycombio/honeytail that referenced this issue Apr 17, 2019
Prereq for honeycombio/honeycomb-kubernetes-agent#35

We assume we're processing lines into string, int, or float values - but there are use cases where the field map passed to GetTimestamp can have an actual time.Time value. If requested explicitly in timeFieldName, we should just return the Time value we already have.
tredman added a commit that referenced this issue May 6, 2019
This should address #35 - allowing GetTimestamp to handle times with type `time.Time` by just returning the value directly without attempting to parse it.
tredman added a commit that referenced this issue May 6, 2019
This should address #35 - allowing GetTimestamp to handle times with type `time.Time` by just returning the value directly without attempting to parse it.
@tredman
Copy link
Contributor

tredman commented May 6, 2019

This is fixed in 1.3.0

@seh
Copy link
Author

seh commented May 7, 2019

I confirmed that this works as expected. Thank you.

@seh seh closed this as completed May 7, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants