Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ingest node date processor uses stale year information on calendar year change #22547

Closed
pprkut opened this issue Jan 11, 2017 · 2 comments
Closed

Ingest node date processor uses stale year information on calendar year change #22547

pprkut opened this issue Jan 11, 2017 · 2 comments
Assignees
@talevy
Labels
>bug :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP

Comments

@pprkut
Copy link

pprkut commented Jan 11, 2017

Elasticsearch version:
5.1.1

JVM version:
1.8.0.111

OS version:
CentOS 7.3

Description of the problem including expected versus actual behavior:

I'm feeding /var/log/secure to elasticsearch, using filebeat and ingest node. This worked fine before the new year, but then I noticed entries appearing with a one year old timestamp.
Turns out, the date processor was parsing the date string "Jan 2 23:59:48" as "2016-01-02T23:59:48.000Z". Restarting elasticsearch caused new entries to get the correct year information.

More info available here.
Possibly related to logstash-plugins/logstash-filter-date#3
@pprkut pprkut changed the title Ingest node data processor uses stale year information on calendar year change Ingest node date processor uses stale year information on calendar year change Jan 11, 2017
@clintongormley clintongormley added :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP >bug labels Jan 11, 2017
@clintongormley
Copy link

@talevy Did you have a solution for this in Logstash that can be ported to ingest?

@talevy
Copy link
Contributor

talevy commented Jan 11, 2017

Oops, I wasn't aware of this pre-existing issue that we had in logstash-filter-date. I see the solution we came up with in Logstash, I'll see how we can do something similar in Elasticsearch

@jordansissel jordansissel mentioned this issue Jan 11, 2017
Closed
@talevy talevy mentioned this issue Jan 12, 2017
Merged
talevy added a commit to talevy/elasticsearch that referenced this issue Jan 23, 2017
@talevy
fix date-processor to a new default year for every new pipeline execu…
3d168d5 
…tion.

Beforehand, the DateProcessor constructs its joda pattern formatter during processor
construction. This led to newly ingested documents being defaulted to
the year that the pipeline was constructed, not that of processing.

Fixes elastic#22547.
talevy added a commit that referenced this issue Jan 25, 2017
@talevy
fix date-processor to a new default year for every new pipeline execu…
e9a68b3 
…tion. (#22601)

Beforehand, the DateProcessor constructs its joda pattern formatter during processor
construction. This led to newly ingested documents being defaulted to
the year that the pipeline was constructed, not that of processing.

Fixes #22547.
talevy added a commit that referenced this issue Jan 25, 2017
@talevy
fix date-processor to a new default year for every new pipeline execu…
148cd68 
…tion. (#22601)

Beforehand, the DateProcessor constructs its joda pattern formatter during processor
construction. This led to newly ingested documents being defaulted to
the year that the pipeline was constructed, not that of processing.

Fixes #22547.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@talevy talevy

Labels
>bug :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@clintongormley @pprkut @talevy