Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick #17428 to 7.7: Fix redundant operator in activemq ingest pipeline #17442

Merged
merged 2 commits into from
Apr 6, 2020

Conversation

adriansr
Copy link
Contributor

@adriansr adriansr commented Apr 2, 2020

Cherry-pick of PR #17428 to 7.7 branch. Original message:

Grok processor has a redundant * operator in a regular expression, which causes the following warning to be printed to the Elasticsearch logs every time the pipeline is loaded:

regular expression has redundant nested repeat operator * /(?<TIMESTAMP_ISO8601:timestamp>(?:(?>\d\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[T ](?:(?:2[0123]|[01][0-9])):?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|[+-](?:(?:2[0123]|[01]?[0-9]))(?::?(?:(?:[0-5][0-9])))))?)(?:\s*)\|(?:\s*)(?<LOGLEVEL:log.level>([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?))(?:\s*)\|(?:\s*)(?<NOPIPEGREEDYDATA:message>((?! \|).)*)(?:\s*)\|(?:\s*)(?<NOPIPEGREEDYDATA:activemq.caller>((?! \|).)*)(?:\s*)\|(?:\s*)(?<THREAD_NAME:activemq.thread>((?!
).)*)(?:\s*)?(?<GREEDYMULTILINE:activemq.log.stack_trace>(.|\n|\t)*)/

In this case the redundancy is caused by following a {SPACE} expression followed by a match-zero-or-one operator: ?. The SPACE pattern expands to (?:\s*) which already matches zero or more spaces. Perhaps a better name would have been SPACES.

I'm still seeing this error after upgrading Filebeat

if you're still seeing this error after upgrading Filebeat, note that the error is still printed when an Elasticsearch node is started if pipelines for older versions are installed (filebeat-X.Y.Z-elasticsearch-gc-pipeline for X.Y.Z < 7.7.0). A newer version won't remove existing pipelines.

Relates #15900 #17156

Grok processor has a redundant `*` operator in a regular expression,
which causes the following warning to be printed to the Elasticsearch
logs every time the pipeline is loaded:

```
regular expression has redundant nested repeat operator * /(?<TIMESTAMP_ISO8601:timestamp>(?:(?>\d\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[T ](?:(?:2[0123]|[01][0-9])):?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|[+-](?:(?:2[0123]|[01]?[0-9]))(?::?(?:(?:[0-5][0-9])))))?)(?:\s*)\|(?:\s*)(?<LOGLEVEL:log.level>([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?))(?:\s*)\|(?:\s*)(?<NOPIPEGREEDYDATA:message>((?! \|).)*)(?:\s*)\|(?:\s*)(?<NOPIPEGREEDYDATA:activemq.caller>((?! \|).)*)(?:\s*)\|(?:\s*)(?<THREAD_NAME:activemq.thread>((?!
).)*)(?:\s*)?(?<GREEDYMULTILINE:activemq.log.stack_trace>(.|\n|\t)*)/
```

(cherry picked from commit 543a5f4)
@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

Copy link
Contributor

@leehinman leehinman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@adriansr adriansr merged commit c8cfd68 into elastic:7.7 Apr 6, 2020
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
…lastic#17442)

Grok processor has a redundant `*` operator in a regular expression,
which causes the following warning to be printed to the Elasticsearch
logs every time the pipeline is loaded:

```
regular expression has redundant nested repeat operator * /(?<TIMESTAMP_ISO8601:timestamp>(?:(?>\d\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[T ](?:(?:2[0123]|[01][0-9])):?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|[+-](?:(?:2[0123]|[01]?[0-9]))(?::?(?:(?:[0-5][0-9])))))?)(?:\s*)\|(?:\s*)(?<LOGLEVEL:log.level>([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?))(?:\s*)\|(?:\s*)(?<NOPIPEGREEDYDATA:message>((?! \|).)*)(?:\s*)\|(?:\s*)(?<NOPIPEGREEDYDATA:activemq.caller>((?! \|).)*)(?:\s*)\|(?:\s*)(?<THREAD_NAME:activemq.thread>((?!
).)*)(?:\s*)?(?<GREEDYMULTILINE:activemq.log.stack_trace>(.|\n|\t)*)/
```

(cherry picked from commit dbc80b9)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants