Recognize log level in Elasticsearch JVM logs (#34159)

* Recognize log level in Elasticsearch JVM logs Elasticsearch will add a log level to its JVM logs to allow users to help them detect errors / warnings more easily. With this commit we detect this new field if present and continue to recognize the prior log format without a log level. Relates elastic/elasticsearch#92382 Closes #34054
elastic · Jun 1, 2023 · 6bfa9d4 · 6bfa9d4
1 parent 121df7d
commit 6bfa9d4
Show file tree

Hide file tree

Showing 4 changed files with 104 additions and 102 deletions.
diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
@@ -191,6 +191,7 @@ https://github.com/elastic/beats/compare/v8.2.0\...main[Check the HEAD diff]
 - Adding filename details from zip to response for httpjson {issue}33952[33952] {pull}34044[34044]
 - Allow user configuration of keep-alive behaviour for HTTPJSON and CEL inputs. {issue}33951[33951] {pull}34014[34014]
 - Add support for polling system UDP stats for UDP input metrics. {pull}34070[34070]
+- Add support for recognizing the log level in Elasticsearch JVM logs {pull}34159[34159]
 
 *Auditbeat*
 

diff --git a/filebeat/module/elasticsearch/gc/ingest/pipeline.yml b/filebeat/module/elasticsearch/gc/ingest/pipeline.yml
@@ -32,7 +32,7 @@ processors:
         (.|
         )*
       JVM8HEADER: '%{TIMESTAMP_ISO8601:timestamp}: %{BASE10NUM:elasticsearch.gc.jvm_runtime_sec}:'
-      JVM9HEADER: \[%{TIMESTAMP_ISO8601:timestamp}\]\[%{POSINT:process.pid}\]\[%{DATA:elasticsearch.gc.tags}%{SPACE}\]
+      JVM9HEADER: \[%{TIMESTAMP_ISO8601:timestamp}\]\[%{POSINT:process.pid}\](\[%{DATA:log.level}%{SPACE}\])?\[%{DATA:elasticsearch.gc.tags}%{SPACE}\]
       PROCTIME: '\[Times: user=%{BASE10NUM:elasticsearch.gc.phase.cpu_time.user_sec}
         sys=%{BASE10NUM:elasticsearch.gc.phase.cpu_time.sys_sec}, real=%{BASE10NUM:elasticsearch.gc.phase.cpu_time.real_sec}
         secs\]'

diff --git a/filebeat/module/elasticsearch/gc/test/gc.log b/filebeat/module/elasticsearch/gc/test/gc.log
@@ -1,4 +1,4 @@
-[2018-06-13T07:44:22.647+0000][32376][gc] Using Concurrent Mark Sweep
+[2018-06-13T07:44:22.647+0000][32376][info         ][gc] Using Concurrent Mark Sweep
 [2018-06-13T07:44:22.647+0000][32376][gc,heap,coops] Heap address: 0x00000000c0000000, size: 1024 MB, Compressed Oops mode: 32-bit
 [2018-06-13T07:44:22.725+0000][32376][safepoint    ] Application time: 0,0011068 seconds
 [2018-06-13T07:44:22.725+0000][32376][safepoint    ] Total time for which application threads were stopped: 0,0000563 seconds, Stopping threads took: 0,0000092 seconds