Refactor json generation log4j2 #142
Conversation
Use jackson directly to serialise the Log4j log events. Removes the old pattern layout approach. The new approach provides several callback and customisation features. They enable users to extend the library within their own applications. Furthermore, support for custom fields has been extended to support non-string values for top level fields. Also, CloudFoundry's VCAP environment variables are no longer added by default. The user has to opt in those fields The refactoring gives a factor 3 improvement on the JMH benchmarks for simple logs and an unchanged performance for request logs: Benchmark Mode Cnt Score Error Units EncodingBenchmarks.minimalRequestRecord thrpt 5 107274,684 ± 39379,575 ops/s EncodingBenchmarks.simpleLog thrpt 5 504838,974 ± 21313,854 ops/s EncodingBenchmarks.singleCommonFieldFromMdc thrpt 5 439796,897 ± 12174,050 ops/s EncodingBenchmarks.singleCustomFieldFromArgument thrpt 5 535903,222 ± 109794,213 ops/s
* add comment on different runtimes than CF * upgrade logback and log4j version Signed-off-by: Karsten Schnitter <k.schnitter@sap.com>
Add a message counter to Spring Boot sample app to show creation of additional fields by custom code. It counts all messages generated by the sample application. This is also helpful to detect missing messages in downstream logging systems. Signed-off-by: Karsten Schnitter <k.schnitter@sap.com>
.../src/main/java/com/sap/hcp/cf/logging/common/serialization/AbstractContextFieldSupplier.java
Show resolved
Hide resolved
|
|
||
| private String getNanoTs(LogEvent event) { | ||
| org.apache.logging.log4j.core.time.Instant instant = event.getInstant(); | ||
| return String.valueOf(instant.getEpochSecond() * 1_000_000_000L + instant.getNanoOfSecond()); |
There was a problem hiding this comment.
Isn't written_at defined as a long variable?
just wondering if this is for json precision issues, or actually an oversight.
There was a problem hiding this comment.
The nano timestamp is used for written_ts. This implementation was used before and is working with our logging services. So far epoch second seems to be small enough.
There was a problem hiding this comment.
Also wondering about this, because we had some precision issues with the number format of json being to small to handle nano timestamps.
We should investigate this further, as it might be a solution to the nano second rounding issues, or at least see to it being consistent with what will be ingested in elasticsearch, if we can interpret it as strings all the way into elasticsearch.
|
|
||
| Map<String, Object> fields = fieldSupplier.map(event); | ||
| assertThat(fields, hasEntry("key", "value")); | ||
| assertThat(fields, hasEntry("this", "that")); |
There was a problem hiding this comment.
Do we only want to support string values in all contexts?
In application logging the enforcement is clear,
just asking from a cloud logging perspective, as this might be seen as a short coming of this library.
There was a problem hiding this comment.
I will adjust this to match the logback test case, that contains a numerical value for the second field.
...java/com/sap/hcp/cf/logging/sample/springboot/statistics/StatisticsContextFieldSupplier.java
Show resolved
Hide resolved
Test numerical valued custom field as well. Signed-off-by: Karsten Schnitter <k.schnitter@sap.com>
The HttpAppender of Log4j2 will use the content type reported by the layout as HTTP header. To work well with e.g. Fluent Bit, the reported value was adapted to the actually generated JSON format. Signed-off-by: Karsten Schnitter <k.schnitter@sap.com>
Use Jackson directly to serialise the Log4j log events.
Removes the old pattern layout approach.
The new approach provides several callback and customisation
features. They enable users to extend the library within their own
applications. Furthermore, support for custom fields has been
extended to support non-string values for top level fields. Also,
CloudFoundry's VCAP environment variables are no longer added
by default. The user has to opt in those fields
The refactoring gives a factor 3 improvement on the JMH benchmarks
for simple logs and an unchanged performance for request logs: