Ensure ECS compliant logging when enabled.

[simitt]

Motivation/summary

Move towards ECS compliant logging within the APM Server code by changing the log format and switching logging.ecs on by default.

Log files created within the libbeat code might still not be ECS compliant.

** Do not merge this PR until we are sure we want to backport to 7.9 **

Checklist

- [ ] I have signed the Contributor License Agreement.

• I have updated CHANGELOG.asciidoc

I have considered changes for:

• documentation
• logging (add log lines, choose appropriate log selector, etc.)
  - [ ] metrics and monitoring (create issue for Kibana team to add metrics to visualizations, e.g. Kibana#44001)
• automated tests (add tests for the code changes, all unit tests pass locally)
  - [ ] telemetry
• Elasticsearch Service (https://cloud.elastic.co)
  - [ ] Elastic Cloud Enterprise (https://www.elastic.co/products/ece)
  - [ ] Elastic Cloud on Kubernetes (https://www.elastic.co/elastic-cloud-kubernetes)

How to test these changes

Start APM Server and observe log lines. Following changes should be applied by default:

• logging.ecs is enabled, therefore ECS logs MVP fields are added
• error -> error.message
• stacktrace -> error.stacktrace
• url -> url.original
• request_id -> http.request.id
• method -> http.request.method
• content_length -> http.request.body.bytes
• remote_address -> source.address
• user-agent -> user_agent.original

Related issues

implements #3796

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #3829 updated

• Start Time: 2020-12-15T07:58:31.412+0000

• Duration: 54 min 39 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4613
Skipped	124
Total	4737

Steps errors

Expand to view the steps failures

Compress

• Took 0 min 0 sec . View more details on here
• Description: tar --exclude=coverage-files.tgz -czf coverage-files.tgz coverage

Compress

• Took 0 min 0 sec . View more details on here
• Description: tar --exclude=system-tests-linux-files.tgz -czf system-tests-linux-files.tgz system-tests

Test Sync

• Took 3 min 17 sec . View more details on here
• Description: ./.ci/scripts/sync.sh

[urso]

Instead of returning []interface and using Infow and Errorw consider to return a logger and add the labels via (*Logger).With.

[codecov-commenter]

Codecov Report

Merging #3829 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3829   +/-   ##
=======================================
  Coverage   75.16%   75.16%           
=======================================
  Files         138      138           
  Lines        6629     6629           
=======================================
  Hits         4983     4983           
  Misses       1646     1646           

[cla-checker-service]

💚 CLA has been signed

[simitt]

cla/check

[axw]

Looks great overall, just one issue with error.stacktrace

[simitt]

jenkins run the hey-apm tests please

[axw]

LGTM apart from the timestamp bit. Please merge away after that one is fixed.

[codecov-io]

Codecov Report

Merging #3829 (29b6d5e) into master (1fd28d8) will decrease coverage by 0.01%.
The diff coverage is 78.57%.

@@            Coverage Diff             @@
##           master    #3829      +/-   ##
==========================================
- Coverage   75.95%   75.94%   -0.02%     
==========================================
  Files         161      161              
  Lines        9787     9789       +2     
==========================================
  Hits         7434     7434              
- Misses       2353     2355       +2     

Impacted Files	Coverage Δ
beater/middleware/log_middleware.go	80.39% <78.57%> (-3.29%)	⬇️
...ack/apm-server/aggregation/txmetrics/aggregator.go	93.36% <0.00%> (ø)

[axw]

Tested with 7.11.0 BC1, using apm-integration-testing. By default, ECS is enabled.

{
  "@timestamp": "2020-12-21T06:06:39.084Z",
  "ecs.version": "1.6.0",
  "error.message": "404 page not found",
  "event.duration": 70288,
  "http.request.body.bytes": 0,
  "http.request.id": "419d573a82ecc524",
  "http.request.method": "GET",
  "http.response.status_code": 404,
  "log.level": "error",
  "log.logger": "request",
  "log.origin": {
    "file.line": 60,
    "file.name": "middleware/log_middleware.go"
  },
  "message": "404 page not found",
  "source.address": "172.29.0.1",
  "trace.id": "419d573a82ecc5245a96abbede83b173",
  "transaction.id": "419d573a82ecc524",
  "url.original": "/verloren",
  "user_agent.original": "curl/7.68.0"
}

http.request.id is not in ECS, but everything else is.

[graphaelli]

Good reminder, we have been meaning to get this into ECS for a long time - elastic/ecs#1208

[axw]

Good call on adding it, thanks!