From 9ba53048c3b9f1b86ddc2eabc53e9bf24e697def Mon Sep 17 00:00:00 2001 From: Parth Pathak <100564707+e-parth-pathak@users.noreply.github.com> Date: Mon, 7 Oct 2024 22:17:50 +0530 Subject: [PATCH] Update elastic-agent-logging.md (#4483) The calculation of the amount of disk had a small mistake on line 70. Changed: ``` 210Mb (20Mb x 7 + 10Mb x 7 = 210Mb) ``` Co-authored-by: Pierre HILBERT Co-authored-by: Julien Lind (cherry picked from commit 1eb5e12dbc20d75334a4f87d0a341f66fee54bad) # Conflicts: # docs/elastic-agent-logging.md --- docs/elastic-agent-logging.md | 81 +++++++++++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100644 docs/elastic-agent-logging.md diff --git a/docs/elastic-agent-logging.md b/docs/elastic-agent-logging.md new file mode 100644 index 00000000000..1505db23092 --- /dev/null +++ b/docs/elastic-agent-logging.md @@ -0,0 +1,81 @@ +# Elastic-Agent logging +The Elastic-Agent process defines two log outputs: + - The "internal" core that is used by monitoring components and + collected in the diagnostics. It's configuration is **hardcoded**. + This output always logs to files in + `data/elastic-agent-/logs` and uses the default configuration + defined in `elastic-agent-libs/logp`. + - The "default" logger that is the user-configurable logger, it logs + to the Elastic-Agent's root folder, it can also be configured to + log to `stderr`. When running in a container environment, it + defaults to logging to `stderr`. + +## Logger initialization +The logger initialization is **not** one of the first things done by +the Elastic-Agent. Looking at the normal Elastic-Agent run, here is +the stack trace from the logging initialization. +``` + 0 0x00005e8768fea7ac in github.com/elastic/elastic-agent/pkg/core/logger.new + at /devel/elastic-agent/pkg/core/logger/logger.go:83 + 1 0x00005e8768fea54f in github.com/elastic/elastic-agent/pkg/core/logger.NewFromConfig + at /devel/elastic-agent/pkg/core/logger/logger.go:65 + 2 0x00005e876b511dba in github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent + at /devel/elastic-agent/internal/pkg/agent/cmd/run.go:151 + 3 0x00005e876b5119e5 in github.com/elastic/elastic-agent/internal/pkg/agent/cmd.run + at /devel/elastic-agent/internal/pkg/agent/cmd/run.go:138 + 4 0x00005e876b51127e in github.com/elastic/elastic-agent/internal/pkg/agent/cmd.newRunCommandWithArgs.func1 + at /devel/elastic-agent/internal/pkg/agent/cmd/run.go:78 +``` +This means some log entries might not be collected by diagnostics or +shipped to the monitoring output. Everything in the `run` function +happens before the logger initialization. + +https://github.com/elastic/elastic-agent/blob/574aa5db629231d56062ab40d27ccceb02cbbe4d/internal/pkg/agent/cmd/run.go#L104-L138 + +## Internal logging +The internal logging output is crucial for the Elastic-Agent +self-monitoring and diagnostics. It is instantiated by +[MakeInternalFileOutput](https://github.com/elastic/elastic-agent/blob/574aa5db629231d56062ab40d27ccceb02cbbe4d/pkg/core/logger/logger.go#L153-L182) +function that is called when a new logger is created. Its default +configuration is: + - 10Mb per log file + - Maximum of 7 log files + - Rotated on startup + - ECS/JSON encoded + - UTC timestamps + +There is also a second file output for events that is configured via +`agent.logging.event_data`. It shares the same log level as the main +logger and can only be configured for standalone agents. For +Fleet-Managed agents it will always use the default values: + - 5Mb per log file + - Maximum of 2 log files + - Do not rotate on startup + - ECS/JSON encoded + - UTC timestamps + +## Default logging +The default logger is the easiest to discover because it's user +configurable, logs to the Agent's root directory and can output to +`stderr`. It's default configuration comes from +https://github.com/elastic/elastic-agent/blob/574aa5db629231d56062ab40d27ccceb02cbbe4d/pkg/core/logger/logger.go#L132-L148 +and defaults to: + - 20Mb per log file + - Maximum of 7 log files + - Rotated on startup + - ECS/JSON are not explicitly set + +## Collecting logs for diagnostics +The Elastic-Agent will only collect +`data/elastic-agent-/logs`. The functions responsible for +collecting logs during diagnostics are: + - [`zipLogs`](https://github.com/elastic/elastic-agent/blob/574aa5db629231d56062ab40d27ccceb02cbbe4d/internal/pkg/diagnostics/diagnostics.go#L383-L415) + - [`zipLogsWithPath`](https://github.com/elastic/elastic-agent/blob/574aa5db629231d56062ab40d27ccceb02cbbe4d/internal/pkg/diagnostics/diagnostics.go#L418-L476) + +`zipLogsWithPath` always appends the `/logs` to whatever path it +receives. + +## Total footprint +Given the two log outputs and their default log rotation policies, the +Elastic-Agent needs about 210Mb (20Mb x 7 + 10Mb x 7 = 210Mb) of disk +for logging.