Add log collector utility #1847
Conversation
Codecov Report
@@ Coverage Diff @@
## develop #1847 +/- ##
===========================================
+ Coverage 69.24% 69.58% +0.34%
===========================================
Files 83 84 +1
Lines 11483 11700 +217
Branches 1597 1626 +29
===========================================
+ Hits 7951 8142 +191
- Misses 3193 3213 +20
- Partials 339 345 +6
Continue to review full report at Codecov.
|
| :return: Returns True if the log collection succeeded | ||
| """ | ||
| try: | ||
| # Clear previous run's output and create base directories if they don't exist already |
There was a problem hiding this comment.
you should clean up as soon as you're done creating the zip instead of cleaning on the next run
There was a problem hiding this comment.
The idea was to leave the previous run's output for debugging purposes. For example, in a situation where the tool is invoked and the archive couldn't be created, not even the results file will be added to the archive. In that case, there is value in leaving that file on disk instead of deleting it.
| def format_command(cmd): | ||
| return " ".join(cmd) if isinstance(cmd, list) else command | ||
|
|
||
| def _encode_command_output(output): |
There was a problem hiding this comment.
I maintain utf-8 encoding throughout the module (when reading files and when writing to them), so I wanted to preserve this behaviour for outputs out of my control, e.g. outputs from shell commands or stack traces. Seems safe to keep it?
There was a problem hiding this comment.
But you are creating a utf8-encoded log file already. Why would this extra encoding would be needed?
| @@ -243,6 +243,10 @@ def get_ext_log_dir(conf=__conf__): | |||
| return conf.get("Extension.LogDir", "/var/log/azure") | |||
|
|
|||
|
|
|||
| def get_agent_log_file(): | |||
There was a problem hiding this comment.
Just a bit concerned about putting this here since its not part of the conf (Maybe osutil would be a better place since the other PR eventually plans to move it there).
There was a problem hiding this comment.
I still have to revisit the logic in the other PR, and if the ultimate goal is to expose the agent log parameter in the config file, this would be the correct place. Of course, this can change, so for now this I settled for this solution.
Description
First step of the feature to enable WALinuxAgent to periodically upload relevant log files to the host.
This PR adds the module that contains the functionality to collect logs based on a given manifest file. The format and contents of the manifest files is taken from Azure Inspect Disk Service.
The log collector module supports prioritization of important files in case there is not enough space for all files to be collected. Singe large files are not dropped, but truncated, where the freshest entries are kept.
During collection, the files are kept in a tarball that is incrementally updated on each collection. This means that in subsequent calls, only new files are added to the tarball, deleted files are removed, and modified files are updated, the rest stays the same. Finally, when all the files are collected, the tarball is compressed into a zip before it is sent to the host.
Future work entails adding a thread that will periodically call this utiilty (with limits on resource consumption) and send the zip to the host.
PR information
Quality of Code and Contribution Guidelines
This change is