Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add log collection tool and thread #1987

Merged
merged 35 commits into from
Sep 10, 2020
Merged

Add log collection tool and thread #1987

merged 35 commits into from
Sep 10, 2020

Conversation

pgombar
Copy link
Contributor

@pgombar pgombar commented Aug 20, 2020

Description

This work is the continuation of #1902 and #1847. In this PR there are two main updates:

  1. Command-line utility for collecting logs is added.
  2. A thread is added that periodically calls the command-line utility to collect logs and upload them to the host. The log collection is invoked with systemd-run and is ran with CPU and memory limits.

Note that the feature is disabled by default for now.


PR information

  • The title of the PR is clear and informative.
  • There are a small number of commits, each of which has an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, see this page.
  • Except for special cases involving multiple contributors, the PR is started from a fork of the main repository, not a branch.
  • If applicable, the PR references the bug/issue that it fixes in the description.
  • New Unit tests were added for the changes made and Travis.CI is passing.

Quality of Code and Contribution Guidelines

for file in files_to_collect:
archive_file_name = LogCollector._convert_file_name_to_archive_name(file)
compressed_archive.write(file, arcname=archive_file_name)
with zipfile.ZipFile(COMPRESSED_ARCHIVE_PATH, "w", compression=zipfile.ZIP_DEFLATED) as compressed_archive:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, I found the change I made to make ZipFile python2.6 compatible: you can check out this commit in ExperimentalTestFramework's history: 600e6bb41f3d5506543454b27983a04cba1f1930.

Here's the before:

with zipfile.ZipFile(zip_file_buffer, "w", zipfile.ZIP_DEFLATED, False) as file:
    file.writestr("HandlerManifest.json", json.dumps(manifest))
    if extra_files != None:
        for key, value in extra_files.items():
            file.writestr(key, value)

return zip_file_buffer.getvalue()

and here's the after:

file = zipfile.ZipFile(zip_file_buffer, "w", zipfile.ZIP_DEFLATED, False)
try:
    file.writestr("HandlerManifest.json", json.dumps(manifest))
    if extra_files != None:
        for key, value in extra_files.items():
            file.writestr(key, value)

finally:
    file.close()

return zip_file_buffer.getvalue()

Don't know if we want to add python2.6 support in this PR, but this would possibly be useful as a comment to make it easy to add it in a future one.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the analysis! Since we are first targeting VMs with systemd (in order to enable resource limiting), none of them actually run py2.6, so I will take this change when we are expanding to non-systemd VMs. This will be useful for sure!

kevinclark19a
kevinclark19a previously approved these changes Sep 3, 2020
Copy link
Contributor

@larohra larohra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initial comments

azurelinuxagent/agent.py Outdated Show resolved Hide resolved
azurelinuxagent/agent.py Outdated Show resolved Hide resolved
azurelinuxagent/agent.py Outdated Show resolved Hide resolved
azurelinuxagent/agent.py Outdated Show resolved Hide resolved
azurelinuxagent/agent.py Show resolved Hide resolved
Copy link
Contributor

@larohra larohra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments on the main code, will review the tests now

azurelinuxagent/common/conf.py Show resolved Hide resolved
azurelinuxagent/ga/update.py Outdated Show resolved Hide resolved
azurelinuxagent/common/utils/textutil.py Outdated Show resolved Hide resolved
azurelinuxagent/common/logcollector.py Outdated Show resolved Hide resolved
azurelinuxagent/common/logcollector.py Show resolved Hide resolved
azurelinuxagent/ga/collect_logs.py Outdated Show resolved Hide resolved
azurelinuxagent/ga/collect_logs.py Outdated Show resolved Hide resolved
azurelinuxagent/ga/collect_logs.py Show resolved Hide resolved
azurelinuxagent/ga/collect_logs.py Outdated Show resolved Hide resolved
azurelinuxagent/ga/collect_logs.py Outdated Show resolved Hide resolved
Copy link
Contributor

@larohra larohra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final comments

tests/ga/test_collect_logs.py Outdated Show resolved Hide resolved
tests/ga/test_collect_logs.py Show resolved Hide resolved
tests/ga/test_collect_logs.py Show resolved Hide resolved
tests/ga/test_update.py Show resolved Hide resolved
tests/ga/test_update.py Show resolved Hide resolved
tests/test_agent.py Show resolved Hide resolved
tests/test_agent.py Outdated Show resolved Hide resolved
azurelinuxagent/agent.py Show resolved Hide resolved
tests/test_agent.py Outdated Show resolved Hide resolved
Copy link
Contributor

@larohra larohra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes, this would be really helpful once enabled! :)

@pgombar pgombar merged commit e79c9da into Azure:develop Sep 10, 2020
@pgombar pgombar deleted the log_tool branch September 10, 2020 17:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants