Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

One thread for telemetry #2019

Merged
merged 83 commits into from
Nov 5, 2020
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
83 commits
Select commit Hold shift + click to select a range
04b4bb4
Added a new thread for sending telemetry data
larohra Sep 11, 2020
97bc0b6
Added a new priority queue for sending events
larohra Sep 14, 2020
10added
Added stacktrace for failure
larohra Sep 15, 2020
647bbaf
Added stacktrace for failures and made it a set
larohra Sep 15, 2020
2896b23
Added a counter to pq to avoid collisions and more logging
larohra Sep 15, 2020
a9e7beb
Changed logging to verbose
larohra Sep 15, 2020
540f5f1
Added a global counter for PQ
larohra Sep 16, 2020
b21f97c
Added more logging
larohra Sep 16, 2020
c00b28d
Fixed faulty condition
larohra Sep 17, 2020
3b7b393
Implemented comparator for TelemetryEvent class
larohra Sep 17, 2020
859843c
Moved class
larohra Sep 17, 2020
76f1290
Code fix and more logging
larohra Sep 17, 2020
e0eb752
Removed dead code
larohra Sep 17, 2020
d45d73e
Made ETP code compatible with new telemetry service
larohra Sep 18, 2020
b55c8e5
Added more debug data
larohra Sep 18, 2020
1b4a240
More logging
larohra Sep 18, 2020
27d5ca8
bug fix
larohra Sep 18, 2020
b019840
Improved logging
larohra Sep 18, 2020
b8a7248
Fixed some failing tests
larohra Sep 19, 2020
38c4e86
Hijacking verbose flag for testing
larohra Sep 21, 2020
7f9a723
Refactored and fixed ETP and ETP tests
larohra Sep 21, 2020
b2b33a0
Added queue.join for telemetry_service
larohra Sep 21, 2020
a072f3d
Added tests for the telemetry_service
larohra Sep 22, 2020
de7d744
Fixed some tests and code
larohra Sep 23, 2020
318f7a1
Fixed final failing tests
larohra Sep 23, 2020
16561fe
Fixed linter errors
larohra Sep 23, 2020
8c7207b
Fixed linter errors - pt2
larohra Sep 23, 2020
98e8c3a
Fixed failing py3 test
larohra Sep 23, 2020
b6b3f07
Merge remote-tracking branch 'upstream/develop' into one-thread-for-t…
larohra Sep 23, 2020
ea94628
Added thread interface to Telemetry service and minor bug fixes
larohra Sep 23, 2020
340c6f4
Final checks, code cleanup
larohra Sep 23, 2020
a686f25
Minor PR comments
larohra Oct 1, 2020
f48fab5
Added a linter comment
larohra Oct 1, 2020
8908db1
Merge remote-tracking branch 'upstream/develop' into one-thread-for-t…
larohra Oct 6, 2020
357403a
Combined both telemetry event collections to a single thread
larohra Oct 7, 2020
ea760ab
Renamed enqueue_event_func to enqueue_event
larohra Oct 7, 2020
59174c1
Added batching support
larohra Oct 7, 2020
4840315
Removed priority queue
larohra Oct 7, 2020
8255dea
Introduced ServiceStoppedError and changes around it
larohra Oct 7, 2020
99b7954
Code cleanup, introduced a small class only for debug purposes
larohra Oct 8, 2020
e8231de
Merge remote-tracking branch 'upstream/develop' into one-thread-for-t…
larohra Oct 8, 2020
905710a
Final review changes, code cleanup and added more try-catches
larohra Oct 9, 2020
6e2f6e7
Reverted the is_extension_telemetry_pipeline_enabled flag position
larohra Oct 9, 2020
4c43bd0
Fixed test_monitor.py
larohra Oct 9, 2020
dfc522b
Fixed failing tests
larohra Oct 12, 2020
7b3226e
Merge remote-tracking branch 'upstream/develop' into one-thread-for-t…
larohra Oct 12, 2020
e33500d
Fixed failing TestUpdate tests
larohra Oct 13, 2020
770bbbc
Merge branch 'develop' into one-thread-for-telemetry
larohra Oct 13, 2020
cc3e23a
Fixed linter errors
larohra Oct 13, 2020
ac4e361
Merge remote-tracking branch 'upstream/develop' into one-thread-for-t…
larohra Oct 13, 2020
cd9ef0c
Merge remote-tracking branch 'origin/one-thread-for-telemetry' into o…
larohra Oct 13, 2020
a6373bd
Change thread name
larohra Oct 13, 2020
560d974
Cleaning up before starting the flaky test
larohra Oct 13, 2020
457cba4
Removed unnecessary logs and added a verbose log
larohra Oct 13, 2020
7d59cca
Improved log message
larohra Oct 13, 2020
c70e3d4
Code cleanup
larohra Oct 14, 2020
a907724
Merge branch 'develop' into one-thread-for-telemetry
larohra Oct 14, 2020
589e669
addressed PR comments
larohra Oct 15, 2020
fb2004e
Merge remote-tracking branch 'origin/one-thread-for-telemetry' into o…
larohra Oct 15, 2020
af0465c
PR comments pt2
larohra Oct 16, 2020
fa51326
Merge remote-tracking branch 'upstream/develop' into one-thread-for-t…
larohra Oct 16, 2020
a24718b
Merge branch 'develop' into one-thread-for-telemetry
larohra Oct 16, 2020
a7e2f6a
Fix for flaky test
larohra Oct 16, 2020
15389cb
Merge remote-tracking branch 'origin/one-thread-for-telemetry' into o…
larohra Oct 16, 2020
6c82c1e
Addressed PR comments
larohra Oct 19, 2020
c990cd8
Fixed failing tests
larohra Oct 19, 2020
6d37dff
Minor code cleanup
larohra Oct 19, 2020
5bc6a10
Renamed telemetry_service.py to send_telemetry_events.py
larohra Oct 20, 2020
0cf5543
Removed event based synchronization
larohra Oct 26, 2020
5af6e1a
Fix failing tests
larohra Oct 26, 2020
8818180
Addressed PR comments
larohra Oct 27, 2020
65c61fe
Merge remote-tracking branch 'upstream/develop' into one-thread-for-t…
larohra Oct 27, 2020
c76cb89
Merge remote-tracking branch 'upstream/develop' into one-thread-for-t…
larohra Oct 28, 2020
7e8a223
Updated changes and removed dead linter errors
larohra Oct 28, 2020
498358c
Merge branch 'develop' into one-thread-for-telemetry
larohra Oct 29, 2020
8f61933
Addressed PR comments
larohra Oct 29, 2020
24b4a13
Improved comments and function names
larohra Nov 4, 2020
6691200
Merge remote-tracking branch 'upstream/develop' into one-thread-for-t…
larohra Nov 4, 2020
722c1fc
Merge branch 'develop' into one-thread-for-telemetry
larohra Nov 4, 2020
6e0609b
Merge branch 'develop' into one-thread-for-telemetry
larohra Nov 4, 2020
99a7bfe
Merge branch 'develop' into one-thread-for-telemetry
larohra Nov 4, 2020
30e56c0
Addressed PR comments and removed unused import
larohra Nov 4, 2020
0852353
Merge branch 'develop' into one-thread-for-telemetry
larohra Nov 4, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions azurelinuxagent/common/cgroupapi.py
Original file line number Diff line number Diff line change
Expand Up @@ -385,7 +385,6 @@ def start_extension_command(self, extension_name, command, timeout, shell, cwd,
:param stderr: File object to redirect stderr to
:param error_code: Extension error code to raise in case of error
"""
# pylint: disable=too-many-locals
try:
extension_cgroups = self.create_extension_cgroups(extension_name)
except Exception as exception:
Expand Down Expand Up @@ -577,7 +576,7 @@ def create_extension_cgroups_root(self):
logger.info("Created slice for walinuxagent extensions {0}".format(unit_filename))

def create_extension_cgroups(self, extension_name):
# TODO: The slice created by this function is not used currently. We need to create the extension scopes within # pylint: disable=W0511
# TODO: The slice created by this function is not used currently. We need to create the extension scopes within
# this slice and use the slice to monitor the cgroups. Also see comment in get_extension_cgroups.
# the slice.
unit_contents = """
Expand Down Expand Up @@ -611,7 +610,7 @@ def remove_extension_cgroups(self, extension_name):
raise CGroupsException("Failed to remove {0}. Error: {1}".format(unit_filename, ustr(e)))

def get_extension_cgroups(self, extension_name):
# TODO: The slice returned by this function is not used currently. We need to create the extension scopes within # pylint: disable=W0511
# TODO: The slice returned by this function is not used currently. We need to create the extension scopes within
# this slice and use the slice to monitor the cgroups. Also see comment in create_extension_cgroups.
slice_name = self._get_extension_cgroup_name(extension_name)

Expand Down Expand Up @@ -661,7 +660,7 @@ def _is_systemd_failure(scope_name, stderr):
unit_not_found = "Unit {0} not found.".format(scope_name)
return unit_not_found in stderr or scope_name not in stderr

def start_extension_command(self, extension_name, command, timeout, shell, cwd, env, stdout, stderr, # pylint: disable=R0913,R0914
def start_extension_command(self, extension_name, command, timeout, shell, cwd, env, stdout, stderr, # pylint: disable=R0913
error_code=ExtensionErrorCodes.PluginUnknownFailure):
scope = "{0}_{1}".format(self._get_extension_cgroup_name(extension_name), uuid.uuid4())

Expand Down
1 change: 0 additions & 1 deletion azurelinuxagent/common/cgroupconfigurator.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,6 @@ def __init__(self):

# too-many-branches<R0912> Disabled: branches are sequential, not nested
def initialize(self): # pylint: disable=R0912
# pylint: disable=too-many-locals
try:
if self._initialized:
return
Expand Down
18 changes: 9 additions & 9 deletions azurelinuxagent/common/dhcp.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,23 +14,23 @@
#
# Requires Python 2.6+ and Openssl 1.0+

import array
import os
import socket
import array
import time

import azurelinuxagent.common.logger as logger
from azurelinuxagent.common.exception import DhcpError
from azurelinuxagent.common.osutil import get_osutil
from azurelinuxagent.common.utils.restutil import KNOWN_WIRESERVER_IP
from azurelinuxagent.common.utils.textutil import hex_dump, hex_dump2, \
hex_dump3, \
compare_bytes, str_to_ord, \
unpack_big_endian, \
int_to_ip4_addr
from azurelinuxagent.common.exception import DhcpError
from azurelinuxagent.common.osutil import get_osutil


# the kernel routing table representation of 168.63.129.16
KNOWN_WIRESERVER_IP_ENTRY = '10813FA8'
from azurelinuxagent.common.utils.restutil import KNOWN_WIRESERVER_IP # pylint: disable=C0413


def get_dhcp_handler():
Expand Down Expand Up @@ -100,7 +100,7 @@ def wireserver_route_exists(self):
logger.error(
"Could not determine whether route exists to {0}: {1}".format(
KNOWN_WIRESERVER_IP, e))

return route_exists

@property
Expand Down Expand Up @@ -162,9 +162,9 @@ def send_dhcp_req(self):
Build dhcp request with mac addr
Configure route to allow dhcp traffic
Stop dhcp service if necessary
"""
"""
# pylint: enable=W0105

logger.info("Send dhcp request")
mac_addr = self.osutil.get_mac_addr()

Expand Down Expand Up @@ -389,7 +389,7 @@ def build_dhcp_request(mac_addr, request_broadcast):
# set broadcast flag to true to request the dhcp server
# to respond to a boradcast address,
# this is useful when user dhclient fails.
request[0x0A] = 0x80; # pylint: disable=W0301
request[0x0A] = 0x80

# fill in ClientHardwareAddress
for a in range(0, 6): # pylint: disable=C0103
Expand Down
4 changes: 0 additions & 4 deletions azurelinuxagent/common/event.py
Original file line number Diff line number Diff line change
Expand Up @@ -523,7 +523,6 @@ def add_metric(self, category, counter, instance, value, log_event=False): # pyl
:param bool log_event: If true, log the collected metric in the agent log
"""
if log_event:
from azurelinuxagent.common.version import AGENT_NAME # pylint: disable=W0404,W0621
message = "Metric {0}/{1} [{2}] = {3}".format(category, counter, instance, value)
_log_event(AGENT_NAME, "METRIC", message, 0)

Expand Down Expand Up @@ -621,7 +620,6 @@ def elapsed_milliseconds(utc_start):


def report_event(op, is_success=True, message='', log_event=True): # pylint: disable=C0103
from azurelinuxagent.common.version import AGENT_NAME, CURRENT_VERSION # pylint: disable=W0404,W0621
add_event(AGENT_NAME,
version=str(CURRENT_VERSION),
is_success=is_success,
Expand All @@ -631,7 +629,6 @@ def report_event(op, is_success=True, message='', log_event=True): # pylint: dis


def report_periodic(delta, op, is_success=True, message=''): # pylint: disable=C0103
from azurelinuxagent.common.version import AGENT_NAME, CURRENT_VERSION # pylint: disable=W0404,W0621
add_periodic(delta, AGENT_NAME,
version=str(CURRENT_VERSION),
is_success=is_success,
Expand All @@ -650,7 +647,6 @@ def report_metric(category, counter, instance, value, log_event=False, reporter=
:param EventLogger reporter: The EventLogger instance to which metric events should be sent
"""
if reporter.event_dir is None:
from azurelinuxagent.common.version import AGENT_NAME # pylint: disable=W0404,W0621
logger.warn("Cannot report metric event -- Event reporter is not initialized.")
message = "Metric {0}/{1} [{2}] = {3}".format(category, counter, instance, value)
_log_event(AGENT_NAME, "METRIC", message, 0)
Expand Down
4 changes: 2 additions & 2 deletions azurelinuxagent/common/future.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,12 +33,12 @@
# unused-import<W0611>, import-error<E0401> Disabled: Due to backward compatibility between py2 and py3
from builtins import int, range # pylint: disable=unused-import,import-error
from collections import OrderedDict # pylint: disable=W0611
from queue import Queue, Full # pylint: disable=W0611,import-error
from queue import Queue, Empty # pylint: disable=W0611,import-error

elif sys.version_info[0] == 2:
import httplib as httpclient # pylint: disable=E0401,W0611
from urlparse import urlparse # pylint: disable=E0401
from Queue import Queue, Full # pylint: disable=W0611,import-error
from Queue import Queue, Empty # pylint: disable=W0611,import-error


# We want to suppress the following:
Expand Down
4 changes: 2 additions & 2 deletions azurelinuxagent/common/logger.py
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ def write_log(log_appender): # pylint: disable=W0612
for appender in self.appenders:
appender.write(level, log_item)
#
# TODO: we should actually call # pylint: disable=W0511
# TODO: we should actually call
#
# write_log(appender)
#
Expand All @@ -158,7 +158,7 @@ def write_log(log_appender): # pylint: disable=W0612
for appender in self.logger.appenders:
appender.write(level, log_item)
#
# TODO: call write_log instead (see comment above) # pylint: disable=W0511
# TODO: call write_log instead (see comment above)
#

def add_appender(self, appender_type, level, path):
Expand Down
4 changes: 2 additions & 2 deletions azurelinuxagent/common/osutil/default.py
Original file line number Diff line number Diff line change
Expand Up @@ -791,7 +791,7 @@ def get_first_if(self):
return '', ''

@staticmethod
def _build_route_list(proc_net_route): # pylint: disable=R0914
def _build_route_list(proc_net_route):
"""
Construct a list of network route entries
:param list(str) proc_net_route: Route table lines, including headers, containing at least one route
Expand Down Expand Up @@ -939,7 +939,7 @@ def get_dhcp_lease_endpoint(self):
return None

@staticmethod
def get_endpoint_from_leases_path(pathglob): # pylint: disable=R0912,R0914
def get_endpoint_from_leases_path(pathglob): # pylint: disable=R0912
"""
Try to discover and decode the wireserver endpoint in the
specified dhcp leases path.
Expand Down
Loading