Skip to content

Commit

Permalink
[ErrorLog] Ignore teamd portchannel and fdb error log (#5951)
Browse files Browse the repository at this point in the history 
Summary:
Use loganalyzer ignore marker to ignore 'config load' error logs.
Ignore LAG and FDB related error logs found in test case sub_port_interfaces/test_sub_port_interfaces.py::test_admin_status_down_disables_forwarding

Jul 8 07:00:39.302949 foo-DUT ERR teamd#tlm_teamd: :- process_add_queue: Can't connect to teamd after 10 attempts. LAG 'PortChannel1.20'
Jul 8 07:28:02.105624 foo-DUT ERR swss#orchagent: :- update: Failed to get port by bridge port ID 0x3a00000000083f.

How did you do it?
For error log 1, Jul 8 07:00:39.302949 foo-DUT ERR teamd#tlm_teamd: :- process_add_queue: Can't connect to teamd after 10 attempts. LAG 'PortChannel1.20'
It's raised after the test case creates some portchannel-sub-interfaces.
But the portchannel-sub-interfaces have been created successfully:

admin@foo:~/tmp/sub_port_interfaces$ show ip int
Interface        Master    IPv4 address/mask    Admin/Oper    BGP Neighbor    Neighbor IP
---------------  --------  -------------------  ------------  --------------  -------------
Ethernet80.10              10.0.0.56/31         up/up         ARISTA01BT1     10.0.0.57
Ethernet84.10              10.0.0.58/31         up/up         ARISTA02BT1     10.0.0.59
Ethernet88.10              10.0.0.60/31         up/up         ARISTA03BT1     10.0.0.61
Ethernet92.10              10.0.0.62/31         up/up         ARISTA04BT1     10.0.0.63
Ethernet96.10              10.0.0.64/31         up/up         ARISTA05BT1     10.0.0.65
Ethernet100.10             10.0.0.66/31         up/up         ARISTA06BT1     10.0.0.67
Ethernet104.10             10.0.0.68/31         up/up         ARISTA07BT1     10.0.0.69
Ethernet108.10             10.0.0.70/31         up/up         ARISTA08BT1     10.0.0.71
Ethernet112.10             10.0.0.72/31         up/up         ARISTA09BT1     10.0.0.73
Ethernet116.10             10.0.0.74/31         up/up         ARISTA10BT1     10.0.0.75
Ethernet120.10             10.0.0.76/31         up/up         ARISTA11BT1     10.0.0.77
Ethernet124.10             10.0.0.78/31         up/up         ARISTA12BT1     10.0.0.79
Loopback0                  10.1.0.32/32         up/up         N/A             N/A
PortChannel1.20            172.16.0.1/30        up/up         N/A             N/A
PortChannel1.30            172.16.0.5/30        up/up         N/A             N/A
PortChannel1.40            172.16.0.9/30        up/up         N/A             N/A
PortChannel1.50            172.16.0.13/30       up/up         N/A             N/A
PortChannel2.20            172.16.4.1/30        up/up         N/A             N/A
PortChannel2.30            172.16.4.5/30        up/up         N/A             N/A
PortChannel2.40            172.16.4.9/30        up/up         N/A             N/A
PortChannel2.50            172.16.4.13/30       up/up         N/A             N/A
Vlan1000                   192.168.0.1/21       up/up         N/A             N/A
docker0                    240.127.1.1/24       up/down       N/A             N/A
eth0                       10.64.247.28/23      up/up         N/A             N/A
lo                         127.0.0.1/16         up/up         N/A             N/A

admin@foo:~/tmp/sub_port_interfaces$ ping 172.16.0.1
PING 172.16.0.1 (172.16.0.1) 56(84) bytes of data.
64 bytes from 172.16.0.1: icmp_seq=1 ttl=64 time=0.095 ms
64 bytes from 172.16.0.1: icmp_seq=2 ttl=64 time=0.078 ms
64 bytes from 172.16.0.1: icmp_seq=3 ttl=64 time=0.124 ms
According to： sonic-net/sonic-buildimage#11218
I think we can safely ignore it in the sub port test case.

For error log 2, Jul 8 07:28:02.105624 foo-DUT ERR swss#orchagent: :- update: Failed to get port by bridge port ID 0x3a00000000083f.
It's raised after removing a member interface from the vlan, because the deleted interface loses its fdb entry.

As for the log analyzer ignore marker in the 'config load', I've to do that slight refactor, otherwise we can't only ignore 2 patterns in ignore_regex_list, but also include all error logs caused by 'config load'

How did you verify/test it?
Run it on physical testbed:

=== Running tests in groups ===
/usr/local/lib/python2.7/dist-packages/ansible/parsing/vault/__init__.py:44: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
  from cryptography.exceptions import InvalidSignature
================================================================================================================== test session starts ===================================================================================================================
platform linux2 -- Python 2.7.17, pytest-4.6.5, py-1.10.0, pluggy-0.13.1
ansible: 2.8.12
rootdir: /var/src/sonic-mgmt-int/tests, inifile: pytest.ini
plugins: celery-4.4.7, forked-1.3.0, metadata-1.11.0, xdist-1.28.0, html-1.22.1, repeat-0.9.1, ansible-2.2.2
collecting ... /usr/local/lib/python2.7/dist-packages/ansible/parsing/vault/__init__.py:44: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
  from cryptography.exceptions import InvalidSignature
collected 1 item                                                                                                                                                                                                                                         

sub_port_interfaces/test_sub_port_interfaces.py::TestSubPorts::test_admin_status_down_disables_forwarding[port_in_lag] PASSED                                                                                                                      [100%]

--------------------------------------------------------------------------------------------- generated xml file: /var/src/sonic-mgmt-int/tests/logs/tr.xml ----------------------------------------------------------------------------------------------
=============================================================================================================== 1 passed in
  • Loading branch information
@yejianquan
yejianquan authored Jul 13, 2022
1 parent 8028f0d commit 4f1cee7
Show file tree
Hide file tree
Showing 2 changed files with 18 additions and 16 deletions.
24 changes: 8 additions & 16 deletions tests/sub_port_interfaces/conftest.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@

from tests.common import config_reload
from tests.common.helpers.assertions import pytest_assert as py_assert
from tests.common.platform.processes_utils import wait_critical_processes
from tests.common.utilities import wait_until
from tests.common.ptf_agent_updater import PtfAgentUpdater
from tests.common import constants
Expand Down Expand Up @@ -496,7 +497,7 @@ def apply_balancing_config(duthost, ptfhost, ptfadapter, define_sub_ports_config


@pytest.fixture
def reload_dut_config(request, duthost, define_sub_ports_configuration):
def reload_dut_config(request, duthost, define_sub_ports_configuration, loganalyzer):
"""
DUT's configuration reload on teardown
Expand All @@ -506,6 +507,9 @@ def reload_dut_config(request, duthost, define_sub_ports_configuration):
define_sub_ports_configuration: Dictonary of parameters for configuration DUT
"""
yield
if loganalyzer and loganalyzer[duthost.hostname]:
loganalyzer[duthost.hostname].add_start_ignore_mark()

sub_ports = define_sub_ports_configuration['sub_ports']
dut_ports = define_sub_ports_configuration['dut_ports']
cfg_facts = duthost.config_facts(host=duthost.hostname, source="running")['ansible_facts']
Expand All @@ -521,7 +525,9 @@ def reload_dut_config(request, duthost, define_sub_ports_configuration):
remove_lag_port(duthost, cfg_facts, lag_port)

duthost.shell('sudo config load -y /etc/sonic/config_db.json')

wait_critical_processes(duthost)
if loganalyzer and loganalyzer[duthost.hostname]:
loganalyzer[duthost.hostname].add_end_ignore_mark()

@pytest.fixture
def reload_ptf_config(request, ptfhost, define_sub_ports_configuration):
Expand Down Expand Up @@ -561,17 +567,3 @@ def teardown_test_class(duthost):
"""
yield
config_reload(duthost)


def pytest_generate_tests(metafunc):
# try to put fixture loganalyzer after fixture reload_dut_config, during
# teardown, loganalyzer will be executed before reload_dut_config, so any
# log error introduced by config load will be skipped.
try:
metafunc.fixturenames.index("loganalyzer")
reload_dut_config_index = metafunc.fixturenames.index("reload_dut_config")
except ValueError:
return
else:
metafunc.fixturenames.remove("loganalyzer")
metafunc.fixturenames.insert(reload_dut_config_index + 1, "loganalyzer")
10 changes: 10 additions & 0 deletions tests/sub_port_interfaces/test_sub_port_interfaces.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -476,3 +476,13 @@ def test_balancing_sub_ports(self, duthost, ptfhost, ptfadapter, apply_balancing
ip_dst=ip_dst,
type_of_traffic='balancing',
ttl=63)


@pytest.fixture(autouse=True)
def ignore_expected_loganalyzer_exception(duthost, loganalyzer):
if loganalyzer and loganalyzer[duthost.hostname]:
ignore_regex_list = [
".*ERR teamd[0-9]*#tlm_teamd.*process_add_queue: Can't connect to teamd after.*attempts. LAG 'PortChannel.*'",
".*ERR swss[0-9]*#orchagent.*update: Failed to get port by bridge port ID.*"
]
loganalyzer[duthost.hostname].ignore_regex.extend(ignore_regex_list)

0 comments on commit 4f1cee7

Please sign in to comment.