[sub-interface] | Err msg detected in syslog when add portchannel sub-interface: "Can't connect to teamd LAG='Po58.58', error='No such file or directory" #11218
Assignees
Labels
Comments
|
Encountered on Mellanox platform. However, issue is generic. Believe that BRCM made some changes lately and caused the regression. |
5 tasks
neethajohn
pushed a commit
to sonic-net/sonic-mgmt
that referenced
this issue
Jul 13, 2022
Summary: Use loganalyzer ignore marker to ignore 'config load' error logs. Ignore LAG and FDB related error logs found in test case sub_port_interfaces/test_sub_port_interfaces.py::test_admin_status_down_disables_forwarding Jul 8 07:00:39.302949 foo-DUT ERR teamd#tlm_teamd: :- process_add_queue: Can't connect to teamd after 10 attempts. LAG 'PortChannel1.20' Jul 8 07:28:02.105624 foo-DUT ERR swss#orchagent: :- update: Failed to get port by bridge port ID 0x3a00000000083f. How did you do it? For error log 1, Jul 8 07:00:39.302949 foo-DUT ERR teamd#tlm_teamd: :- process_add_queue: Can't connect to teamd after 10 attempts. LAG 'PortChannel1.20' It's raised after the test case creates some portchannel-sub-interfaces. But the portchannel-sub-interfaces have been created successfully: admin@foo:~/tmp/sub_port_interfaces$ show ip int Interface Master IPv4 address/mask Admin/Oper BGP Neighbor Neighbor IP --------------- -------- ------------------- ------------ -------------- ------------- Ethernet80.10 10.0.0.56/31 up/up ARISTA01BT1 10.0.0.57 Ethernet84.10 10.0.0.58/31 up/up ARISTA02BT1 10.0.0.59 Ethernet88.10 10.0.0.60/31 up/up ARISTA03BT1 10.0.0.61 Ethernet92.10 10.0.0.62/31 up/up ARISTA04BT1 10.0.0.63 Ethernet96.10 10.0.0.64/31 up/up ARISTA05BT1 10.0.0.65 Ethernet100.10 10.0.0.66/31 up/up ARISTA06BT1 10.0.0.67 Ethernet104.10 10.0.0.68/31 up/up ARISTA07BT1 10.0.0.69 Ethernet108.10 10.0.0.70/31 up/up ARISTA08BT1 10.0.0.71 Ethernet112.10 10.0.0.72/31 up/up ARISTA09BT1 10.0.0.73 Ethernet116.10 10.0.0.74/31 up/up ARISTA10BT1 10.0.0.75 Ethernet120.10 10.0.0.76/31 up/up ARISTA11BT1 10.0.0.77 Ethernet124.10 10.0.0.78/31 up/up ARISTA12BT1 10.0.0.79 Loopback0 10.1.0.32/32 up/up N/A N/A PortChannel1.20 172.16.0.1/30 up/up N/A N/A PortChannel1.30 172.16.0.5/30 up/up N/A N/A PortChannel1.40 172.16.0.9/30 up/up N/A N/A PortChannel1.50 172.16.0.13/30 up/up N/A N/A PortChannel2.20 172.16.4.1/30 up/up N/A N/A PortChannel2.30 172.16.4.5/30 up/up N/A N/A PortChannel2.40 172.16.4.9/30 up/up N/A N/A PortChannel2.50 172.16.4.13/30 up/up N/A N/A Vlan1000 192.168.0.1/21 up/up N/A N/A docker0 240.127.1.1/24 up/down N/A N/A eth0 10.64.247.28/23 up/up N/A N/A lo 127.0.0.1/16 up/up N/A N/A admin@foo:~/tmp/sub_port_interfaces$ ping 172.16.0.1 PING 172.16.0.1 (172.16.0.1) 56(84) bytes of data. 64 bytes from 172.16.0.1: icmp_seq=1 ttl=64 time=0.095 ms 64 bytes from 172.16.0.1: icmp_seq=2 ttl=64 time=0.078 ms 64 bytes from 172.16.0.1: icmp_seq=3 ttl=64 time=0.124 ms According to： sonic-net/sonic-buildimage#11218 I think we can safely ignore it in the sub port test case. For error log 2, Jul 8 07:28:02.105624 foo-DUT ERR swss#orchagent: :- update: Failed to get port by bridge port ID 0x3a00000000083f. It's raised after removing a member interface from the vlan, because the deleted interface loses its fdb entry. As for the log analyzer ignore marker in the 'config load', I've to do that slight refactor, otherwise we can't only ignore 2 patterns in ignore_regex_list, but also include all error logs caused by 'config load' How did you verify/test it? Run it on physical testbed: === Running tests in groups === /usr/local/lib/python2.7/dist-packages/ansible/parsing/vault/__init__.py:44: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release. from cryptography.exceptions import InvalidSignature ================================================================================================================== test session starts =================================================================================================================== platform linux2 -- Python 2.7.17, pytest-4.6.5, py-1.10.0, pluggy-0.13.1 ansible: 2.8.12 rootdir: /var/src/sonic-mgmt-int/tests, inifile: pytest.ini plugins: celery-4.4.7, forked-1.3.0, metadata-1.11.0, xdist-1.28.0, html-1.22.1, repeat-0.9.1, ansible-2.2.2 collecting ... /usr/local/lib/python2.7/dist-packages/ansible/parsing/vault/__init__.py:44: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release. from cryptography.exceptions import InvalidSignature collected 1 item sub_port_interfaces/test_sub_port_interfaces.py::TestSubPorts::test_admin_status_down_disables_forwarding[port_in_lag] PASSED [100%] --------------------------------------------------------------------------------------------- generated xml file: /var/src/sonic-mgmt-int/tests/logs/tr.xml ---------------------------------------------------------------------------------------------- =============================================================================================================== 1 passed in
|
@adyeung @preetham-singh Can you please share status on this issue? |
yejianquan
added a commit
to sonic-net/sonic-mgmt
that referenced
this issue
Jul 18, 2022
Summary: Use loganalyzer ignore marker to ignore 'config load' error logs. Ignore LAG and FDB related error logs found in test case sub_port_interfaces/test_sub_port_interfaces.py::test_admin_status_down_disables_forwarding Jul 8 07:00:39.302949 foo-DUT ERR teamd#tlm_teamd: :- process_add_queue: Can't connect to teamd after 10 attempts. LAG 'PortChannel1.20' Jul 8 07:28:02.105624 foo-DUT ERR swss#orchagent: :- update: Failed to get port by bridge port ID 0x3a00000000083f. How did you do it? For error log 1, Jul 8 07:00:39.302949 foo-DUT ERR teamd#tlm_teamd: :- process_add_queue: Can't connect to teamd after 10 attempts. LAG 'PortChannel1.20' It's raised after the test case creates some portchannel-sub-interfaces. But the portchannel-sub-interfaces have been created successfully: admin@foo:~/tmp/sub_port_interfaces$ show ip int Interface Master IPv4 address/mask Admin/Oper BGP Neighbor Neighbor IP --------------- -------- ------------------- ------------ -------------- ------------- Ethernet80.10 10.0.0.56/31 up/up ARISTA01BT1 10.0.0.57 Ethernet84.10 10.0.0.58/31 up/up ARISTA02BT1 10.0.0.59 Ethernet88.10 10.0.0.60/31 up/up ARISTA03BT1 10.0.0.61 Ethernet92.10 10.0.0.62/31 up/up ARISTA04BT1 10.0.0.63 Ethernet96.10 10.0.0.64/31 up/up ARISTA05BT1 10.0.0.65 Ethernet100.10 10.0.0.66/31 up/up ARISTA06BT1 10.0.0.67 Ethernet104.10 10.0.0.68/31 up/up ARISTA07BT1 10.0.0.69 Ethernet108.10 10.0.0.70/31 up/up ARISTA08BT1 10.0.0.71 Ethernet112.10 10.0.0.72/31 up/up ARISTA09BT1 10.0.0.73 Ethernet116.10 10.0.0.74/31 up/up ARISTA10BT1 10.0.0.75 Ethernet120.10 10.0.0.76/31 up/up ARISTA11BT1 10.0.0.77 Ethernet124.10 10.0.0.78/31 up/up ARISTA12BT1 10.0.0.79 Loopback0 10.1.0.32/32 up/up N/A N/A PortChannel1.20 172.16.0.1/30 up/up N/A N/A PortChannel1.30 172.16.0.5/30 up/up N/A N/A PortChannel1.40 172.16.0.9/30 up/up N/A N/A PortChannel1.50 172.16.0.13/30 up/up N/A N/A PortChannel2.20 172.16.4.1/30 up/up N/A N/A PortChannel2.30 172.16.4.5/30 up/up N/A N/A PortChannel2.40 172.16.4.9/30 up/up N/A N/A PortChannel2.50 172.16.4.13/30 up/up N/A N/A Vlan1000 192.168.0.1/21 up/up N/A N/A docker0 240.127.1.1/24 up/down N/A N/A eth0 10.64.247.28/23 up/up N/A N/A lo 127.0.0.1/16 up/up N/A N/A admin@foo:~/tmp/sub_port_interfaces$ ping 172.16.0.1 PING 172.16.0.1 (172.16.0.1) 56(84) bytes of data. 64 bytes from 172.16.0.1: icmp_seq=1 ttl=64 time=0.095 ms 64 bytes from 172.16.0.1: icmp_seq=2 ttl=64 time=0.078 ms 64 bytes from 172.16.0.1: icmp_seq=3 ttl=64 time=0.124 ms According to： sonic-net/sonic-buildimage#11218 I think we can safely ignore it in the sub port test case. For error log 2, Jul 8 07:28:02.105624 foo-DUT ERR swss#orchagent: :- update: Failed to get port by bridge port ID 0x3a00000000083f. It's raised after removing a member interface from the vlan, because the deleted interface loses its fdb entry. As for the log analyzer ignore marker in the 'config load', I've to do that slight refactor, otherwise we can't only ignore 2 patterns in ignore_regex_list, but also include all error logs caused by 'config load' How did you verify/test it? Run it on physical testbed: === Running tests in groups === /usr/local/lib/python2.7/dist-packages/ansible/parsing/vault/__init__.py:44: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release. from cryptography.exceptions import InvalidSignature ================================================================================================================== test session starts =================================================================================================================== platform linux2 -- Python 2.7.17, pytest-4.6.5, py-1.10.0, pluggy-0.13.1 ansible: 2.8.12 rootdir: /var/src/sonic-mgmt-int/tests, inifile: pytest.ini plugins: celery-4.4.7, forked-1.3.0, metadata-1.11.0, xdist-1.28.0, html-1.22.1, repeat-0.9.1, ansible-2.2.2 collecting ... /usr/local/lib/python2.7/dist-packages/ansible/parsing/vault/__init__.py:44: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release. from cryptography.exceptions import InvalidSignature collected 1 item sub_port_interfaces/test_sub_port_interfaces.py::TestSubPorts::test_admin_status_down_disables_forwarding[port_in_lag] PASSED [100%] --------------------------------------------------------------------------------------------- generated xml file: /var/src/sonic-mgmt-int/tests/logs/tr.xml ---------------------------------------------------------------------------------------------- =============================================================================================================== 1 passed in
|
BRCM will take a look |
tshalvi
pushed a commit
to tshalvi/sonic-buildimage
that referenced
this issue
Dec 20, 2022
…ABLE (sonic-net#2408) * Fix for issue sonic-net#11218 Avoid processing portchannel subinterfaces in teamd
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
Err msg detected in sysog: "Can't connect to teamd LAG='Po58.58', error='No such file or directory"
Steps to reproduce the issue:
1./home/admin# config portchannel add PortChannel58
2./home/admin# config portchannel member add PortChannel58 Ethernet0
3./home/admin# config subinterface add Po58.58 58
4.check the syslog, there is Err msg in the syslog: "Can't connect to teamd LAG='Po58.58', error='No such file or directory"
Describe the results you received:
WARNING teamd#tlm_teamd: :- try_add_lag: Can't connect to teamd LAG='Po58.58', error='No such file or directory'. attempt=1
WARNING teamd#tlm_teamd: :- try_add_lag: Can't connect to teamd LAG='Po58.58', error='No such file or directory'. attempt=2
WARNING teamd#tlm_teamd: :- try_add_lag: Can't connect to teamd LAG='Po58.58', error='No such file or directory'. attempt=3
WARNING teamd#tlm_teamd: :- try_add_lag: Can't connect to teamd LAG='Po58.58', error='No such file or directory'. attempt=4
WARNING teamd#tlm_teamd: :- try_add_lag: Can't connect to teamd LAG='Po58.58', error='No such file or directory'. attempt=5
WARNING teamd#tlm_teamd: :- try_add_lag: Can't connect to teamd LAG='Po58.58', error='No such file or directory'. attempt=6
WARNING teamd#tlm_teamd: :- try_add_lag: Can't connect to teamd LAG='Po58.58', error='No such file or directory'. attempt=7
WARNING teamd#tlm_teamd: :- try_add_lag: Can't connect to teamd LAG='Po58.58', error='No such file or directory'. attempt=8
WARNING teamd#tlm_teamd: :- try_add_lag: Can't connect to teamd LAG='Po58.58', error='No such file or directory'. attempt=9
Describe the results you expected:
There should be no such err msg
Output of
show version:Output of
show techsupport:Additional information you deem important (e.g. issue happens only occasionally):
The text was updated successfully, but these errors were encountered: