Add UT for orchagent watchdog #8306

liuh-80 · 2023-05-15T02:14:34Z

Description of PR

Add UT for orchagent watchdog.

Summary:
SWSS service will add watchdog mechanism to generate keepalive message, and generate alert when swss have issue.
This PR will add new UT to cover the watchdog mechanism.

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Back port request

201911
202012
202205

Approach

What is the motivation for this PR?

Add new UT to test and protect watchdog mechanism from code change.

How did you do it?

Pause orchagent service with 'kill -stop' command and check if the watchdog can send alert.

How did you verify/test it?

Manually test new UT.
Pass PR validation.

Any platform specific information?

No

Supported testbed topology if it's a new test case?

Any

Documentation

mssonicbld · 2023-05-15T02:15:32Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Failed - hook id: trailing-whitespace - exit code: 1 - files were modified by this hook Fixing tests/system_health/test_watchdog.py fix end of files.........................................................Passed check yaml...............................................................Passed check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/system_health/test_watchdog.py:10:10: F821 undefined name 'logging' tests/system_health/test_watchdog.py:14:1: E302 expected 2 blank lines, found 1 tests/system_health/test_watchdog.py:17:121: E501 line too long (122 > 120 characters) tests/system_health/test_watchdog.py:34:120: E203 whitespace before ',' tests/system_health/test_watchdog.py:45:121: E501 line too long (126 > 120 characters) ... [truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-05-15T02:23:07Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Failed - hook id: trailing-whitespace - exit code: 1 - files were modified by this hook Fixing tests/system_health/test_watchdog.py fix end of files.........................................................Passed check yaml...............................................................Passed check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Passed check conditional mark sort..............................................Passed

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-05-15T02:25:42Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Failed - hook id: end-of-file-fixer - exit code: 1 - files were modified by this hook Fixing tests/system_health/test_watchdog.py check yaml...............................................................Passed check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Passed check conditional mark sort..............................................Passed

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

prgeor · 2023-05-16T01:13:46Z

tests/system_health/test_watchdog.py

+    duthost = duthosts[enum_rand_one_per_hwsku_hostname]
+
+    result = duthost.shell(
+                        r"docker exec -i swss sh -c 'test -f /usr/bin/supervisor-proc-watchdog-listener && echo exist'",


@liuh-80 can you point me to the watchdog listner code?

@prgeor here is the listener: sonic-net/sonic-buildimage#14686
I will update the PR later to merge the listener with process exit listener.

**What I did** Improve orch agent: output heartbeat message to systemd. **Why I did it** Currently SONiC monit system only monit orchagent process exist or not. If orchagent process stuck and stop processing, current monit can't find and report it. **How I verified it** Pass all UT. Manually validate the heartbeat message works correctly. **Details if related** Another inprogress PR will add watchdog for this heartbeat message: sonic-net/sonic-buildimage#14686 sonic-mgmt UT PR: sonic-net/sonic-mgmt#8306

…ave issue. (#14686) This PR depends on sonic-net/sonic-swss#2737 merge first. **What I did** Add orchagent watchdog to monitor and alert orchagent stuck issue. **Why I did it** Currently SONiC monit system only monit orchagent process exist or not. If orchagent process stuck and stop processing, current monit can't find and report it. **How I verified it** Pass all UT. Add new UT sonic-net/sonic-mgmt#8306 to check watchdog works correctly. Manually test, after pause orchagent with 'kill -STOP <pid>', check there are warning message exist in log: Apr 28 23:36:41.504923 vlab-01 ERR swss#supervisor-proc-watchdog-listener: Process 'orchagent' is stuck in namespace 'host' (1.0 minutes). **Details if related** Heartbeat message PR: sonic-net/sonic-swss#2737 UT PR: sonic-net/sonic-mgmt#8306

qiluo-msft · 2023-06-07T18:55:09Z

tests/system_health/test_watchdog.py

+        pytest.skip("Skip orchagent watchdog test.")
+
+    # wait watchdog emit alert
+    WATCHDOG_TIMEOUT = 120


120

Magic number. How do you get the value? Is it too long or possible too short if anyone change code accidently? #Closed

The watchdog will send alert after 60 seconds, so I wait for 60*2 here.

qiluo-msft · 2023-06-07T18:55:43Z

tests/system_health/test_watchdog.py

+def pause_orchagent(duthost):
+    # find orchagent pid
+    pid = duthost.shell(
+                    r"ps -ef | grep orchagent | grep -v grep | awk '{print $2}'",


ps -ef | grep orchagent | grep

Not a blocking issue, you may consider pgrep. #Closed

…ave issue. (#15429) Add watchdog mechanism to swss service and generate alert when swss have issue. **Work item tracking** Microsoft ADO (number only): 16578912 **What I did** Add orchagent watchdog to monitor and alert orchagent stuck issue. **Why I did it** Currently SONiC monit system only monit orchagent process exist or not. If orchagent process stuck and stop processing, current monit can't find and report it. **How I verified it** Pass all UT. Manually test process_monitoring/test_critical_process_monitoring.py can pass. Add new UT sonic-net/sonic-mgmt#8306 to check watchdog works correctly. Manually test, after pause orchagent with 'kill -STOP <pid>', check there are warning message exist in log: Apr 28 23:36:41.504923 vlab-01 ERR swss#supervisor-proc-watchdog-listener: Process 'orchagent' is stuck in namespace 'host' (1.0 minutes). **Details if related** Heartbeat message PR: sonic-net/sonic-swss#2737 UT PR: sonic-net/sonic-mgmt#8306

liuh-80 · 2023-06-13T04:20:23Z

/azp run

azure-pipelines · 2023-06-13T04:20:32Z

Azure Pipelines successfully started running 1 pipeline(s).

liuh-80 · 2023-06-14T00:56:02Z

/azp run

azure-pipelines · 2023-06-14T00:56:11Z

Azure Pipelines successfully started running 1 pipeline(s).

liuh-80 · 2023-06-14T07:31:57Z

system_health/test_watchdog.py::test_orchagent_watchdog[vlab-01] PASSED [100%]

Will merge PR.

**What I did** Improve orch agent: output heartbeat message to systemd. **Why I did it** Currently SONiC monit system only monit orchagent process exist or not. If orchagent process stuck and stop processing, current monit can't find and report it. **How I verified it** Pass all UT. Manually validate the heartbeat message works correctly. **Details if related** Another inprogress PR will add watchdog for this heartbeat message: sonic-net/sonic-buildimage#14686 sonic-mgmt UT PR: sonic-net/sonic-mgmt#8306

…ave issue. (sonic-net#14686) This PR depends on sonic-net/sonic-swss#2737 merge first. **What I did** Add orchagent watchdog to monitor and alert orchagent stuck issue. **Why I did it** Currently SONiC monit system only monit orchagent process exist or not. If orchagent process stuck and stop processing, current monit can't find and report it. **How I verified it** Pass all UT. Add new UT sonic-net/sonic-mgmt#8306 to check watchdog works correctly. Manually test, after pause orchagent with 'kill -STOP <pid>', check there are warning message exist in log: Apr 28 23:36:41.504923 vlab-01 ERR swss#supervisor-proc-watchdog-listener: Process 'orchagent' is stuck in namespace 'host' (1.0 minutes). **Details if related** Heartbeat message PR: sonic-net/sonic-swss#2737 UT PR: sonic-net/sonic-mgmt#8306

…ave issue. (sonic-net#15429) Add watchdog mechanism to swss service and generate alert when swss have issue. **Work item tracking** Microsoft ADO (number only): 16578912 **What I did** Add orchagent watchdog to monitor and alert orchagent stuck issue. **Why I did it** Currently SONiC monit system only monit orchagent process exist or not. If orchagent process stuck and stop processing, current monit can't find and report it. **How I verified it** Pass all UT. Manually test process_monitoring/test_critical_process_monitoring.py can pass. Add new UT sonic-net/sonic-mgmt#8306 to check watchdog works correctly. Manually test, after pause orchagent with 'kill -STOP <pid>', check there are warning message exist in log: Apr 28 23:36:41.504923 vlab-01 ERR swss#supervisor-proc-watchdog-listener: Process 'orchagent' is stuck in namespace 'host' (1.0 minutes). **Details if related** Heartbeat message PR: sonic-net/sonic-swss#2737 UT PR: sonic-net/sonic-mgmt#8306

### Description of PR Add UT for orchagent watchdog. Summary: SWSS service will add watchdog mechanism to generate keepalive message, and generate alert when swss have issue. This PR will add new UT to cover the watchdog mechanism. ### Type of change - [ ] Bug fix - [ ] Testbed and Framework(new/improvement) - [x] Test case(new/improvement) ### Back port request - [ ] 201911 - [ ] 202012 - [ ] 202205 ### Approach #### What is the motivation for this PR? Add new UT to test and protect watchdog mechanism from code change. #### How did you do it? Pause orchagent service with 'kill -stop' command and check if the watchdog can send alert. #### How did you verify/test it? Manually test new UT. Pass PR validation. #### Any platform specific information? No #### Supported testbed topology if it's a new test case? Any ### Documentation

Add UT for orchagent watchdog

fc81909

Fix code format

5220eb2

liuh-80 mentioned this pull request May 15, 2023

Add watchdog mechanism to swss service and generate alert when swss have issue. sonic-net/sonic-buildimage#14686

Merged

Fix code format

42d7c83

Fix code format

b537019

liuh-80 marked this pull request as ready for review May 15, 2023 02:41

liuh-80 requested a review from prgeor as a code owner May 15, 2023 02:41

liuh-80 requested a review from qiluo-msft May 15, 2023 02:41

liuh-80 mentioned this pull request May 15, 2023

Add orchagent heart beat message for watchdog. sonic-net/sonic-swss#2737

Merged

prgeor reviewed May 16, 2023

View reviewed changes

Improve UT

a7e11a8

qiluo-msft reviewed Jun 7, 2023

View reviewed changes

Improve UT

6d214e3

liuh-80 mentioned this pull request Jun 12, 2023

Add watchdog mechanism to swss service and generate alert when swss have issue. sonic-net/sonic-buildimage#15429

Merged

qiluo-msft approved these changes Jun 13, 2023

View reviewed changes

liuh-80 merged commit 16ca5ad into sonic-net:master Jun 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add UT for orchagent watchdog #8306

Add UT for orchagent watchdog #8306

liuh-80 commented May 15, 2023

mssonicbld commented May 15, 2023

mssonicbld commented May 15, 2023

mssonicbld commented May 15, 2023

prgeor May 16, 2023

liuh-80 May 22, 2023

qiluo-msft Jun 7, 2023 •

edited

Loading

liuh-80 Jun 8, 2023

qiluo-msft Jun 7, 2023 •

edited

Loading

liuh-80 Jun 8, 2023

liuh-80 commented Jun 13, 2023

azure-pipelines bot commented Jun 13, 2023

liuh-80 commented Jun 14, 2023

azure-pipelines bot commented Jun 14, 2023

liuh-80 commented Jun 14, 2023

Add UT for orchagent watchdog #8306

Add UT for orchagent watchdog #8306

Conversation

liuh-80 commented May 15, 2023

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

mssonicbld commented May 15, 2023

mssonicbld commented May 15, 2023

mssonicbld commented May 15, 2023

prgeor May 16, 2023

Choose a reason for hiding this comment

liuh-80 May 22, 2023

Choose a reason for hiding this comment

qiluo-msft Jun 7, 2023 • edited Loading

Choose a reason for hiding this comment

liuh-80 Jun 8, 2023

Choose a reason for hiding this comment

qiluo-msft Jun 7, 2023 • edited Loading

Choose a reason for hiding this comment

liuh-80 Jun 8, 2023

Choose a reason for hiding this comment

liuh-80 commented Jun 13, 2023

azure-pipelines bot commented Jun 13, 2023

liuh-80 commented Jun 14, 2023

azure-pipelines bot commented Jun 14, 2023

liuh-80 commented Jun 14, 2023

qiluo-msft Jun 7, 2023 •

edited

Loading

qiluo-msft Jun 7, 2023 •

edited

Loading