[Sys Mon] Fix the service entry delete in state_db because of timer job #14702
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>
Junchao-Mellanox
approved these changes
Apr 19, 2023
|
@lguohan could you pls suggest reviewer? |
|
@sg893052 could you please help to review the issue? |
sg893052
approved these changes
Apr 24, 2023
|
@lguohan @qiluo-msft could you please help to merge? |
mssonicbld
pushed a commit
to mssonicbld/sonic-buildimage
that referenced
this pull request
May 17, 2023
…ob (sonic-net#14702) Why I did it systemd stop event on service with timers can sometime delete the state_db entry for the corresponding service. Note: This won't be observed on the latest master label since the dependency on timer was removed with the recent config reload enhancement. However, it is better to have the fix since there might be some systemd services added to system health daemon in the future which may contain timers root@qa-eth-vt01-4-3700c:/home/admin# systemctl stop snmp root@qa-eth-vt01-4-3700c:/home/admin# show system-health sysready-status System is not ready - one or more services are not up Service-Name Service-Status App-Ready-Status Down-Reason ---------------------- ---------------- ------------------ ------------- <Truncated> ssh OK OK - swss OK OK - syncd OK OK - sysstat OK OK - teamd OK OK - telemetry OK OK - what-just-happened OK OK - ztp OK OK - <Truncated> Expected Should see a Down entry for SNMP instead of the entry being deleted from the STATE_DB root@qa-eth-vt01-4-3700c:/home/admin# show system-health sysready-status System is not ready - one or more services are not up Service-Name Service-Status App-Ready-Status Down-Reason ---------------------- ---------------- ------------------ ------------- <Truncated> snmp Down Down Inactive ssh OK OK - swss OK OK - syncd OK OK - sysstat OK OK - teamd OK OK - telemetry OK OK - what-just-happened OK OK - ztp OK OK - <Truncated> How I did it Happens because the timer is usually a PartOf service and thus a stop on service is propagated to timer. Fixed the logic to handle this Apr 18 02:06:47.711252 r-lionfish-16 DEBUG healthd: Main process- received event:snmp.service from source:sysbus time:2023-04-17 23:06:47 Apr 18 02:06:47.711347 r-lionfish-16 INFO healthd: check_unit_status for [ snmp.service ] Apr 18 02:06:47.722363 r-lionfish-16 INFO healthd: snmp.service service state changed to [inactive/dead] Apr 18 02:06:47.723230 r-lionfish-16 DEBUG healthd: Main process- received event:snmp.timer from source:sysbus time:2023-04-17 23:06:47 Apr 18 02:06:47.723328 r-lionfish-16 INFO healthd: check_unit_status for [ snmp.timer ] Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>
|
Cherry-pick PR to 202205: #15121 |
Merged
10 tasks
mssonicbld
pushed a commit
to mssonicbld/sonic-buildimage
that referenced
this pull request
May 17, 2023
…ob (sonic-net#14702) Why I did it systemd stop event on service with timers can sometime delete the state_db entry for the corresponding service. Note: This won't be observed on the latest master label since the dependency on timer was removed with the recent config reload enhancement. However, it is better to have the fix since there might be some systemd services added to system health daemon in the future which may contain timers root@qa-eth-vt01-4-3700c:/home/admin# systemctl stop snmp root@qa-eth-vt01-4-3700c:/home/admin# show system-health sysready-status System is not ready - one or more services are not up Service-Name Service-Status App-Ready-Status Down-Reason ---------------------- ---------------- ------------------ ------------- <Truncated> ssh OK OK - swss OK OK - syncd OK OK - sysstat OK OK - teamd OK OK - telemetry OK OK - what-just-happened OK OK - ztp OK OK - <Truncated> Expected Should see a Down entry for SNMP instead of the entry being deleted from the STATE_DB root@qa-eth-vt01-4-3700c:/home/admin# show system-health sysready-status System is not ready - one or more services are not up Service-Name Service-Status App-Ready-Status Down-Reason ---------------------- ---------------- ------------------ ------------- <Truncated> snmp Down Down Inactive ssh OK OK - swss OK OK - syncd OK OK - sysstat OK OK - teamd OK OK - telemetry OK OK - what-just-happened OK OK - ztp OK OK - <Truncated> How I did it Happens because the timer is usually a PartOf service and thus a stop on service is propagated to timer. Fixed the logic to handle this Apr 18 02:06:47.711252 r-lionfish-16 DEBUG healthd: Main process- received event:snmp.service from source:sysbus time:2023-04-17 23:06:47 Apr 18 02:06:47.711347 r-lionfish-16 INFO healthd: check_unit_status for [ snmp.service ] Apr 18 02:06:47.722363 r-lionfish-16 INFO healthd: snmp.service service state changed to [inactive/dead] Apr 18 02:06:47.723230 r-lionfish-16 DEBUG healthd: Main process- received event:snmp.timer from source:sysbus time:2023-04-17 23:06:47 Apr 18 02:06:47.723328 r-lionfish-16 INFO healthd: check_unit_status for [ snmp.timer ] Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>
|
Cherry-pick PR to 202211: #15122 |
Merged
10 tasks
mssonicbld
pushed a commit
that referenced
this pull request
May 18, 2023
…ob (#14702) Why I did it systemd stop event on service with timers can sometime delete the state_db entry for the corresponding service. Note: This won't be observed on the latest master label since the dependency on timer was removed with the recent config reload enhancement. However, it is better to have the fix since there might be some systemd services added to system health daemon in the future which may contain timers root@qa-eth-vt01-4-3700c:/home/admin# systemctl stop snmp root@qa-eth-vt01-4-3700c:/home/admin# show system-health sysready-status System is not ready - one or more services are not up Service-Name Service-Status App-Ready-Status Down-Reason ---------------------- ---------------- ------------------ ------------- <Truncated> ssh OK OK - swss OK OK - syncd OK OK - sysstat OK OK - teamd OK OK - telemetry OK OK - what-just-happened OK OK - ztp OK OK - <Truncated> Expected Should see a Down entry for SNMP instead of the entry being deleted from the STATE_DB root@qa-eth-vt01-4-3700c:/home/admin# show system-health sysready-status System is not ready - one or more services are not up Service-Name Service-Status App-Ready-Status Down-Reason ---------------------- ---------------- ------------------ ------------- <Truncated> snmp Down Down Inactive ssh OK OK - swss OK OK - syncd OK OK - sysstat OK OK - teamd OK OK - telemetry OK OK - what-just-happened OK OK - ztp OK OK - <Truncated> How I did it Happens because the timer is usually a PartOf service and thus a stop on service is propagated to timer. Fixed the logic to handle this Apr 18 02:06:47.711252 r-lionfish-16 DEBUG healthd: Main process- received event:snmp.service from source:sysbus time:2023-04-17 23:06:47 Apr 18 02:06:47.711347 r-lionfish-16 INFO healthd: check_unit_status for [ snmp.service ] Apr 18 02:06:47.722363 r-lionfish-16 INFO healthd: snmp.service service state changed to [inactive/dead] Apr 18 02:06:47.723230 r-lionfish-16 DEBUG healthd: Main process- received event:snmp.timer from source:sysbus time:2023-04-17 23:06:47 Apr 18 02:06:47.723328 r-lionfish-16 INFO healthd: check_unit_status for [ snmp.timer ] Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>
mssonicbld
pushed a commit
that referenced
this pull request
May 18, 2023
…ob (#14702) Why I did it systemd stop event on service with timers can sometime delete the state_db entry for the corresponding service. Note: This won't be observed on the latest master label since the dependency on timer was removed with the recent config reload enhancement. However, it is better to have the fix since there might be some systemd services added to system health daemon in the future which may contain timers root@qa-eth-vt01-4-3700c:/home/admin# systemctl stop snmp root@qa-eth-vt01-4-3700c:/home/admin# show system-health sysready-status System is not ready - one or more services are not up Service-Name Service-Status App-Ready-Status Down-Reason ---------------------- ---------------- ------------------ ------------- <Truncated> ssh OK OK - swss OK OK - syncd OK OK - sysstat OK OK - teamd OK OK - telemetry OK OK - what-just-happened OK OK - ztp OK OK - <Truncated> Expected Should see a Down entry for SNMP instead of the entry being deleted from the STATE_DB root@qa-eth-vt01-4-3700c:/home/admin# show system-health sysready-status System is not ready - one or more services are not up Service-Name Service-Status App-Ready-Status Down-Reason ---------------------- ---------------- ------------------ ------------- <Truncated> snmp Down Down Inactive ssh OK OK - swss OK OK - syncd OK OK - sysstat OK OK - teamd OK OK - telemetry OK OK - what-just-happened OK OK - ztp OK OK - <Truncated> How I did it Happens because the timer is usually a PartOf service and thus a stop on service is propagated to timer. Fixed the logic to handle this Apr 18 02:06:47.711252 r-lionfish-16 DEBUG healthd: Main process- received event:snmp.service from source:sysbus time:2023-04-17 23:06:47 Apr 18 02:06:47.711347 r-lionfish-16 INFO healthd: check_unit_status for [ snmp.service ] Apr 18 02:06:47.722363 r-lionfish-16 INFO healthd: snmp.service service state changed to [inactive/dead] Apr 18 02:06:47.723230 r-lionfish-16 DEBUG healthd: Main process- received event:snmp.timer from source:sysbus time:2023-04-17 23:06:47 Apr 18 02:06:47.723328 r-lionfish-16 INFO healthd: check_unit_status for [ snmp.timer ] Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I did it
systemd stop event on service with timers can sometime delete the state_db entry for the corresponding service.
Note: This won't be observed on the latest master label since the dependency on timer was removed with the recent config reload enhancement. However, it is better to have the fix since there might be some systemd services added to system health daemon in the future which may contain timers
Expected
Should see a Down entry for SNMP instead of the entry being deleted from the STATE_DB
How I did it
Happens because the timer is usually a PartOf service and thus a stop on service is propagated to timer. Fixed the logic to handle this
How to verify it
Which release branch to backport (provide reason below if selected)
Tested branch (Please provide the tested image version)
Description for the changelog
Link to config_db schema for YANG module changes
A picture of a cute animal (not mandatory but encouraged)