[Auto Techsupport] Event driven Techsupport Changes #8670

vivekrnv · 2021-09-03T00:38:29Z

Why I did it

Changes required for feature "Event Driven TechSupport Invocation & CoreDump Mgmt". HLD

Requires: sonic-net/sonic-utilities#1796.
Merging in any order would be fine.

Summary of the changes:

Added the YANG Models for the new tables introduces as a part of this feature.
Enhanced init_cfg.json with the default config required
Added a compile Time flag which enables/disables the config required for this feature inside the init_cfg.json
Enhanced the supervisor-proc-exit-listener script to populate <feature>:<critical_proc> = <comm>:<pid> info in the STATE_DB when it observes an proc exit notification for the critical processes running inside the docker.

How I did it

How to verify it

##### No Core files and ts dumps initially
admin@sonic:~$ ls /var/core/
admin@sonic:~$ ls /var/dump

#### verify auto-techsupport status

admin@sonic:~$ show auto-techsupport global
STATE      RATE LIMIT INTERVAL    MAX TECHSUPPORT SIZE    MAX CORE SIZE  SINCE
-------  ---------------------  ----------------------  ---------------  ----------
enabled                    180                      10                5  2 days ago

admin@sonic:~$ show auto-techsupport-feature 
FEATURE NAME    STATE       RATE LIMIT INTERVAL
--------------  --------  ---------------------
bgp             enabled                     600
database        enabled                     600
dhcp_relay      enabled                     600
lldp            enabled                     600
macsec          enabled                     600
mgmt-framework  enabled                     600
nat             enabled                     600
pmon            enabled                     600
radv            enabled                     600
restapi         disabled                    800
sflow           enabled                     600
snmp            enabled                     600
swss            disabled                    800
syncd           enabled                     600
teamd           enabled                     600
telemetry       enabled                     600

#### Kill a critical Process t and trigger a coredump
admin@sonic:~$ docker exec -it snmp ps -aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root          23  1.6  0.5 138316 42296 pts/0    Sl   19:09   0:15 python3 -m sonic_ax_impl
...............

admin@sonic:~$ docker exec -it snmp kill -11 23

#### Coredump is created
admin@sonic:~$ ls /var/core/
python3.1629401152.23.core.gz 

#### Techsupport Dump creation in progress
admin@sonic:~$ ls /var/dump/
sonic_dump_sonic_20210819_192558  sonic_dump_sonic_20210819_192558.tar

admin@sonic:~$ ps -aux | grep coredump_gen
root       17823  0.1  0.2  30960 16736 ?        S    19:25   0:00 python3 /usr/local/bin/coredump_gen_handler.py python3.1629401152.23.core.gz

#### Wait until the techsupport dump execution has finished
admin@sonic:~$ ls /var/dump/
sonic_dump_sonic_20210819_192558.tar.gz


admin@sonic:~$ show auto-techsupport history

TECHSUPPORT DUMP                          TRIGGERED BY    CORE DUMP
----------------------------------------  --------------  -----------------------------
sonic_dump_sonic_device_20210819_192558  snmp-subagent   python3.1629401152.23.core.gz

This feature required changes to supervisor-proc-exit-listener script:
Changes made are backward compatible with python2. Tested this on docker running python3 & python2 (restapi)

admin@sonic:~$ docker exec -it snmp ps -aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.5  0.3  35676 24316 pts/0    Ss+  21:43   0:00 /usr/bin/pyth
root           9  0.1  0.3  43608 24888 pts/0    S    21:43   0:00 python3 /usr/
root          17  0.0  0.0 225856  5616 pts/0    Sl   21:43   0:00 /usr/sbin/rsy
Debian-+      21  0.9  0.1  32932 12524 pts/0    S    21:43   0:00 /usr/sbin/snm
root          23  8.4  0.4 133116 36764 pts/0    Sl   21:43   0:06 python3 -m so
root          26  0.0  0.0  11248  3036 pts/1    Rs+  21:44   0:00 ps -aux
admin@r-lionfish-16:~$ docker exec -it restapi ps -aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.5  0.2  59112 21552 pts/0    Ss+  21:43   0:00 /usr/bin/pyth
root           9  0.1  0.2  84932 22932 pts/0    S    21:43   0:00 python2 /usr/
root          14  0.0  0.0 262988  3596 pts/0    Sl   21:43   0:00 /usr/sbin/rsy
root          20  0.0  0.0  17964  2948 pts/0    S    21:43   0:00 bash /usr/bin
root          31  0.0  0.0   4188   660 pts/0    S    21:44   0:00 sleep 60
root          32  0.0  0.0  36636  2840 pts/1    Rs+  21:44   0:00 ps -aux
admin@sonic:~$ docker exec -it restapi kill -11 20
admin@sonic:~$ docker exec -it snmp kill -11 23
admin@sonic:~$ redis-cli -n 6 hgetall "AUTO_TECHSUPPORT|FEATURE_PROC_INFO"
1) "restapi;restapi"
2) "20;bash"
3) "snmp;snmp-subagent"
4) "23;python3"

Which release branch to backport (provide reason below if selected)

Description for the changelog

A picture of a cute animal (not mandatory but encouraged)

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

…to event_driven_ts

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

…to event_driven_ts

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

…to event_driven_ts

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

…ic-buildimage into event_driven_ts

azure-pipelines · 2021-09-03T18:54:57Z

Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

vivekrnv · 2021-09-28T22:52:21Z

/azpw run

mssonicbld · 2021-09-28T22:52:23Z

/AzurePipelines run

azure-pipelines · 2021-09-28T22:52:42Z

Azure Pipelines successfully started running 1 pipeline(s).

vivekrnv · 2021-10-11T15:23:21Z

/azpw run

mssonicbld · 2021-10-11T15:23:22Z

/AzurePipelines run

azure-pipelines · 2021-10-11T15:23:44Z

Azure Pipelines successfully started running 1 pipeline(s).

vivekrnv · 2021-10-18T03:41:57Z

/azpw run

mssonicbld · 2021-10-18T03:41:58Z

/AzurePipelines run

azure-pipelines · 2021-10-18T03:42:16Z

Azure Pipelines successfully started running 1 pipeline(s).

…to event_driven_ts

vivekrnv · 2021-11-05T18:20:45Z

/azpw run

mssonicbld · 2021-11-05T18:20:46Z

/AzurePipelines run

azure-pipelines · 2021-11-05T18:21:06Z

Azure Pipelines successfully started running 1 pipeline(s).

ganglyu · 2021-11-09T02:20:34Z

src/sonic-yang-models/yang-models/sonic-auto_techsupport.yang

+                        When the limit is crossed, the older core files are incrementally deleted
+                        */
+                        description "Max Limit in percentage for the cummulative size of ts dumps. No cleanup is performed if the value isn't configured or is 0.0";
+                        type decimal-repr;


value is between (0, 100) here. But decimal-repr range is from 0.0 to 99.99, can you clarify?

It was intended to be b/w 0.0 to 99.99. As for the comment (A value between (0,100) should be specified, Upto two decimal places will be used in the calculation) , I've used open brackets to signify the value is between 0 & 100. and the next comment suggests that upto two decimal places are used. So, i think this is clear. Although, let me know, if i have to re-write comments to make them clear

Open brackets does not include 0 and 100? Please update comment.

Updated the comment

ganglyu · 2021-11-09T02:22:39Z

src/sonic-yang-models/yang-models/sonic-auto_techsupport.yang

+                        When the limit is crossed, the older core files are deleted
+                        */
+                        description "Max Limit in percentage for the cummulative size of core dumps. No cleanup is performed if the value isn't congiured or is 0.0";
+                        type decimal-repr;


range 0.0..99.99?

commented above

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

vivekrnv · 2021-11-10T00:05:04Z

/azpw run

mssonicbld · 2021-11-10T00:05:05Z

/AzurePipelines run

azure-pipelines · 2021-11-10T00:05:10Z

You have several pipelines (over 10) configured to build pull requests in this repository. Specify which pipelines you would like to run by using /azp run [pipelines] command. You can specify multiple pipelines using a comma separated list.

vivekrnv and others added 29 commits August 10, 2021 22:05

AUTO_TECHUPPORT yang model added

436f4ec

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

auto TS changes

c88529f

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Flag added to slave.mk

ba435fb

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

auto_ts model updated

9c68bc2

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Merge branch 'master' of https://github.com/Azure/sonic-buildimage in…

995034b

…to event_driven_ts

init_cfg is modified

acca347

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Merge branch 'master' of https://github.com/Azure/sonic-buildimage in…

09dd71a

…to event_driven_ts

YANG model edited

f2754d6

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

init_cfg edit

9128b6e

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

init_cfg corrected

91a1468

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

tests modified

ea0966e

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Made the Compile time flag interactive

3531d47

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

exit-listener updated

2359bb8

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Yang model updated

b375827

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Table Name Change for proc_exit_event

750afc9

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Yang Updated for comments

8324fc3

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Merge Conflict Resolved

7479a1f

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

init_cfg and YANG updated

e10bf07

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Script updated for backward compatibility

ae855e0

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Description tag added to YANG

301fb48

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Merge branch 'master' of https://github.com/Azure/sonic-buildimage in…

9b29fca

…to event_driven_ts

Indentation changes and minor edits

fc7b979

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

YANG model updated

6dd95b8

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

init_cfg edited

d0c70ed

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

init_cfg updated

c822b76

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

YANG Models Updated

8bc38f9

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Merge branch 'master' into event_driven_ts

661065c

Updated the YANG model

1f957c9

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Merge branch 'event_driven_ts' of https://github.com/vivekreddynv/son…

2b79d52

…ic-buildimage into event_driven_ts

vivekrnv requested a review from lguohan as a code owner September 3, 2021 00:38

vivekrnv added 3 commits September 14, 2021 01:05

YANG updated

798b6b5

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Supervisor Proc exit listener dependency removed

eaf28a2

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Updated description

ee95b7a

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

qiluo-msft requested a review from ganglyu September 21, 2021 08:31

ganglyu previously approved these changes Oct 17, 2021

View reviewed changes

Merge branch 'master' into event_driven_ts

f19eb34

vivekrnv dismissed ganglyu’s stale review via f19eb34 October 17, 2021 08:05

Merge branch 'master' of https://github.com/Azure/sonic-buildimage in…

c32e68f

…to event_driven_ts

ganglyu reviewed Nov 9, 2021

View reviewed changes

Comment Updated

559a5e2

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

ganglyu approved these changes Nov 10, 2021

View reviewed changes

Merge branch 'Azure:master' into event_driven_ts

27a674b

qiluo-msft merged commit ff32ac3 into sonic-net:master Nov 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Auto Techsupport] Event driven Techsupport Changes #8670

[Auto Techsupport] Event driven Techsupport Changes #8670

vivekrnv commented Sep 3, 2021 •

edited

Loading

azure-pipelines bot commented Sep 3, 2021

vivekrnv commented Sep 28, 2021

mssonicbld commented Sep 28, 2021

azure-pipelines bot commented Sep 28, 2021

vivekrnv commented Oct 11, 2021

mssonicbld commented Oct 11, 2021

azure-pipelines bot commented Oct 11, 2021

vivekrnv commented Oct 18, 2021

mssonicbld commented Oct 18, 2021

azure-pipelines bot commented Oct 18, 2021

vivekrnv commented Nov 5, 2021

mssonicbld commented Nov 5, 2021

azure-pipelines bot commented Nov 5, 2021

ganglyu Nov 9, 2021

vivekrnv Nov 9, 2021

ganglyu Nov 9, 2021

vivekrnv Nov 9, 2021

ganglyu Nov 9, 2021

vivekrnv Nov 9, 2021

vivekrnv commented Nov 10, 2021

mssonicbld commented Nov 10, 2021

azure-pipelines bot commented Nov 10, 2021

[Auto Techsupport] Event driven Techsupport Changes #8670

[Auto Techsupport] Event driven Techsupport Changes #8670

Conversation

vivekrnv commented Sep 3, 2021 • edited Loading

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

A picture of a cute animal (not mandatory but encouraged)

azure-pipelines bot commented Sep 3, 2021

vivekrnv commented Sep 28, 2021

mssonicbld commented Sep 28, 2021

azure-pipelines bot commented Sep 28, 2021

vivekrnv commented Oct 11, 2021

mssonicbld commented Oct 11, 2021

azure-pipelines bot commented Oct 11, 2021

vivekrnv commented Oct 18, 2021

mssonicbld commented Oct 18, 2021

azure-pipelines bot commented Oct 18, 2021

vivekrnv commented Nov 5, 2021

mssonicbld commented Nov 5, 2021

azure-pipelines bot commented Nov 5, 2021

ganglyu Nov 9, 2021

Choose a reason for hiding this comment

vivekrnv Nov 9, 2021

Choose a reason for hiding this comment

ganglyu Nov 9, 2021

Choose a reason for hiding this comment

vivekrnv Nov 9, 2021

Choose a reason for hiding this comment

ganglyu Nov 9, 2021

Choose a reason for hiding this comment

vivekrnv Nov 9, 2021

Choose a reason for hiding this comment

vivekrnv commented Nov 10, 2021

mssonicbld commented Nov 10, 2021

azure-pipelines bot commented Nov 10, 2021

vivekrnv commented Sep 3, 2021 •

edited

Loading