Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SNMP request timeout when CPU utilization is high #21314

Closed
yejianquan opened this issue Jan 1, 2025 · 2 comments · Fixed by sonic-net/sonic-snmpagent#345 or #21316
Closed

SNMP request timeout when CPU utilization is high #21314

yejianquan opened this issue Jan 1, 2025 · 2 comments · Fixed by sonic-net/sonic-snmpagent#345 or #21316
Assignees
Labels
Chassis 🤖 Modular chassis support chassis-packet P0 Priority of the issue

Comments

@yejianquan
Copy link
Contributor

Description

SNMP request timeout when CPU utilization is high
Test module[snmp/test_snmp_cpu.py] in sonic-mgmt repo stably reproduce this issue.
Currently the issue is only seen on Cisco 8800 chassis.
Suspectation is on the RP card, including backend ports, there're too much interfaces.

Steps to reproduce the issue:

  1. Use 'yes' command to simulate 100% utilization on all CPU processors.
  2. Send snmp requests, the timeout generate been seen on interface snmp request.
  3. SNMP request time out and fail.

Describe the results you received:

After simulated 100% CPU utilization.
args.json contains mgmt-ip of the RP card.
jianquanye@sonic-mgmt-jianquanye:/var/src/sonic-mgmt-int/tests$ python ../ansible/library/snmp_facts.py args.json
Traceback (most recent call last):
File "../ansible/library/snmp_facts.py", line 1075, in
main()
File "../ansible/library/snmp_facts.py", line 496, in main
for oid, val in varBinds:
ValueError: too many values to unpack (expected 2)
jianquanye@sonic-mgmt-jianquanye:/var/src/sonic-mgmt-int/tests$

Describe the results you expected:

SNMP can successfully pass even with 100% CPU utilization

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

@yejianquan
Copy link
Contributor Author

sonic-net/sonic-snmpagent#345
#21316
These 2 PRs will fix the image issue, the SNMP request will be always success even if CPU utilization is 100%.

sonic-net/sonic-mgmt#16290
This PR enhances the test so that it can pass on Cisco chassis

@yejianquan
Copy link
Contributor Author

The short RCA is because MIB Updaters and SONiC SNMP AgentX share the same python Asyncio/Coroutine.
The AgentX can't respond to snmpd request during MIB Updaters updating.
On Cisco chassis RP, there are lots of interfaces(600+), so the update is slower compared with other platforms.
On 100% CPU utilization, the update is slower and the snmpd request timeout, becuase AgentX can't respond.

More details in PR sonic-net/sonic-snmpagent#345

@yxieca yxieca closed this as completed in 4317df5 Jan 8, 2025
mssonicbld added a commit to mssonicbld/sonic-buildimage that referenced this issue Jan 8, 2025
…n high CPU utilization scenario

<!--
     Please make sure you've read and understood our contributing guidelines:
     https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md

     ** Make sure all your commits include a signature generated with `git commit -s` **

     If this is a bug fix, make sure your description includes "fixes #xxxx", or
     "closes #xxxx" or "resolves #xxxx"

     Please provide the following information:
-->

#### Why I did it
Fix sonic-net#21314
Update and prolong the timeout of the requests between snmpd and SNMP AgentX.

In SONiC SNMP AgentX, the MIB updaters and AgentX client shares the same AsyncIO/Coroutine event loop.
During the MIB updaters update the SNMP values, the AgentX client can't respond to the snmpd request.

The default value of snmpd request is 1s(timeout) * 5(retries)

When the CPU is high, the MIB updaters are slow, 1s timeout is not enough, even if it retries 5 times.
Hence update to 5s(timeout) * 4(retries), the time windows = 20s, which makes sure the SNMP request can be handled even with 100% CPU utilization.

##### Work item tracking
- Microsoft ADO **30112399**:

#### How I did it
Update the default value(https://linux.die.net/man/5/snmpd.conf):

agentXTimeout 1(default value) -> 5
agentXRetries 5(default value) -> 4

#### How to verify it

<!--
If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012.
-->
Test on Cisco chassis, test_snmp_cpu.py which triggers 100% CPU utilization test whether snmp requests work well.

#### Which release branch to backport (provide reason below if selected)

<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->

- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
- [ ] 202111
- [ ] 202205
- [ ] 202211
- [ ] 202305
- [x] 202405

#### Tested branch (Please provide the tested image version)

<!--
- Please provide tested image version
- e.g.
- [x] 20201231.100
-->

- [x] 202405
- [ ] <!-- image version 2 -->

#### Description for the changelog
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->

<!--
 Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.
-->

#### Link to config_db schema for YANG module changes
<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md
-->

#### A picture of a cute animal (not mandatory but encouraged)
mssonicbld added a commit to mssonicbld/sonic-buildimage that referenced this issue Jan 8, 2025
…n high CPU utilization scenario

<!--
     Please make sure you've read and understood our contributing guidelines:
     https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md

     ** Make sure all your commits include a signature generated with `git commit -s` **

     If this is a bug fix, make sure your description includes "fixes #xxxx", or
     "closes #xxxx" or "resolves #xxxx"

     Please provide the following information:
-->

#### Why I did it
Fix sonic-net#21314
Update and prolong the timeout of the requests between snmpd and SNMP AgentX.

In SONiC SNMP AgentX, the MIB updaters and AgentX client shares the same AsyncIO/Coroutine event loop.
During the MIB updaters update the SNMP values, the AgentX client can't respond to the snmpd request.

The default value of snmpd request is 1s(timeout) * 5(retries)

When the CPU is high, the MIB updaters are slow, 1s timeout is not enough, even if it retries 5 times.
Hence update to 5s(timeout) * 4(retries), the time windows = 20s, which makes sure the SNMP request can be handled even with 100% CPU utilization.

##### Work item tracking
- Microsoft ADO **30112399**:

#### How I did it
Update the default value(https://linux.die.net/man/5/snmpd.conf):

agentXTimeout 1(default value) -> 5
agentXRetries 5(default value) -> 4

#### How to verify it

<!--
If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012.
-->
Test on Cisco chassis, test_snmp_cpu.py which triggers 100% CPU utilization test whether snmp requests work well.

#### Which release branch to backport (provide reason below if selected)

<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->

- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
- [ ] 202111
- [ ] 202205
- [ ] 202211
- [ ] 202305
- [x] 202405

#### Tested branch (Please provide the tested image version)

<!--
- Please provide tested image version
- e.g.
- [x] 20201231.100
-->

- [x] 202405
- [ ] <!-- image version 2 -->

#### Description for the changelog
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->

<!--
 Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.
-->

#### Link to config_db schema for YANG module changes
<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md
-->

#### A picture of a cute animal (not mandatory but encouraged)
mssonicbld added a commit that referenced this issue Jan 8, 2025
…n high CPU utilization scenario (#21350)

<!--
 Please make sure you've read and understood our contributing guidelines:
 https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md

 failure_prs.log Make sure all your commits include a signature generated with `git commit -s` **

 If this is a bug fix, make sure your description includes "fixes #xxxx", or
 "closes #xxxx" or "resolves #xxxx"

 Please provide the following information:
-->

#### Why I did it
Fix #21314
Update and prolong the timeout of the requests between snmpd and SNMP AgentX.

In SONiC SNMP AgentX, the MIB updaters and AgentX client shares the same AsyncIO/Coroutine event loop.
During the MIB updaters update the SNMP values, the AgentX client can't respond to the snmpd request.

The default value of snmpd request is 1s(timeout) failure_prs.log 5(retries)

When the CPU is high, the MIB updaters are slow, 1s timeout is not enough, even if it retries 5 times.
Hence update to 5s(timeout) failure_prs.log 4(retries), the time windows = 20s, which makes sure the SNMP request can be handled even with 100% CPU utilization.

##### Work item tracking
- Microsoft ADO **30112399**:

#### How I did it
Update the default value(https://linux.die.net/man/5/snmpd.conf):

agentXTimeout 1(default value) -> 5
agentXRetries 5(default value) -> 4

#### How to verify it

<!--
If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012.
-->
Test on Cisco chassis, test_snmp_cpu.py which triggers 100% CPU utilization test whether snmp requests work well.

#### Which release branch to backport (provide reason below if selected)

<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->

- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
- [ ] 202111
- [ ] 202205
- [ ] 202211
- [ ] 202305
- [x] 202405

#### Tested branch (Please provide the tested image version)

<!--
- Please provide tested image version
- e.g.
- [x] 20201231.100
-->

- [x] 202405
- [ ] <!-- image version 2 -->

#### Description for the changelog
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->

<!--
 Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.
-->

#### Link to config_db schema for YANG module changes
<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md
-->

#### A picture of a cute animal (not mandatory but encouraged)
mssonicbld added a commit that referenced this issue Jan 8, 2025
…n high CPU utilization scenario (#21349)

<!--
 Please make sure you've read and understood our contributing guidelines:
 https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md

 failure_prs.log Make sure all your commits include a signature generated with `git commit -s` **

 If this is a bug fix, make sure your description includes "fixes #xxxx", or
 "closes #xxxx" or "resolves #xxxx"

 Please provide the following information:
-->

#### Why I did it
Fix #21314
Update and prolong the timeout of the requests between snmpd and SNMP AgentX.

In SONiC SNMP AgentX, the MIB updaters and AgentX client shares the same AsyncIO/Coroutine event loop.
During the MIB updaters update the SNMP values, the AgentX client can't respond to the snmpd request.

The default value of snmpd request is 1s(timeout) failure_prs.log 5(retries)

When the CPU is high, the MIB updaters are slow, 1s timeout is not enough, even if it retries 5 times.
Hence update to 5s(timeout) failure_prs.log 4(retries), the time windows = 20s, which makes sure the SNMP request can be handled even with 100% CPU utilization.

##### Work item tracking
- Microsoft ADO **30112399**:

#### How I did it
Update the default value(https://linux.die.net/man/5/snmpd.conf):

agentXTimeout 1(default value) -> 5
agentXRetries 5(default value) -> 4

#### How to verify it

<!--
If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012.
-->
Test on Cisco chassis, test_snmp_cpu.py which triggers 100% CPU utilization test whether snmp requests work well.

#### Which release branch to backport (provide reason below if selected)

<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->

- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
- [ ] 202111
- [ ] 202205
- [ ] 202211
- [ ] 202305
- [x] 202405

#### Tested branch (Please provide the tested image version)

<!--
- Please provide tested image version
- e.g.
- [x] 20201231.100
-->

- [x] 202405
- [ ] <!-- image version 2 -->

#### Description for the changelog
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->

<!--
 Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.
-->

#### Link to config_db schema for YANG module changes
<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md
-->

#### A picture of a cute animal (not mandatory but encouraged)
yejianquan added a commit to sonic-net/sonic-snmpagent that referenced this issue Jan 14, 2025
Fix sonic-net/sonic-buildimage#21314
The SNMP agent and MIB updaters are basic on Asyncio/Coroutine,
the mib updaters share the same Asyncio event loop with the SNMP agent client.
Hence during the updaters executing, the agent client can't receive/respond to new requests.

When the CPU utilization is high (In some stress test we make CPU 100% utilization), the updates are slow, and this causes the snmpd request to be timeout because the agent got suspended during updating.

- What I did
Decrease the MIB update frequency when the update execution is slow.
pros:

The snmp request can success even if 100% CPU utilization.
The snmpd request seldomly fails due to timeout, combined with Update snmpd.conf.j2 prolong agentXTimeout to avoid timeout failure in high CPU utilization scenario sonic-buildimage#21316 , we have 4*5 = 20s time windows for the SNMP agent to wait for the MIB updates finish and respond to snmpd request.
Relief the CPU cost when CPU is high, the can avoid CPU becomes more crowded.
cons:

Tested on pizzabox (4600c), the updaters are very fast, generally finished within 0.001~0.02s, the chagne won't actually affect the frequency and interval.
On Cisco chassis, the update of SNMP data could be delayed for 10~20s(at most 60s in extreme situation). Per my oberservation, most of the updater finishes within 0.5s. But for
1.a ciscoSwitchQosMIB.QueueStatUpdater, generally finishs in 0.5-2s, expected to be delayed to 5-20s
1.b PhysicalTableMIBUpdater, generally finishs in 0.5-1.5s, expected to be delayed to 5-1.5s
1.c ciscoPfcExtMIB.PfcPrioUpdater, generally finishs in 0.5-3s, expected to be delayed to 5-30s
- How I did it
In get_next_update_interval, we compute the interval based on current execution time.
Roughly, we make the 'update interval'/'update execution time' >= UPDATE_FREQUENCY_RATE(10)
More specifically, if the execution time is 2.000001s, we sleep 21s before next update round.
And the max interval won't be longer than MAX_UPDATE_INTERVAL(60s).

- How to verify it
Test on Cisco chassis, test_snmp_cpu.py which triggers 100% CPU utilization test whether snmp requests work well.

co-authorized by: jianquanye@microsoft.com
mssonicbld added a commit to mssonicbld/sonic-snmpagent that referenced this issue Jan 14, 2025
<!--
Please make sure you've read and understood our contributing guidelines;
https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md

** Make sure all your commits include a signature generated with `git commit -s` **

If this is a bug fix, make sure your description includes "fixes #xxxx", or
"closes #xxxx"

Please provide the following information:
-->
Fix sonic-net/sonic-buildimage#21314
The SNMP agent and MIB updaters are basic on Asyncio/Coroutine,
  the mib updaters share the same Asyncio event loop with the SNMP agent client.
  Hence during the updaters executing, the agent client can't receive/respond to new requests.

When the CPU utilization is high (In some stress test we make CPU 100% utilization), the updates are slow, and this causes the snmpd request to be timeout because the agent got suspended during updating.

**- What I did**
Decrease the MIB update frequency when the update execution is slow.
pros:
   1.  The snmp request can success even if 100% CPU utilization.
   2. The snmpd request seldomly fails due to timeout, combined with sonic-net/sonic-buildimage#21316 , we have 4*5 = 20s time windows for the SNMP agent to wait for the MIB updates finish and respond to snmpd request.
   3. Relief the CPU cost when CPU is high, the can avoid CPU becomes more crowded.

cons:
   1. Tested on pizzabox (4600c), the updaters are very fast, generally finished within 0.001~0.02s, the chagne won't actually affect the frequency and interval.
   2. On Cisco chassis, the update of SNMP data could be delayed for 10~20s(at most 60s in extreme situation). Per my oberservation, most of the updater finishes within 0.5s. But for
         1.a ciscoSwitchQosMIB.QueueStatUpdater, generally finishs in 0.5-2s, expected to be delayed to 5-20s
         1.b PhysicalTableMIBUpdater, generally finishs in 0.5-1.5s, expected to be delayed to 5-1.5s
         1.c ciscoPfcExtMIB.PfcPrioUpdater, generally finishs in 0.5-3s, expected to be delayed to 5-30s

**- How I did it**
In get_next_update_interval, we compute the interval based on current execution time.
Roughly, we make the 'update interval'/'update execution time' >= UPDATE_FREQUENCY_RATE(10)
More specifically, if the execution time is 2.000001s, we sleep 21s before next update round.
And the max interval won't be longer than MAX_UPDATE_INTERVAL(60s).

**- How to verify it**
Test on Cisco chassis, test_snmp_cpu.py which triggers 100% CPU utilization test whether snmp requests work well.

**- Description for the changelog**
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->
mssonicbld added a commit to mssonicbld/sonic-snmpagent that referenced this issue Jan 14, 2025
<!--
Please make sure you've read and understood our contributing guidelines;
https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md

** Make sure all your commits include a signature generated with `git commit -s` **

If this is a bug fix, make sure your description includes "fixes #xxxx", or
"closes #xxxx"

Please provide the following information:
-->
Fix sonic-net/sonic-buildimage#21314
The SNMP agent and MIB updaters are basic on Asyncio/Coroutine,
  the mib updaters share the same Asyncio event loop with the SNMP agent client.
  Hence during the updaters executing, the agent client can't receive/respond to new requests.

When the CPU utilization is high (In some stress test we make CPU 100% utilization), the updates are slow, and this causes the snmpd request to be timeout because the agent got suspended during updating.

**- What I did**
Decrease the MIB update frequency when the update execution is slow.
pros:
   1.  The snmp request can success even if 100% CPU utilization.
   2. The snmpd request seldomly fails due to timeout, combined with sonic-net/sonic-buildimage#21316 , we have 4*5 = 20s time windows for the SNMP agent to wait for the MIB updates finish and respond to snmpd request.
   3. Relief the CPU cost when CPU is high, the can avoid CPU becomes more crowded.

cons:
   1. Tested on pizzabox (4600c), the updaters are very fast, generally finished within 0.001~0.02s, the chagne won't actually affect the frequency and interval.
   2. On Cisco chassis, the update of SNMP data could be delayed for 10~20s(at most 60s in extreme situation). Per my oberservation, most of the updater finishes within 0.5s. But for
         1.a ciscoSwitchQosMIB.QueueStatUpdater, generally finishs in 0.5-2s, expected to be delayed to 5-20s
         1.b PhysicalTableMIBUpdater, generally finishs in 0.5-1.5s, expected to be delayed to 5-1.5s
         1.c ciscoPfcExtMIB.PfcPrioUpdater, generally finishs in 0.5-3s, expected to be delayed to 5-30s

**- How I did it**
In get_next_update_interval, we compute the interval based on current execution time.
Roughly, we make the 'update interval'/'update execution time' >= UPDATE_FREQUENCY_RATE(10)
More specifically, if the execution time is 2.000001s, we sleep 21s before next update round.
And the max interval won't be longer than MAX_UPDATE_INTERVAL(60s).

**- How to verify it**
Test on Cisco chassis, test_snmp_cpu.py which triggers 100% CPU utilization test whether snmp requests work well.

**- Description for the changelog**
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->
mssonicbld added a commit to sonic-net/sonic-snmpagent that referenced this issue Jan 14, 2025
<!--
Please make sure you've read and understood our contributing guidelines;
https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md

** Make sure all your commits include a signature generated with `git commit -s` **

If this is a bug fix, make sure your description includes "fixes #xxxx", or
"closes #xxxx"

Please provide the following information:
-->
Fix sonic-net/sonic-buildimage#21314
The SNMP agent and MIB updaters are basic on Asyncio/Coroutine,
 the mib updaters share the same Asyncio event loop with the SNMP agent client.
 Hence during the updaters executing, the agent client can't receive/respond to new requests.

When the CPU utilization is high (In some stress test we make CPU 100% utilization), the updates are slow, and this causes the snmpd request to be timeout because the agent got suspended during updating.

**- What I did**
Decrease the MIB update frequency when the update execution is slow.
pros:
 1. The snmp request can success even if 100% CPU utilization.
 2. The snmpd request seldomly fails due to timeout, combined with sonic-net/sonic-buildimage#21316 , we have 4*5 = 20s time windows for the SNMP agent to wait for the MIB updates finish and respond to snmpd request.
 3. Relief the CPU cost when CPU is high, the can avoid CPU becomes more crowded.

cons:
 1. Tested on pizzabox (4600c), the updaters are very fast, generally finished within 0.001~0.02s, the chagne won't actually affect the frequency and interval.
 2. On Cisco chassis, the update of SNMP data could be delayed for 10~20s(at most 60s in extreme situation). Per my oberservation, most of the updater finishes within 0.5s. But for
 1.a ciscoSwitchQosMIB.QueueStatUpdater, generally finishs in 0.5-2s, expected to be delayed to 5-20s
 1.b PhysicalTableMIBUpdater, generally finishs in 0.5-1.5s, expected to be delayed to 5-1.5s
 1.c ciscoPfcExtMIB.PfcPrioUpdater, generally finishs in 0.5-3s, expected to be delayed to 5-30s

**- How I did it**
In get_next_update_interval, we compute the interval based on current execution time.
Roughly, we make the 'update interval'/'update execution time' >= UPDATE_FREQUENCY_RATE(10)
More specifically, if the execution time is 2.000001s, we sleep 21s before next update round.
And the max interval won't be longer than MAX_UPDATE_INTERVAL(60s).

**- How to verify it**
Test on Cisco chassis, test_snmp_cpu.py which triggers 100% CPU utilization test whether snmp requests work well.

**- Description for the changelog**
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->
mssonicbld added a commit to sonic-net/sonic-snmpagent that referenced this issue Jan 14, 2025
<!--
Please make sure you've read and understood our contributing guidelines;
https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md

** Make sure all your commits include a signature generated with `git commit -s` **

If this is a bug fix, make sure your description includes "fixes #xxxx", or
"closes #xxxx"

Please provide the following information:
-->
Fix sonic-net/sonic-buildimage#21314
The SNMP agent and MIB updaters are basic on Asyncio/Coroutine,
 the mib updaters share the same Asyncio event loop with the SNMP agent client.
 Hence during the updaters executing, the agent client can't receive/respond to new requests.

When the CPU utilization is high (In some stress test we make CPU 100% utilization), the updates are slow, and this causes the snmpd request to be timeout because the agent got suspended during updating.

**- What I did**
Decrease the MIB update frequency when the update execution is slow.
pros:
 1. The snmp request can success even if 100% CPU utilization.
 2. The snmpd request seldomly fails due to timeout, combined with sonic-net/sonic-buildimage#21316 , we have 4*5 = 20s time windows for the SNMP agent to wait for the MIB updates finish and respond to snmpd request.
 3. Relief the CPU cost when CPU is high, the can avoid CPU becomes more crowded.

cons:
 1. Tested on pizzabox (4600c), the updaters are very fast, generally finished within 0.001~0.02s, the chagne won't actually affect the frequency and interval.
 2. On Cisco chassis, the update of SNMP data could be delayed for 10~20s(at most 60s in extreme situation). Per my oberservation, most of the updater finishes within 0.5s. But for
 1.a ciscoSwitchQosMIB.QueueStatUpdater, generally finishs in 0.5-2s, expected to be delayed to 5-20s
 1.b PhysicalTableMIBUpdater, generally finishs in 0.5-1.5s, expected to be delayed to 5-1.5s
 1.c ciscoPfcExtMIB.PfcPrioUpdater, generally finishs in 0.5-3s, expected to be delayed to 5-30s

**- How I did it**
In get_next_update_interval, we compute the interval based on current execution time.
Roughly, we make the 'update interval'/'update execution time' >= UPDATE_FREQUENCY_RATE(10)
More specifically, if the execution time is 2.000001s, we sleep 21s before next update round.
And the max interval won't be longer than MAX_UPDATE_INTERVAL(60s).

**- How to verify it**
Test on Cisco chassis, test_snmp_cpu.py which triggers 100% CPU utilization test whether snmp requests work well.

**- Description for the changelog**
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->
yxieca pushed a commit that referenced this issue Jan 14, 2025
#21355)

Why I did it
It's one part of the fixes of #21314
SNMP walker request will always timeout when 100% CPU utilization.

Work item tracking
Microsoft ADO 30112399:

How I did it
Enable SNMP dynamic frequency on packet chassis.

How to verify it
snmp/test_snmp_cpu.py(https://github.com/sonic-net/sonic-mgmt/blob/master/tests/snmp/test_snmp_cpu.py) tests the scenario.
mssonicbld added a commit to mssonicbld/sonic-buildimage that referenced this issue Jan 14, 2025
<!--
     Please make sure you've read and understood our contributing guidelines:
     https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md

     ** Make sure all your commits include a signature generated with `git commit -s` **

     If this is a bug fix, make sure your description includes "fixes #xxxx", or
     "closes #xxxx" or "resolves #xxxx"

     Please provide the following information:
-->

#### Why I did it
It's one part of the fixes of sonic-net#21314
SNMP walker request will always timeout when 100% CPU utilization.
##### Work item tracking
- Microsoft ADO **30112399**:

#### How I did it
Enable SNMP dynamic frequency on packet chassis.
#### How to verify it
snmp/test_snmp_cpu.py(https://github.com/sonic-net/sonic-mgmt/blob/master/tests/snmp/test_snmp_cpu.py) tests the scenario.

And the test case passes in my local run.
<!--
If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012.
-->

#### Which release branch to backport (provide reason below if selected)

<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->

- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
- [ ] 202111
- [ ] 202205
- [ ] 202211
- [ ] 202305
- [x] 202405

#### Tested branch (Please provide the tested image version)

<!--
- Please provide tested image version
- e.g.
- [x] 20201231.100
-->

- [ ] <!-- image version 1 -->
- [ ] <!-- image version 2 -->

#### Description for the changelog
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->
For issue:
sonic-net#21314

Change of SNMP-AgentX sonic-net/sonic-snmpagent#345 will fix the issue.
But we want to limit the scope of the change, since we only see this issue on packet-chassis and old devices with low performance CPUs.

<!--
 Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.
-->

#### Link to config_db schema for YANG module changes
<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md
-->

#### A picture of a cute animal (not mandatory but encouraged)
mssonicbld added a commit to mssonicbld/sonic-buildimage that referenced this issue Jan 14, 2025
<!--
     Please make sure you've read and understood our contributing guidelines:
     https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md

     ** Make sure all your commits include a signature generated with `git commit -s` **

     If this is a bug fix, make sure your description includes "fixes #xxxx", or
     "closes #xxxx" or "resolves #xxxx"

     Please provide the following information:
-->

#### Why I did it
It's one part of the fixes of sonic-net#21314
SNMP walker request will always timeout when 100% CPU utilization.
##### Work item tracking
- Microsoft ADO **30112399**:

#### How I did it
Enable SNMP dynamic frequency on packet chassis.
#### How to verify it
snmp/test_snmp_cpu.py(https://github.com/sonic-net/sonic-mgmt/blob/master/tests/snmp/test_snmp_cpu.py) tests the scenario.

And the test case passes in my local run.
<!--
If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012.
-->

#### Which release branch to backport (provide reason below if selected)

<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->

- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
- [ ] 202111
- [ ] 202205
- [ ] 202211
- [ ] 202305
- [x] 202405

#### Tested branch (Please provide the tested image version)

<!--
- Please provide tested image version
- e.g.
- [x] 20201231.100
-->

- [ ] <!-- image version 1 -->
- [ ] <!-- image version 2 -->

#### Description for the changelog
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->
For issue:
sonic-net#21314

Change of SNMP-AgentX sonic-net/sonic-snmpagent#345 will fix the issue.
But we want to limit the scope of the change, since we only see this issue on packet-chassis and old devices with low performance CPUs.

<!--
 Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.
-->

#### Link to config_db schema for YANG module changes
<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md
-->

#### A picture of a cute animal (not mandatory but encouraged)
mssonicbld added a commit that referenced this issue Jan 14, 2025
#21430)

<!--
 Please make sure you've read and understood our contributing guidelines:
 https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md

 failure_prs.log Make sure all your commits include a signature generated with `git commit -s` **

 If this is a bug fix, make sure your description includes "fixes #xxxx", or
 "closes #xxxx" or "resolves #xxxx"

 Please provide the following information:
-->

#### Why I did it
It's one part of the fixes of #21314
SNMP walker request will always timeout when 100% CPU utilization.
##### Work item tracking
- Microsoft ADO **30112399**:

#### How I did it
Enable SNMP dynamic frequency on packet chassis.
#### How to verify it
snmp/test_snmp_cpu.py(https://github.com/sonic-net/sonic-mgmt/blob/master/tests/snmp/test_snmp_cpu.py) tests the scenario.

And the test case passes in my local run.
<!--
If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012.
-->

#### Which release branch to backport (provide reason below if selected)

<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->

- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
- [ ] 202111
- [ ] 202205
- [ ] 202211
- [ ] 202305
- [x] 202405

#### Tested branch (Please provide the tested image version)

<!--
- Please provide tested image version
- e.g.
- [x] 20201231.100
-->

- [ ] <!-- image version 1 -->
- [ ] <!-- image version 2 -->

#### Description for the changelog
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->
For issue:
#21314

Change of SNMP-AgentX sonic-net/sonic-snmpagent#345 will fix the issue.
But we want to limit the scope of the change, since we only see this issue on packet-chassis and old devices with low performance CPUs.

<!--
 Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.
-->

#### Link to config_db schema for YANG module changes
<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md
-->

#### A picture of a cute animal (not mandatory but encouraged)
mssonicbld added a commit that referenced this issue Jan 15, 2025
#21429)

<!--
 Please make sure you've read and understood our contributing guidelines:
 https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md

 failure_prs.log Make sure all your commits include a signature generated with `git commit -s` **

 If this is a bug fix, make sure your description includes "fixes #xxxx", or
 "closes #xxxx" or "resolves #xxxx"

 Please provide the following information:
-->

#### Why I did it
It's one part of the fixes of #21314
SNMP walker request will always timeout when 100% CPU utilization.
##### Work item tracking
- Microsoft ADO **30112399**:

#### How I did it
Enable SNMP dynamic frequency on packet chassis.
#### How to verify it
snmp/test_snmp_cpu.py(https://github.com/sonic-net/sonic-mgmt/blob/master/tests/snmp/test_snmp_cpu.py) tests the scenario.

And the test case passes in my local run.
<!--
If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012.
-->

#### Which release branch to backport (provide reason below if selected)

<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->

- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
- [ ] 202111
- [ ] 202205
- [ ] 202211
- [ ] 202305
- [x] 202405

#### Tested branch (Please provide the tested image version)

<!--
- Please provide tested image version
- e.g.
- [x] 20201231.100
-->

- [ ] <!-- image version 1 -->
- [ ] <!-- image version 2 -->

#### Description for the changelog
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->
For issue:
#21314

Change of SNMP-AgentX sonic-net/sonic-snmpagent#345 will fix the issue.
But we want to limit the scope of the change, since we only see this issue on packet-chassis and old devices with low performance CPUs.

<!--
 Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.
-->

#### Link to config_db schema for YANG module changes
<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md
-->

#### A picture of a cute animal (not mandatory but encouraged)
@rlhui rlhui moved this to Done in SONiC Chassis Jan 15, 2025
VladimirKuk pushed a commit to Marvell-switching/sonic-buildimage that referenced this issue Jan 21, 2025
…n high CPU utilization scenario (sonic-net#21316)

Why I did it
Fix sonic-net#21314
Update and prolong the timeout of the requests between snmpd and SNMP AgentX.

In SONiC SNMP AgentX, the MIB updaters and AgentX client shares the same AsyncIO/Coroutine event loop.
During the MIB updaters update the SNMP values, the AgentX client can't respond to the snmpd request.

The default value of snmpd request is 1s(timeout) * 5(retries)

When the CPU is high, the MIB updaters are slow, 1s timeout is not enough, even if it retries 5 times.
Hence update to 5s(timeout) * 4(retries), the time windows = 20s, which makes sure the SNMP request can be handled even with 100% CPU utilization.

Work item tracking
Microsoft ADO 30112399:

How I did it
Update the default value(https://linux.die.net/man/5/snmpd.conf):

agentXTimeout 1(default value) -> 5
agentXRetries 5(default value) -> 4

How to verify it
Test on Cisco chassis, test_snmp_cpu.py which triggers 100% CPU utilization test whether snmp requests work well.
VladimirKuk pushed a commit to Marvell-switching/sonic-buildimage that referenced this issue Jan 21, 2025
sonic-net#21355)

Why I did it
It's one part of the fixes of sonic-net#21314
SNMP walker request will always timeout when 100% CPU utilization.

Work item tracking
Microsoft ADO 30112399:

How I did it
Enable SNMP dynamic frequency on packet chassis.

How to verify it
snmp/test_snmp_cpu.py(https://github.com/sonic-net/sonic-mgmt/blob/master/tests/snmp/test_snmp_cpu.py) tests the scenario.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Chassis 🤖 Modular chassis support chassis-packet P0 Priority of the issue
Projects
Status: Done
1 participant