Optimizing testQosSaiHeadroomPoolSize runtime #8775

ansrajpu-git · 2023-06-29T18:38:03Z

Description of PR

HeadroomPoolSize was taking more that ~14 mins to complete the test which results in the port flapping while test run

The sleep time added inside the loop contribute a lot to the long running iterations.

Summary:
Below are the changes in 3 loop structure in HeadroomPoolSize test:
1.Loop to fill all pgs_num service pool: Instead of 8 sec sleep every time after send packet and show port. Sleep time is added after packets are sent to all the pgs_num and then added another loop for show_port_counter iterating pgs_num .

2.Loop iterating pgs_num for sending TCP packet: Instead of sleep time every iteration before show_port_counter; Sleep time of 8 sec is allotted for non broadcom-dnx box.

3.Loop to fill all pgs_num headroom pool: Instead of 8 sec sleep every time after send packet and run show port.Sleep time is added after packets are sent to all the pgs_num and added another loop for show_port_counter iterating pgs_num .
Fixes # (issue)

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Back port request

201911
202012
202205

Approach

What is the motivation for this PR?

HeadroomPoolsize test failing with ".....Caught exception socket.timeout: timeout(), , <class 'socket.timeout'>" ERROR in sai_thrift_port_tx_enable..........

Observed that each test takes ~14mins .since the port channel ports are kept tx disabled for that long duration which results in lacp packet drop.This is due to voq credits getting exhausted on the destination port , and port channel bounces

How did you do it?

Reduce the test execution runtime by removing redundant sleep time in the loop.

How did you verify/test it?

Executed qos test cases for T2 topology

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

mssonicbld · 2023-06-29T18:39:04Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/saitests/py3/sai_qos_tests.py:2303:21: E128 continuation line under-indented for visual indent tests/saitests/py3/sai_qos_tests.py:2303:121: E501 line too long (200 > 120 characters) tests/saitests/py3/sai_qos_tests.py:2345:21: E128 continuation line under-indented for visual indent tests/saitests/py3/sai_qos_tests.py:2345:121: E501 line too long (194 > 120 characters) tests/saitests/py3/sai_qos_tests.py:2395:121: E501 line too long (154 > 120 characters) tests/saitests/py3/sai_qos_tests.py:2397:21: E128 continuation line under-indented for visual indent tests/saitests/py3/sai_qos_tests.py:2397:121: E501 line too long (201 > 120 characters) check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-06-29T19:26:05Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/saitests/py3/sai_qos_tests.py:2303:21: E128 continuation line under-indented for visual indent tests/saitests/py3/sai_qos_tests.py:2303:121: E501 line too long (200 > 120 characters) tests/saitests/py3/sai_qos_tests.py:2345:21: E128 continuation line under-indented for visual indent tests/saitests/py3/sai_qos_tests.py:2345:121: E501 line too long (194 > 120 characters) tests/saitests/py3/sai_qos_tests.py:2395:121: E501 line too long (154 > 120 characters) tests/saitests/py3/sai_qos_tests.py:2397:21: E128 continuation line under-indented for visual indent tests/saitests/py3/sai_qos_tests.py:2397:121: E501 line too long (201 > 120 characters) check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-06-29T20:11:02Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/saitests/py3/sai_qos_tests.py:2303:21: E128 continuation line under-indented for visual indent tests/saitests/py3/sai_qos_tests.py:2346:21: E128 continuation line under-indented for visual indent tests/saitests/py3/sai_qos_tests.py:2400:21: E128 continuation line under-indented for visual indent check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-07-05T15:04:27Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/saitests/py3/sai_qos_tests.py:2303:121: E501 line too long (137 > 120 characters) tests/saitests/py3/sai_qos_tests.py:2304:40: E128 continuation line under-indented for visual indent tests/saitests/py3/sai_qos_tests.py:2346:121: E501 line too long (131 > 120 characters) tests/saitests/py3/sai_qos_tests.py:2347:40: E128 continuation line under-indented for visual indent tests/saitests/py3/sai_qos_tests.py:2400:121: E501 line too long (138 > 120 characters) tests/saitests/py3/sai_qos_tests.py:2401:40: E128 continuation line under-indented for visual indent check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-07-06T13:54:10Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/saitests/py3/sai_qos_tests.py:2303:121: E501 line too long (137 > 120 characters) tests/saitests/py3/sai_qos_tests.py:2304:40: E128 continuation line under-indented for visual indent tests/saitests/py3/sai_qos_tests.py:2346:121: E501 line too long (131 > 120 characters) tests/saitests/py3/sai_qos_tests.py:2347:40: E128 continuation line under-indented for visual indent tests/saitests/py3/sai_qos_tests.py:2400:121: E501 line too long (138 > 120 characters) tests/saitests/py3/sai_qos_tests.py:2401:40: E128 continuation line under-indented for visual indent check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-07-12T20:39:41Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/saitests/py3/sai_qos_tests.py:2305:40: E128 continuation line under-indented for visual indent tests/saitests/py3/sai_qos_tests.py:2349:40: E128 continuation line under-indented for visual indent tests/saitests/py3/sai_qos_tests.py:2404:40: E128 continuation line under-indented for visual indent check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-07-12T21:26:54Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/saitests/py3/sai_qos_tests.py:2306:21: E128 continuation line under-indented for visual indent tests/saitests/py3/sai_qos_tests.py:2351:40: E128 continuation line under-indented for visual indent tests/saitests/py3/sai_qos_tests.py:2406:40: E128 continuation line under-indented for visual indent check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-07-13T14:37:24Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/saitests/py3/sai_qos_tests.py:2305:36: E231 missing whitespace after ',' tests/saitests/py3/sai_qos_tests.py:2349:36: E231 missing whitespace after ',' tests/saitests/py3/sai_qos_tests.py:2352:73: E225 missing whitespace around operator tests/saitests/py3/sai_qos_tests.py:2403:36: E231 missing whitespace after ',' check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

judyjoseph · 2023-07-25T21:41:44Z

@vmittal-msft Could you check and review this PR ?

vmittal-msft · 2023-08-01T19:17:59Z

tests/saitests/py3/sai_qos_tests.py

@@ -2269,7 +2269,14 @@ def runTest(self):
                send_packet(
                    self, self.src_port_ids[sidx_dscp_pg_tuples[i][0]], pkt, int(pkt_cnt))

-                time.sleep(8)   # wait pfc counter refresh
+            time.sleep(8)  # wait pfc counter refresh and show the counters
+            for i in range(0, self.pgs_num):


Please modify for voq chassis only and no change for other platforms.

mssonicbld · 2023-08-07T19:12:36Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/saitests/py3/sai_qos_tests.py:2278:44: E128 continuation line under-indented for visual indent tests/saitests/py3/sai_qos_tests.py:2279:44: E122 continuation line missing indentation or outdented tests/saitests/py3/sai_qos_tests.py:2293:44: E128 continuation line under-indented for visual indent tests/saitests/py3/sai_qos_tests.py:2328:121: E501 line too long (121 > 120 characters) tests/saitests/py3/sai_qos_tests.py:2389:44: E128 continuation line under-indented for visual indent tests/saitests/py3/sai_qos_tests.py:2390:44: E122 continuation line missing indentation or outdented tests/saitests/py3/sai_qos_tests.py:2395:121: E501 line too long (158 > 120 characters) tests/saitests/py3/sai_qos_tests.py:2399:44: E128 continuation line under-indented for visual indent flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

(cherry picked from commit f35a45c9958716d5ed01f2134a3288ae0119ec48) (cherry picked from commit c815ae141e59e00fd4e9d83212aed184fe90c8a3)

judyjoseph · 2023-08-14T17:12:29Z

@ansrajpu-git Could you confirm the qos tests result with this change with latest master, We can merge it in if things are fine,

mssonicbld · 2023-08-28T09:22:59Z

@ansrajpu-git PR conflicts with 202205 branch

* Optimizing HeadroomPoolSize runtime

mssonicbld · 2023-08-28T09:23:05Z

Cherry-pick PR to 202305: #9704

* Optimizing HeadroomPoolSize runtime

…#8775) * Optimizing HeadroomPoolSize runtime

…2159) * Optimizing HeadroomPoolSize runtime (cherry picked from commit f35a45c9958716d5ed01f2134a3288ae0119ec48)

rlhui requested a review from vmittal-msft July 12, 2023 17:34

ansrajpu-git force-pushed the qos_HdrmPoolSize branch from 630edbc to 8192278 Compare July 12, 2023 20:38

ansrajpu-git force-pushed the qos_HdrmPoolSize branch 2 times, most recently from 974a6d1 to 6bdf682 Compare July 28, 2023 15:32

vmittal-msft reviewed Aug 1, 2023

View reviewed changes

ansrajpu-git force-pushed the qos_HdrmPoolSize branch from 6bdf682 to 3f996ce Compare August 2, 2023 18:46

ansrajpu-git force-pushed the qos_HdrmPoolSize branch from 09a6484 to 8ee93fb Compare August 7, 2023 20:12

vmittal-msft added chassis Request for 202205 branch labels Aug 8, 2023

vmittal-msft requested a review from judyjoseph August 8, 2023 21:22

ansrajpu-git added 10 commits August 9, 2023 12:19

Optimizing HeadroomPoolSize runtime

3323a19

(cherry picked from commit f35a45c9958716d5ed01f2134a3288ae0119ec48) (cherry picked from commit c815ae141e59e00fd4e9d83212aed184fe90c8a3)

Merge conflict change -reverted

7e5a23b

Fix flake8 issues

c50f206

Flake8 fix

59244c6

flake8 correction

54f5e1d

Flake8 error fix

3acfe34

flake8 error fix

69cdf4b

flake8 error fix

29c0075

PR8775 -adding check in HdrmPoolSize for chassis

7de9e7f

flake8 error corrected

d0dfe63

ansrajpu-git added 2 commits August 9, 2023 12:19

Merge conflict corrected

6f382b1

Change in check condition in sai_qos

fa17a19

ansrajpu-git force-pushed the qos_HdrmPoolSize branch from 2758fd2 to fa17a19 Compare August 9, 2023 16:22

vmittal-msft self-requested a review August 11, 2023 19:42

vmittal-msft approved these changes Aug 11, 2023

View reviewed changes

judyjoseph merged commit d674a2f into sonic-net:master Aug 17, 2023
11 checks passed

wangxin added Request for 202305 branch Approved for 202205 branch Approved for 202305 branch labels Aug 28, 2023

mssonicbld added the Cherry Pick Conflict_202205 label Aug 28, 2023

mssonicbld pushed a commit to mssonicbld/sonic-mgmt that referenced this pull request Aug 28, 2023

Optimizing testQosSaiHeadroomPoolSize runtime (sonic-net#8775)

4810a0c

* Optimizing HeadroomPoolSize runtime

mssonicbld added the Created PR to 202305 branch label Aug 28, 2023

mssonicbld mentioned this pull request Aug 28, 2023

[action] [PR:8775] Optimizing testQosSaiHeadroomPoolSize runtime #9704

Merged

6 tasks

mssonicbld pushed a commit that referenced this pull request Aug 29, 2023

Optimizing testQosSaiHeadroomPoolSize runtime (#8775)

a8eea02

* Optimizing HeadroomPoolSize runtime

mssonicbld added Included in 202305 branch and removed Created PR to 202305 branch labels Aug 29, 2023

AharonMalkin pushed a commit to AharonMalkin/sonic-mgmt that referenced this pull request Jan 25, 2024

Optimizing testQosSaiHeadroomPoolSize runtime (sonic-net#8775)

8ca4243

* Optimizing HeadroomPoolSize runtime

StormLiangMS mentioned this pull request Mar 14, 2024

QoSHdrmPoolsize _tx_disable_enable for all destination ports in lag #11008

Merged

7 tasks

ansrajpu-git added a commit to ansrajpu-git/sonic-mgmt that referenced this pull request Mar 25, 2024

[Cherry-pick]Optimizing testQosSaiHeadroomPoolSize runtime (sonic-net…

26bdd06

…#8775) * Optimizing HeadroomPoolSize runtime

ansrajpu-git mentioned this pull request Mar 25, 2024

[Cherry-pick] Optimizing testQosSaiHeadroomPoolSize runtime #8775 #12159

Merged

8 tasks

yxieca pushed a commit that referenced this pull request May 15, 2024

[Cherry-pick] Optimizing testQosSaiHeadroomPoolSize runtime #8775 (#1…

9d72265

…2159) * Optimizing HeadroomPoolSize runtime (cherry picked from commit f35a45c9958716d5ed01f2134a3288ae0119ec48)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimizing testQosSaiHeadroomPoolSize runtime #8775

Optimizing testQosSaiHeadroomPoolSize runtime #8775

ansrajpu-git commented Jun 29, 2023 •

edited

Loading

mssonicbld commented Jun 29, 2023

mssonicbld commented Jun 29, 2023

mssonicbld commented Jun 29, 2023

mssonicbld commented Jul 5, 2023

mssonicbld commented Jul 6, 2023

mssonicbld commented Jul 12, 2023

mssonicbld commented Jul 12, 2023

mssonicbld commented Jul 13, 2023

judyjoseph commented Jul 25, 2023

vmittal-msft Aug 1, 2023

mssonicbld commented Aug 7, 2023

judyjoseph commented Aug 14, 2023

mssonicbld commented Aug 28, 2023

mssonicbld commented Aug 28, 2023

Optimizing testQosSaiHeadroomPoolSize runtime #8775

Optimizing testQosSaiHeadroomPoolSize runtime #8775

Conversation

ansrajpu-git commented Jun 29, 2023 • edited Loading

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

mssonicbld commented Jun 29, 2023

mssonicbld commented Jun 29, 2023

mssonicbld commented Jun 29, 2023

mssonicbld commented Jul 5, 2023

mssonicbld commented Jul 6, 2023

mssonicbld commented Jul 12, 2023

mssonicbld commented Jul 12, 2023

mssonicbld commented Jul 13, 2023

judyjoseph commented Jul 25, 2023

vmittal-msft Aug 1, 2023

Choose a reason for hiding this comment

mssonicbld commented Aug 7, 2023

judyjoseph commented Aug 14, 2023

mssonicbld commented Aug 28, 2023

mssonicbld commented Aug 28, 2023

ansrajpu-git commented Jun 29, 2023 •

edited

Loading