Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SAI-PTF]Precheck or handle the error on unimplemented interface/attribute #1610

Closed
richardyu-ms opened this issue Sep 26, 2022 · 3 comments
Closed

[SAI-PTF]Precheck or handle the error on unimplemented interface/attribute #1610

richardyu-ms opened this issue Sep 26, 2022 · 3 comments

Comments

@richardyu-ms
Copy link
Collaborator

richardyu-ms commented Sep 26, 2022 •
edited
Loading

Summary

Some interfaces are not implemented on some platform, then the test cases might failed in different test process, like setup, teardown and runtest method.
But for the unimplemented interface, they should be distinguished or they should not be failed but skipped.

Details

For example, for the test cases https://github.com/opencomputeproject/SAI/blob/master/ptf/saifdb.py#L36 and https://github.com/opencomputeproject/SAI/blob/master/ptf/saifdb.py#L1737.

        sai_thrift_flush_fdb_entries(
            self.client,
            bv_id=self.vlan10,
            entry_type=SAI_FDB_FLUSH_ENTRY_TYPE_STATIC)


        sai_thrift_create_fdb_entry(self.client,
                                    self.fdb_entry,
                                    type=SAI_FDB_ENTRY_TYPE_STATIC,
                                    bridge_port_id=self.port0_bp)

Those methods are not implemented in some platforms.

When we run it, the test cases will be failed.

Error message

======================================================================
ERROR [31.780s]: saifdb.FdbAttributeTest
----------------------------------------------------------------------
Traceback (most recent call last):
  File "ptf/saifdb.py", line 56, in runTest
    self.assertEqual(self.status(), SAI_STATUS_SUCCESS)
AssertionError: -2 != 0

Proposal

Need to skip the test when found any unimplemented interface in test.

Sample code or Pull requst for the proposal

#1609
@richardyu-ms richardyu-ms mentioned this issue Sep 26, 2022
Merged
@richardyu-ms
Copy link
Collaborator Author

@ravi861 could you help check this issue

@richardyu-ms richardyu-ms changed the title [SAI-PTF]Need to skip the test on unimplemented interface/attribute [SAI-PTF]Precheck or handle the error on unimplemented interface/attribute Oct 16, 2022
@richardyu-ms richardyu-ms mentioned this issue Oct 16, 2022
Open
@richardyu-ms
Copy link
Collaborator Author

Merged a PR for partially resolved this issue.
#1609
And cherry-picked to v1.7
#1625

To-Dos
As comments in #1609
"The correct approach is by verifying if the attribute is supported using sai attribute query capability."
Use a more elegant way to check the attribute and APIs.
Then the items could be

  1. implement the method in SAI-PTF for those SAI attr/api capability check API definitaions(SAI enabled, but SAI-PTF still not)
  2. refactor the decorator, different version might need different approach

richardyu-ms added a commit to richardyu-ms/SAI that referenced this issue Dec 24, 2022
@richardyu-ms
[SAI-PTF] Skip test when hit expected error from sai api
dfc7bf4 
Why
When hit the expected error like SAI_STATUS_NOT_SUPPORTED = -2, then skip the test.
related to issue opencomputeproject#1610

How
Add conditional exeption handler when return from sai api invocation

Test:
Test on brcm platform on case sainexthop.tunnelVrfTest

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>
richardyu-ms added a commit to richardyu-ms/SAI that referenced this issue Dec 24, 2022
@richardyu-ms
[SAI-PTF] Skip test when hit expected error from sai api
7e21180 
Why
When hit the expected error like SAI_STATUS_NOT_SUPPORTED = -2, then skip the test.
related to issue opencomputeproject#1610

How
Add conditional exeption handler when return from sai api invocation

Test:
Test on brcm platform on case sainexthop.tunnelVrfTest

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>
richardyu-ms added a commit to richardyu-ms/SAI that referenced this issue Dec 24, 2022
@richardyu-ms
[SAI-PTF] Skip test when hit expected error from sai api
ae3eb7f 
Why
When hit the expected error like SAI_STATUS_NOT_SUPPORTED = -2, then skip the test.
related to issue opencomputeproject#1610

How
Add conditional exeption handler when return from sai api invocation

Test:
Test on brcm platform on case sainexthop.tunnelVrfTest

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>
richardyu-ms added a commit to richardyu-ms/SAI that referenced this issue Dec 24, 2022
@richardyu-ms
[SAI-PTF] Skip test when hit expected error from sai api
eb83c2d 
Why
When hit the expected error like SAI_STATUS_NOT_SUPPORTED = -2, then skip the test.
related to issue opencomputeproject#1610

How
Add conditional exeption handler when return from sai api invocation

Test:
Test on brcm platform on case sainexthop.tunnelVrfTest

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>
This was referenced Dec 24, 2022
Merged
Merged
Merged
@richardyu-ms
Copy link
Collaborator Author

#1697
#1698
#1699

richardyu-ms added a commit that referenced this issue Dec 24, 2022
@richardyu-ms
[SAI-PTF] Skip test when hit expected error from sai api (#1699)
0fd23d2 
Why
When hit the expected error like SAI_STATUS_NOT_SUPPORTED = -2, then skip the test.
related to issue #1610

How
Add conditional exeption handler when return from sai api invocation

Test:
Test on brcm platform on case sainexthop.tunnelVrfTest

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>
richardyu-ms added a commit that referenced this issue Dec 24, 2022
@richardyu-ms
[SAI-PTF] Skip test when hit expected error from sai api (#1698)
a17d258 
Why
When hit the expected error like SAI_STATUS_NOT_SUPPORTED = -2, then skip the test.
related to issue #1610

How
Add conditional exeption handler when return from sai api invocation

Test:
Test on brcm platform on case sainexthop.tunnelVrfTest

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>
richardyu-ms added a commit that referenced this issue Dec 24, 2022
@richardyu-ms
[SAI-PTF] Skip test when hit expected error from sai api (#1697)
1d36e6d 
Why
When hit the expected error like SAI_STATUS_NOT_SUPPORTED = -2, then skip the test.
related to issue #1610

How
Add conditional exeption handler when return from sai api invocation

Test:
Test on brcm platform on case sainexthop.tunnelVrfTest

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>
richardyu-ms added a commit to richardyu-ms/SAI that referenced this issue Dec 26, 2022
@richardyu-ms
[SAI-PTF] Skip test when hit expected error from sai api (opencompute…
ecf19fa 
…project#1699)

Why
When hit the expected error like SAI_STATUS_NOT_SUPPORTED = -2, then skip the test.
related to issue opencomputeproject#1610

How
Add conditional exeption handler when return from sai api invocation

Test:
Test on brcm platform on case sainexthop.tunnelVrfTest

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>
@richardyu-ms richardyu-ms mentioned this issue Jan 3, 2023
Merged
richardyu-ms added a commit that referenced this issue Jan 3, 2023
@richardyu-ms
[master][SAI-PTF] Skip test when hit expected error from sai api (#1699…
e9e0c9b 
…) (#1701)

* [SAI-PTF]Add return value in the SAI-PTF log

Why
In order to track the SAI-API result and add the return value for each SAI_thrift api.

How
In sai_adapter, use the invocation_logger, log the return value when sai_thrift API returned

Test:
Unit test and DUT test

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* [SAI-PTF] API Logger - reformat arg values (#1696)

Why
Base on some requirement, when logging the arg values, need return the value for each key as a string.

For example,  returning this.
```
sai_adapter_invoke func:[sai_thrift_create_route_entry] args: [{'client': <sai_thrift.sai_rpc.Client object at 0x7f9e07154438>, 'route_entry': sai_thrift_route_entry_t(switch_id=None, vr_id=12884901888, destination=sai_thrift_ip_prefix_t(addr_family=1, addr=sai_thrift_ip_addr_t(ip4=None, ip6='0000:0000:0000:0000:0000:0000:0000:0000'), mask=sai_thrift_ip_addr_t(ip4=None, ip6='0000:0000:0000:0000:0000:0000:0000:0000'))), 'packet_action': 0}]
```
 Turn into
 ```
sai_adapter_invoke func:[sai_thrift_create_route_entry] args: [{'client': '<sai_thrift.sai_rpc.Client object at 0x7f9e07154438>', 'route_entry': 'sai_thrift_route_entry_t(switch_id=None, vr_id=12884901888, destination=sai_thrift_ip_prefix_t(addr_family=1, addr=sai_thrift_ip_addr_t(ip4=None, ip6='0000:0000:0000:0000:0000:0000:0000:0000'), mask=sai_thrift_ip_addr_t(ip4=None, ip6='0000:0000:0000:0000:0000:0000:0000:0000')))', 'packet_action': '0'}]
```

How
Convert the dict value to string

Test:
Unit test and DUT test

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

* [SAI-PTF] Skip test when hit expected error from sai api (#1699)

Why
When hit the expected error like SAI_STATUS_NOT_SUPPORTED = -2, then skip the test.
related to issue #1610

How
Add conditional exeption handler when return from sai api invocation

Test:
Test on brcm platform on case sainexthop.tunnelVrfTest

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>

Signed-off-by: richardyu-ms <richard.yu@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@richardyu-ms