Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Multiasic]: Provide namespace support for ipNetToMediaPhysAddress #129

Merged
merged 15 commits into from
Jul 28, 2020

Conversation

SuvarnaMeenakshi
Copy link
Contributor

@SuvarnaMeenakshi SuvarnaMeenakshi commented May 1, 2020

MIB in rfc1213.

Signed-off-by: SuvarnaMeenakshi sumeenak@microsoft.com

- What I did
Provide namespace support for ipNetToMediaPhysAddress MIB.

- How I did it
In multi-asic platform, retrieve host arp information from kernel using python_arptable.
Retrieve namespace arp information from NEIGH_TABLE in APP_DB.
For single-asic platform - Implementation modified to read NEIGH_TABLE in APP_DB.

- How to verify it
Tested on single-asic platform to make sure the output is the same as before the changes were added.
Tested on multi-asic platform.

- Description for the changelog

MIB in rfc1213.

Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com>
Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com>

def update_data(self):
self.arp_dest_map = {}
self.arp_dest_list = []
Copy link
Contributor

@qiluo-msft qiluo-msft May 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need if you immediately assign them. #Closed

dest_map, dest_list = self.get_arp_table()
self.arp_dest_map.update(dest_map)
self.arp_dest_list.extend(dest_list)
# Revert back to original namespace
Copy link
Contributor

@qiluo-msft qiluo-msft May 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revert back to original namespace [](start = 16, length = 35)

Could you revert back only at the end of loop? #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Modified implementation to use NEIGH_TABLE instead of using linux namespace.

for ns in self.ns_list:
dest_map = {}
dest_list = []
ns_fd = open(mibs.get_netns_path(nsname = ns))
Copy link
Contributor

@qiluo-msft qiluo-msft May 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ns_fd = open(mibs.get_netns_path(nsname = ns)) [](start = 16, length = 46)

use with statement, no need to close explicitly #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vModified implementation to use NEIGH_TABLE instead of using linux namespace.

self.arp_dest_map.update(dest_map)
self.arp_dest_list.extend(dest_list)
# Revert back to original namespace
mibs.setns(current_fd, mibs.CLONE_NEWNET)
Copy link
Contributor

@qiluo-msft qiluo-msft May 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mibs.setns(current_fd, mibs.CLONE_NEWNET) [](start = 16, length = 41)

Use a finally to make sure it is executed even there is exception? #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Modified implementation to use NEIGH_TABLE instead of using linux namespace.

@@ -8,6 +10,10 @@
from ax_interface.util import oid2tuple
from sonic_ax_impl import logger

libc = cdll.LoadLibrary('libc.so.6')
Copy link
Contributor

@qiluo-msft qiluo-msft May 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

libc [](start = 0, length = 4)

handle exception or error? #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed, not used anymore.


redis_kwargs = {'unix_socket_path': '/var/run/redis/redis.sock'}

def setns(fd, nstype):
Copy link
Contributor

@qiluo-msft qiluo-msft May 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

setns [](start = 4, length = 5)

These new features has nothing to do with mibs, and it may be reused outside this repo? Could you move into a common repo? Better to encapsulate a class. #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed this.


redis_kwargs = {'unix_socket_path': '/var/run/redis/redis.sock'}

def setns(fd, nstype):
if hasattr(fd, 'fileno'):
fd = fd.fileno()
Copy link
Contributor

@qiluo-msft qiluo-msft May 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why we need the flexibility? #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed this function, not using this.

_setns(fd, nstype)

def get_netns_path(nspath=None, nsname=None, nspid=None):
if nsname:
Copy link
Contributor

@qiluo-msft qiluo-msft May 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nsname [](start = 7, length = 6)

check it is a string. #Closed


def get_netns_path(nspath=None, nsname=None, nspid=None):
if nsname:
if (os.path.isdir(NET_NS_PATH) == False):
Copy link
Contributor

@qiluo-msft qiluo-msft May 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

os.path.isdir(NET_NS_PATH) [](start = 12, length = 26)

We only need to check this path once? #Closed

fd = fd.fileno()
_setns(fd, nstype)

def get_netns_path(nspath=None, nsname=None, nspid=None):
Copy link
Contributor

@qiluo-msft qiluo-msft May 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nspath=None [](start = 19, length = 11)

Can you remove it from arg list? #Closed

fd = fd.fileno()
_setns(fd, nstype)

def get_netns_path(nspath=None, nsname=None, nspid=None):
Copy link
Contributor

@qiluo-msft qiluo-msft May 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nsname=None, nspid=None [](start = 32, length = 23)

Better to separate into 2 functions. #Closed


def get_netns_path(nspath=None, nsname=None, nspid=None):
if nsname:
if (os.path.isdir(NET_NS_PATH) == False):
Copy link
Contributor

@qiluo-msft qiluo-msft May 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (os.path.isdir(NET_NS_PATH) == False):
if not os.path.isdir(NET_NS_PATH):
``` #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed this function, not used.

if ns_list is None:
return None
else:
return ns_list
Copy link
Contributor

@qiluo-msft qiluo-msft May 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just return ns_list #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed this, not used.

nspath = '/proc/%d/ns/net' % nspid
return nspath

def get_namespace_list():
Copy link
Contributor

@qiluo-msft qiluo-msft May 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get_namespace_list [](start = 4, length = 18)

Add some unit tests #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added unit-tests

arp_dest_list.append(subid)
return arp_dest_map, arp_dest_list

def update_data(self):
Copy link
Contributor

@qiluo-msft qiluo-msft May 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update_data [](start = 8, length = 11)

Add some unit tests #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added unit-tests

Copy link
Contributor

@qiluo-msft qiluo-msft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As comments

@lguohan
Copy link
Contributor

lguohan commented May 21, 2020

why not use the neighbor table information in the app db?

Copy link
Contributor

@lguohan lguohan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please switch to use neighbor table in app db.

@SuvarnaMeenakshi
Copy link
Contributor Author

please switch to use neighbor table in app db.

sure, will update the changes.

Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com>
Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com>
namespaces. Add function to retrieve data from NEIGH_TABLE in
APP_DB of each namespace.

Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com>
Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com>
@@ -62,6 +62,13 @@

redis_kwargs = {'unix_socket_path': '/var/run/redis/redis.sock'}


def get_neigh_info(neigh_key):
neigh_key = neigh_key[neigh_key.find(':')+1:len(neigh_key)]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

':' [](start = 41, length = 3)

Instead of hard-coded, could you get it by get_separator(). But only call this function minimal times.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed this, to only split neigh_key information of IPv4 neigh address.

def reinit_data(self):
if len(self.db_conn) > 1:
Namespace.connect_all_dbs(self.db_conn, mibs.APPL_DB)
self.neigh_key_list = Namespace.dbs_keys_namespace(self.db_conn, mibs.APPL_DB, "NEIGH_TABLE:*")
Copy link
Contributor

@qiluo-msft qiluo-msft Jul 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dbs_keys_namespace [](start = 44, length = 18)

Function does not exist. #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requires PR#137

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added function as PR #137 is not merged yet.

neigh_str = neigh_key.decode()
db_index = self.neigh_key_list[neigh_key]
dev, ip = mibs.get_neigh_info(neigh_str)
"""
Copy link
Contributor

@qiluo-msft qiluo-msft Jul 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

""" [](start = 12, length = 3)

Docstrings are for all public modules, functions, classes, and methods. Here '# ' will be better. #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

"""
if dev == "eth0":
continue
neigh_info = self.db_conn[db_index].get_all(mibs.APPL_DB, neigh_key, blocking=True)
Copy link
Contributor

@qiluo-msft qiluo-msft Jul 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get_all [](start = 48, length = 7)

Did you call get_all too many times? #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for all NEIGH_TABLE keys, we have to call get_all

@@ -22,6 +22,7 @@
class TestSonicMIB(TestCase):
@classmethod
def setUpClass(cls):
tests.mock_tables.python_arptable.arp_filename = '/arp.txt'
Copy link
Contributor

@qiluo-msft qiluo-msft Jul 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

arp_filename [](start = 42, length = 12)

Not used? #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is used in mock python_arptable.
For single namespace, we will use arp.txt - this will have mgmt interfaces and front panel interfaces data.
For multiple namespace, we use host_arp.txt - this have only mgmt interface related data.
Rest of namespace information will be retrieved from mock appl_db.json.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is to test single ASIC, and we don't need to mock kernel ARP any more?


In reply to: 451193110 [](ancestors = 451193110)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we don't need this, fixed.

@@ -0,0 +1,46 @@
IP address HW type Flags HW address Mask Device
Copy link
Contributor

@qiluo-msft qiluo-msft Jul 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IP address [](start = 0, length = 10)

Delete original tests/mock_tables/arp.txt file? #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

@@ -86,5 +86,17 @@
"INTF_TABLE:PortChannel01:10.10.0.1/31": {
"scope": "global",
"family": "IPv4"
},
"NEIGH_TABLE:Ethernet0:10.0.0.0": {
Copy link
Contributor

@qiluo-msft qiluo-msft Jul 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

10.0.0.0 [](start = 25, length = 8)

Add some mock data for IPv6
Add some mock data for PortChannel
Will new mock data need more test cases? #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added data for IPv6 and Port Channel. Added one more test case to make sure that eth0 of namespace, which is a part of docker0 bridge, is not used, for multi-asic platform.

# for updater in self.lut.updater_instances:
# updater.update_data()
# updater.reinit_data()
# updater.update_data()
Copy link
Contributor

@qiluo-msft qiluo-msft Jul 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it working?
Or remove it? #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not there in the latest file.
d728e97

from kernel and namespace arp table from NEIGH_TABLE in APP_DB
in each namespace.
"""
self._update_host_data()
Copy link
Contributor

@qiluo-msft qiluo-msft Jul 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_update_host_data [](start = 13, length = 17)

For single ASIC, we should treat 'NEIGH_TABLE' as the ground truth. No need to check arp_table.
And we need to adjust test for single ASIC. #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to read 'NEIGH_TABLE' for single asic as well. Updated mock db to reflect this change.

Changes made so that for single asic platform, arp table
information is retrieved from NEIGH_TABLE.

Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com>
mock arp.txt file as data will be retrieved from APPL_DB.

Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com>
namespace or docker's eth0 information is not retrieved.

Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com>
Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com>
as information is retrieved from NEIGH_TABLE.

Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com>
for neigh_key in self.neigh_key_list:
neigh_str = neigh_key.decode()
db_index = self.neigh_key_list[neigh_key]
neigh_info = self.db_conn[db_index].get_all(mibs.APPL_DB, neigh_key, blocking=True)
Copy link
Contributor

@qiluo-msft qiluo-msft Jul 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocking=True [](start = 81, length = 13)

Let's use non blocking, which make snmpagent robust even if there is an issue in any namespace, such as process crash, DB corruption. #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated as per review comment.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non blocking may return a None (please verify), then you need to handle this corner case.


In reply to: 458323712 [](ancestors = 458323712)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added check, will continue with other keys if None is returned on get_all on any key.

to make sure snmpagent is robust even if there is issue
in any one namespace.

Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com>
for entry in python_arptable.get_arp_table():
dev = entry['Device']
mac = entry['HW address']
ip = entry['IP address']
self._update_arp_info(dev, mac, ip)

def _update_namespace_data(self):
Copy link
Contributor

@qiluo-msft qiluo-msft Jul 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_update_namespace_data [](start = 8, length = 22)

_update_from_db #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

Namespace.connect_all_dbs(self.db_conn, mibs.APPL_DB)
self.neigh_key_list = Namespace.dbs_keys_namespace(self.db_conn, mibs.APPL_DB, "NEIGH_TABLE:*")

def _update_host_data(self):
Copy link
Contributor

@qiluo-msft qiluo-msft Jul 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_update_host_data [](start = 8, length = 17)

_update_from_arptable #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com>
@lgtm-com
Copy link

lgtm-com bot commented Jul 21, 2020

This pull request introduces 1 alert when merging 70633d6 into c702a57 - view on LGTM.com

new alerts:

  • 1 for Testing equality to None

Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com>
@SuvarnaMeenakshi SuvarnaMeenakshi requested a review from lguohan July 26, 2020 23:13
@qiluo-msft
Copy link
Contributor

For single-asic platform - not change.

I think this description is not up to date.

@SuvarnaMeenakshi
Copy link
Contributor Author

For single-asic platform - not change.

I think this description is not up to date.

Updated description.

@SuvarnaMeenakshi SuvarnaMeenakshi merged commit d5f2b92 into sonic-net:master Jul 28, 2020
abdosi pushed a commit that referenced this pull request Sep 2, 2020
)

* [Multiasic]: Provide namespace support for ipNetToMediaPhysAddress
MIB in rfc1213.

Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com>

* [Namespace][ArpUpdater]: Update ArpUpdater to support multiple
namespaces. Add function to retrieve data from NEIGH_TABLE in
APP_DB of each namespace.

Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com>

* Updated as per review comments.
Changes made so that for single asic platform, arp table
information is retrieved from NEIGH_TABLE.

Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants