Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initialize the HNAS driver #767

Merged
merged 31 commits into from
Dec 24, 2021
Merged

Initialize the HNAS driver #767

merged 31 commits into from
Dec 24, 2021

Conversation

guankc
Copy link
Contributor

@guankc guankc commented Oct 25, 2021

Initialize the HNAS driver

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Special notes for your reviewer:

Release note:

@codecov
Copy link

codecov bot commented Oct 25, 2021

Codecov Report

Merging #767 (a1e22ae) into master (320b7fe) will increase coverage by 0.04%.
The diff coverage is 71.47%.

@@            Coverage Diff             @@
##           master     #767      +/-   ##
==========================================
+ Coverage   70.55%   70.60%   +0.04%     
==========================================
  Files         167      170       +3     
  Lines       16626    17235     +609     
  Branches     2387     2492     +105     
==========================================
+ Hits        11730    12168     +438     
- Misses       4192     4341     +149     
- Partials      704      726      +22     
Impacted Files Coverage Δ
delfin/drivers/netapp/dataontap/constants.py 100.00% <ø> (ø)
delfin/drivers/utils/ssh_client.py 16.81% <5.55%> (-2.07%) ⬇️
delfin/drivers/hitachi/hnas/nas_handler.py 72.87% <72.87%> (ø)
delfin/drivers/hitachi/hnas/hds_nas.py 86.04% <86.04%> (ø)
delfin/drivers/hitachi/hnas/constants.py 100.00% <100.00%> (ø)
delfin/drivers/utils/tools.py 92.53% <100.00%> (+4.47%) ⬆️
delfin/drivers/fake_storage/__init__.py 95.06% <0.00%> (ø)

header_index = i
return table[(header_index + 1):]

def get_storage(self):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function is bigger than 50 lines, please split it into multiple functions.
Please check other functions as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has modified

def ssh_do_exec(self, command_list):
res = ''
res = self.ssh_pool.do_exec_command(command_list)
while 'Failed to establish SSC connection' in res:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would happend if device always return 'Failed to establish SSC connection'

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SSH command line of the HNAS must be first accessed to the Linux server of the HNAS and then to the SSC subsystem of Linux. If the SSC subsystem cannot be accessed due to network fluctuation or other reasons, the SSH command fails and the system displays 'Failed to establish SSC connection'. In this case, run the command again to re-establish the connection with the SSC subsystem.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean if ssh always return this, it would be a dead loop

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are considering setting a retry count or timeout

pass

@staticmethod
def get_capabilities(context):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get_capabilities is missing argument filter, and if this driver cannot collect performance data, it's unnecessary to implement these two functions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has deleted

import time

import six

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This blank line is redundant.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has deleted


from delfin import exception, utils
from delfin.common import constants
from delfin.drivers.utils.ssh_client import SSHPool
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This import syntax is in a wrong order.

return int(Tools.get_capacity_size(limit))

def ssh_do_exec(self, command_list):
res = ''
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line46 is redundant.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has deleted

result = ''
try:
with self.item() as ssh:
if command_list is not None and len(command_list) > 0 \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about if command_list and ssh: xxx ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has been modify

and ssh is not None:
channel = ssh.invoke_shell()
for command in command_list:
utils.check_ssh_injection(command)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parameter of function check_ssh_injection is list type, so this line should be utils.check_ssh_injection(command.split()) , and same problem in line254, it should be utils.check_ssh_injection(command_str.split()) , please helpt to fix it in this pr.

if not resp:
break
result += resp
except paramiko.AuthenticationException as ae:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are too many duplicate codes in do_exec and do_exec_command, these two functions should be optimized.

@@ -278,3 +280,44 @@ def do_exec(self, command_str):
'is not a recognized command' in result:
raise exception.StorageBackendException(result)
return result

def do_exec_command(self, command_list):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Function name might be named do_exec_shell or do_exec_channel, or other names.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has been modify

def login(self):
try:
result = self.ssh_do_exec(['cluster-show -y'])
if 'is not a recognized command' in result \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This if syntax may be encountered in every command, so it should be in function self.ssh_pool.do_exec_command .

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has been modify

def ssh_do_exec(self, command_list):
res = ''
res = self.ssh_pool.do_exec_command(command_list)
while 'Failed to establish SSC connection' in res:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean if ssh always return this, it would be a dead loop

self.evs_list = []

@staticmethod
def get_size(limit, is_calculate=False):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the parameter of get_size is limit, should it be size?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function do the same thing as NetApp's get_size, so could put this part into Tools.get_capacity_size

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has deleted

raise e

@staticmethod
def split_value_map_list(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's this function for? Can not get the purpose of this function by the name and the implementation.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why use parameter map_list as return value instead of use return sentence

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has been modify

header_index = 0
table = values.split("\r\n")
for i in range(0, len(table)):
if constant.PATTERN.search(table[i]):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's this PATTERN for? Please use a meaningful name.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In order to search out the header of the data table and delete the part above the header

map_list.append(value_map)

@staticmethod
def get_table_data(values):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's this function for? values is a table? Which table?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After SSH executes the interface, it will return useless information other than the required data, such as login information, header and the execution command itself. We need to eliminate the invalid data. This is mainly the function

location_map = {}
serial_map = {}
if len(model_map_list) > 0:
model_map = model_map_list[-1]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If only the last item is needed, why just pass the last one as parameter instead of the whole list?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because there may be dirty data, only the last data object is needed

raise e

@staticmethod
def split_value_map_list(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why use parameter map_list as return value instead of use return sentence

disk_info, disk_map_list, 'Capacity', split=":")
disks_list = []
for disk_map in disk_map_list:
if 'Status' in disk_map:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this if sentence is needed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because parsed data may be dirty data with incomplete or useless information in the data object map, only if this key exists in the data object map can it be confirmed that it has completed reliable data, and then proceed to the next step of parsi

size_array = size_info.split("\r\n")
size_map = {}
pool_name = None
count = 0
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this count for? Seems not used.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has deleted

pool_name = None
count = 0
for size in size_array:
if 'Span ' in size:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why put size to 0 when has 'span' in size?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because of its size, there may be two data objects that need to be added up

free_array = size.split()
if len(free_array) > 2:
count += 1
free_size = free_array[0].replace('GiB', 'GB')
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GiB and GB should be different unit, please make sure whether we can do this that change GiB to GB directly

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can be converted to GB as determined on hnAS management interface

size_map = self.get_pool_size()
for pool in pool_array:
value_array = pool.split()
if len(value_array) == 6:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the 6 mean? Why only the length is 6 then we can add it to pool?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some useless information will be obtained through SSH.the information in each storage pool is larger than 6 columns, the useless information that is smaller than 6 columns is excluded to avoid dirty data

value_array = pool.split()
if len(value_array) == 6:
total_capacity = \
self.get_size(value_array[3] + "GB")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not use this magic number
Same as other places
https://stackoverflow.com/questions/47882/what-is-a-magic-number-and-why-is-it-bad

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has been modify

nodes_array = self.get_table_data(node_info)
for nodes in nodes_array:
node = nodes.split()
if len(node) > 2:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not use this magic number
Please investigate all the magic numbers and give it a meaningful name if needed.
Not all the magic number is not allowed, for example:

if len(xxx) > 0

this is allowed, because we can know what we want to judge whether xxx is empty, but for this:

if len(xxx) > 3

we can know nothing about what this for

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Numbers are now consolidated in constant files and given names to make them easier to read

else:
value_map[key] = value
else:
if value_map != {} and value_key in value_map:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about if value_key in value_map: xxx ? and also line94, line97.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has been modify

def get_table_data(values):
header_index = 0
table = values.split("\r\n")
for i in range(0, len(table)):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about for i in range(len(table)): xxx ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has been modify

def format_storage_info(self, storage_map_list,
model_map_list, version_map_list,
location_map_list, serial_map_list):
if len(storage_map_list) > 0:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about if not storage_map_list: raise StorageBackendException('xxx') ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has been modify

version_map = {}
location_map = {}
serial_map = {}
if len(model_map_list) > 0:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about model_map = model_map_list[-1] if model_map_list else {} ? and the following 3 lines are the same.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has been modify

location_map = location_map_list[-1]
if len(serial_map_list) > 0:
serial_map = serial_map_list[-1]
version = version_map.get("Software").split('(')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If version_map.get("Software") is None, None.split() will raise an exception, and next line is same.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has been modify

if 'Portname' in value_map:
status = value_map.get('Status', None)
health = constants.PortHealthStatus.ABNORMAL
if status and status == 'Good':
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about if status == 'Good': xxx ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has been modify

'name': 'FC' + port_id,
'storage_id': storage_id,
'native_port_id': port_id,
'location': None,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If info cannot be attained from storage, it can be deleted, and same as 7 lines below.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has been modify

'health_status': health,
'type': constants.PortType.FC,
'logical_type': None,
'speed': speed * (1000 ** 3),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use oslo_utils.units.G instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has been modify

for fs in fs_array:
fs_info = list(filter(None, fs.split(" ")))
if len(fs_info) > 8:
total_capacity = fs_info[3].replace(" ", '')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is recommended to use only one of " and ', and it isn't recommended to use " in one place and use ' in the other place.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has been modify

total_capacity = self.get_size(total_capacity)
used_capacity = self.get_size(used_capacity)
free_capacity = self.get_size(free_capacity)
type = constants.VolumeType.THICK \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

type is a built-in name, please use other variable name.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has been modify

@@ -0,0 +1,832 @@
# Copyright 2021 The SODA Authors.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just a constants file, no test in this file, so please change the file name from test_constans to constants

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has been modify

type = constants.VolumeType.THICK \
if fs_info[8] == 'No' \
else constants.VolumeType.THIN
pool_id = None \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about status, pool_id = status_map.get(fs_info[0]) if status_map.get(fs_info[0]) else (None, None) ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has been modify

'storage_id': storage_id,
'native_filesystem_id': fs_info[1],
'native_pool_id': pool_id,
'compressed': None,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If info cannot be attained from storage, it can be deleted, and same as the 3 lines below.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has been modify

fs_info = self.ssh_do_exec([constant.FS_STATUS_COMMAND])
fs_array = self.get_table_data(fs_info)
evs_list = []
for fs in fs_array:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about code as following, so we can know what's the meaning of number 0 and 4.

xxx_index = 0
xxx_index = 4
...
evs_list.append([fs_info_array[xxx_index], fs_info_array[xxx_index]])

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has been modify

quota_map_list = []
self.split_value_map_list(quota_info, quota_map_list, 'Usage')
for quota_map in quota_map_list:
type = None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

type is a built-in name, please use other variable name.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has been modify

type = None
user_group_name = None
qtree_id = None
if 'Group' in quota_map.get('Target'):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If quota_map.get('Target') is None, Group in None will raise an exception.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has been modify

if nfs.get('Exportpath') == qtree['path'] \
and qtree['native_filesystem_id'] \
== nfs.get('Filesystemlabel'):
qtree_id = qtree['native_qtree_id']
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here maybe need a break statement.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now there's something else that needs to be done, so can't break it

class Request:
def __init__(self):
self.environ = {'delfin.context': context.RequestContext()}
pass
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pass is redundant.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Has been modified

from delfin.drivers.utils.ssh_client import SSHPool


class Request:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't find where this class is used, please confirm it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Has been modified



class TestHitachiHNasDriver(TestCase):
SSHPool.get = mock.Mock(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The return type of SSHPool.get is not set, please confirm it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sshclient is returned, please confirm

SSHPool.get = mock.Mock(
return_value={paramiko.SSHClient()})

SSHPool.do_exec_command = mock.Mock(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every use case has SSHPool.do_exec_command = mock.Mock ..., I don't understand the meaning of this class variable, please confirm it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The purpose of the mock SSH is to call the interface and return the specified result

guankc and others added 14 commits November 9, 2021 11:13
Optimize code format specification and unit test specification
add storage degraded status
2. Delete the Ethernet logical port collection.

3. Change the collection mode of the hard upper limit for quota collection.

4. Optimize code based on test questions.

5. Refine unit tests based on modifications
2. Delete the Ethernet logical port collection.

3. Change the collection mode of the hard upper limit for quota collection.

4. Optimize code based on test questions.

5. Refine unit tests based on modifications
1. Optimize alarm paging and complete alarm location information.
if 'EVS' not in result:
raise exception.InvalidIpOrPort()
except Exception as e:
LOG.error("Failed to login netapp %s" %
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The log is incorrect. The device is HNAS, not NetApp.


CLUSTER_STATUS = {
'Robust': constants.StorageStatus.NORMAL,
'Degraded': constants.StorageStatus.ABNORMAL,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The device has a degraded state. The degraded state should not correspond to abnormal.

@joseph-v
Copy link
Collaborator

Please update setup.py to add entrypoint for the driver

Copy link
Collaborator

@joseph-v joseph-v left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Collaborator

@wisererik wisererik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wisererik wisererik merged commit b004d74 into sodafoundation:master Dec 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants