Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce sonic-py-common package #5003

Merged
merged 25 commits into from
Jul 27, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
5f74856
Introduce sonic-py-common package
jleveque Jul 18, 2020
ac83ce3
Install sonic_py_common in PMon container
jleveque Jul 19, 2020
55a1d79
Add sonic-py-common as dependency of mlnx-platform-api
jleveque Jul 19, 2020
862e786
Add sonic-py-common as dependency of sonic-thermalctld
jleveque Jul 19, 2020
284c0db
Fix LGTM alerts
jleveque Jul 19, 2020
1de1385
Update docstring for get_asic_conf_file_path()
jleveque Jul 19, 2020
97fef08
Update comment
jleveque Jul 19, 2020
bedbc1a
Docstring style
jleveque Jul 19, 2020
f49c4da
Remove BGP cache file that snuck in to a commit
jleveque Jul 20, 2020
d11f1a6
Add TODO comment above moving multi-ASIC functionality to its own module
jleveque Jul 20, 2020
40a450b
Revert changes to celestica helper.py file; Will save for a separate PR
jleveque Jul 20, 2020
d65f683
Add ability to control minimum log priority, set default to NOTICE
jleveque Jul 22, 2020
4288a8b
Fix LGTM alert
jleveque Jul 22, 2020
0b488e2
Add convenience functions for setting logger minimum priority
jleveque Jul 22, 2020
1dec66a
Modify imports in daemon_base.py to simplifys usage of Logger
jleveque Jul 22, 2020
f8c19a2
Remove extra blank lines pre PEP8, add comments as visual separators …
jleveque Jul 22, 2020
50e7b49
Build Python 3 verison of package, install in host OS and SNMP container
jleveque Jul 22, 2020
c1e84de
Install Python 3 package using pip3
jleveque Jul 22, 2020
a3d280e
Make exception statements Python3-compliant
jleveque Jul 23, 2020
f2d0fb0
Add natsort as dependency
jleveque Jul 23, 2020
f1a541f
Separate standard imports from third-party imports with a blank line
jleveque Jul 23, 2020
59f56d1
Add pyyaml as dependency
jleveque Jul 23, 2020
461a56c
Add 'LOG_' prefix to facilities and priorities
jleveque Jul 23, 2020
95419cd
Install python3-setuptools in build_debian.sh
jleveque Jul 24, 2020
3365c7b
Add missing 'LOG_' prefix
jleveque Jul 25, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,16 @@ installer/x86_64/platforms/
src/sonic-config-engine/**/*.pyc
src/sonic-config-engine/build
src/sonic-config-engine/sonic_config_engine.egg-info

src/sonic-daemon-base/**/*.pyc
src/sonic-daemon-base/build
src/sonic-daemon-base/sonic_daemon_base.egg-info

src/sonic-py-common/**/*.pyc
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

**/*.pyc [](start = 20, length = 8)

pyc should always be ignored, so suggest consolidate them into one line.

Copy link
Contributor Author

@jleveque jleveque Jul 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this makes sense in a separate PR designed for general cleanup purposes, as it touches more than this new package. Please open an issue to track it.

src/sonic-py-common/build
src/sonic-py-common/dist
src/sonic-py-common/sonic_py_common.egg-info
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

egg-info [](start = 36, length = 8)

Same to *.egg-info/

Copy link
Contributor Author

@jleveque jleveque Jul 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this makes sense in a separate PR designed for general cleanup purposes, as it touches more than this new package. Please open an issue to track it.


# Misc. files
asic_config_checksum
files/Aboot/boot0
Expand Down
1 change: 1 addition & 0 deletions build_debian.sh
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,7 @@ sudo LANG=C DEBIAN_FRONTEND=noninteractive chroot $FILESYSTEM_ROOT apt-get -y in
openssh-server \
python \
python-setuptools \
python3-setuptools \
python-apt \
traceroute \
iputils-ping \
Expand Down
12 changes: 12 additions & 0 deletions files/build_templates/sonic_debian_extension.j2
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,18 @@ sudo cp {{daemon_base_py2_wheel_path}} $FILESYSTEM_ROOT/$DAEMON_BASE_PY2_WHEEL_N
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip install $DAEMON_BASE_PY2_WHEEL_NAME
sudo rm -rf $FILESYSTEM_ROOT/$DAEMON_BASE_PY2_WHEEL_NAME

# Install sonic-py-common Python 2 package
SONIC_PY_COMMON_PY2_WHEEL_NAME=$(basename {{sonic_py_common_py2_wheel_path}})
sudo cp {{sonic_py_common_py2_wheel_path}} $FILESYSTEM_ROOT/$SONIC_PY_COMMON_PY2_WHEEL_NAME
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip install $SONIC_PY_COMMON_PY2_WHEEL_NAME
sudo rm -rf $FILESYSTEM_ROOT/$SONIC_PY_COMMON_PY2_WHEEL_NAME

# Install sonic-py-common Python 3 package
SONIC_PY_COMMON_PY3_WHEEL_NAME=$(basename {{sonic_py_common_py3_wheel_path}})
sudo cp {{sonic_py_common_py3_wheel_path}} $FILESYSTEM_ROOT/$SONIC_PY_COMMON_PY3_WHEEL_NAME
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install $SONIC_PY_COMMON_PY3_WHEEL_NAME
sudo rm -rf $FILESYSTEM_ROOT/$SONIC_PY_COMMON_PY3_WHEEL_NAME

# Install built Python Click package (and its dependencies via 'apt-get -y install -f')
# Do this before installing sonic-utilities so that it doesn't attempt to install
# an older version as part of its dependencies
Expand Down
2 changes: 1 addition & 1 deletion platform/mellanox/mlnx-platform-api.mk
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
SONIC_PLATFORM_API_PY2 = mlnx_platform_api-1.0-py2-none-any.whl
$(SONIC_PLATFORM_API_PY2)_SRC_PATH = $(PLATFORM_PATH)/mlnx-platform-api
$(SONIC_PLATFORM_API_PY2)_PYTHON_VERSION = 2
$(SONIC_PLATFORM_API_PY2)_DEPENDS = $(SONIC_PLATFORM_COMMON_PY2) $(SONIC_DAEMON_BASE_PY2) $(SONIC_CONFIG_ENGINE)
$(SONIC_PLATFORM_API_PY2)_DEPENDS = $(SONIC_PY_COMMON_PY2) $(SONIC_PLATFORM_COMMON_PY2) $(SONIC_DAEMON_BASE_PY2) $(SONIC_CONFIG_ENGINE)
SONIC_PYTHON_WHEELS += $(SONIC_PLATFORM_API_PY2)

export mlnx_platform_api_py2_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_PLATFORM_API_PY2))"
1 change: 1 addition & 0 deletions rules/docker-platform-monitor.mk
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ endif
$(DOCKER_PLATFORM_MONITOR)_PYTHON_DEBS += $(SONIC_LEDD) $(SONIC_XCVRD) $(SONIC_PSUD) $(SONIC_SYSEEPROMD) $(SONIC_THERMALCTLD)
$(DOCKER_PLATFORM_MONITOR)_PYTHON_WHEELS += $(SONIC_PLATFORM_COMMON_PY2)
$(DOCKER_PLATFORM_MONITOR)_PYTHON_WHEELS += $(SWSSSDK_PY2)
$(DOCKER_PLATFORM_MONITOR)_PYTHON_WHEELS += $(SONIC_PY_COMMON_PY2)
$(DOCKER_PLATFORM_MONITOR)_PYTHON_WHEELS += $(SONIC_PLATFORM_API_PY2)
$(DOCKER_PLATFORM_MONITOR)_PYTHON_WHEELS += $(SONIC_DAEMON_BASE_PY2)

Expand Down
2 changes: 1 addition & 1 deletion rules/docker-snmp.mk
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ $(DOCKER_SNMP)_DBG_DEPENDS += $(SNMP_DBG) $(SNMPD_DBG) $(LIBSNMP_DBG)

$(DOCKER_SNMP)_DBG_IMAGE_PACKAGES = $($(DOCKER_CONFIG_ENGINE_BUSTER)_DBG_IMAGE_PACKAGES)

$(DOCKER_SNMP)_PYTHON_WHEELS += $(SONIC_PLATFORM_COMMON_PY3) $(SWSSSDK_PY3) $(ASYNCSNMP_PY3)
$(DOCKER_SNMP)_PYTHON_WHEELS += $(SONIC_PY_COMMON_PY3) $(SONIC_PLATFORM_COMMON_PY3) $(SWSSSDK_PY3) $(ASYNCSNMP_PY3)
$(DOCKER_SNMP)_LOAD_DOCKERS += $(DOCKER_CONFIG_ENGINE_BUSTER)

SONIC_DOCKER_IMAGES += $(DOCKER_SNMP)
Expand Down
12 changes: 12 additions & 0 deletions rules/sonic-py-common.dep
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
SPATH := $($(SONIC_PY_COMMON_PY2)_SRC_PATH)
DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/sonic-py-common.mk rules/sonic-py-common.dep
DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST)
DEP_FILES += $(shell git ls-files $(SPATH))

$(SONIC_PY_COMMON_PY2)_CACHE_MODE := GIT_CONTENT_SHA
$(SONIC_PY_COMMON_PY2)_DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST)
$(SONIC_PY_COMMON_PY2)_DEP_FILES := $(DEP_FILES)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xumia Could you help review?


$(SONIC_PY_COMMON_PY3)_CACHE_MODE := GIT_CONTENT_SHA
$(SONIC_PY_COMMON_PY3)_DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST)
$(SONIC_PY_COMMON_PY3)_DEP_FILES := $(DEP_FILES)
15 changes: 15 additions & 0 deletions rules/sonic-py-common.mk
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# SONIC_PY_COMMON_PY2 package

SONIC_PY_COMMON_PY2 = sonic_py_common-1.0-py2-none-any.whl
$(SONIC_PY_COMMON_PY2)_SRC_PATH = $(SRC_PATH)/sonic-py-common
$(SONIC_PY_COMMON_PY2)_DEPENDS += $(SWSSSDK_PY2)
$(SONIC_PY_COMMON_PY2)_PYTHON_VERSION = 2
SONIC_PYTHON_WHEELS += $(SONIC_PY_COMMON_PY2)

# SONIC_PY_COMMON_PY3 package

SONIC_PY_COMMON_PY3 = sonic_py_common-1.0-py3-none-any.whl
$(SONIC_PY_COMMON_PY3)_SRC_PATH = $(SRC_PATH)/sonic-py-common
$(SONIC_PY_COMMON_PY3)_DEPENDS += $(SWSSSDK_PY3)
$(SONIC_PY_COMMON_PY3)_PYTHON_VERSION = 3
SONIC_PYTHON_WHEELS += $(SONIC_PY_COMMON_PY3)
2 changes: 1 addition & 1 deletion rules/sonic-thermalctld.mk
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@

SONIC_THERMALCTLD = python-sonic-thermalctld_1.0-1_all.deb
$(SONIC_THERMALCTLD)_SRC_PATH = $(SRC_PATH)/sonic-platform-daemons/sonic-thermalctld
$(SONIC_THERMALCTLD)_WHEEL_DEPENDS = $(SONIC_DAEMON_BASE_PY2)
$(SONIC_THERMALCTLD)_WHEEL_DEPENDS = $(SONIC_PY_COMMON_PY2) $(SONIC_DAEMON_BASE_PY2)
Copy link
Collaborator

@qiluo-msft qiluo-msft Jul 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$(SONIC_PY_COMMON_PY2) $(SONIC_DAEMON_BASE_PY2) [](start = 37, length = 47)

I see some code duplication in both repo. Can we merge 2 into 1? #Pending

Copy link
Contributor Author

@jleveque jleveque Jul 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's the ultimate goal. After this merges, I intend to shift all code away from sonic-daemon-base and sonic-device-util, and remove them completely in separate PRs. Please read the "Next Steps" in my description above and see the linked issue: #4999

SONIC_PYTHON_STDEB_DEBS += $(SONIC_THERMALCTLD)
4 changes: 4 additions & 0 deletions slave.mk
Original file line number Diff line number Diff line change
Expand Up @@ -805,6 +805,8 @@ $(addprefix $(TARGET_PATH)/, $(SONIC_INSTALLERS)) : $(TARGET_PATH)/% : \
$$(addprefix $(FILES_PATH)/,$$($$*_FILES)) \
$(if $(findstring y,$(ENABLE_ZTP)),$(addprefix $(IMAGE_DISTRO_DEBS_PATH)/,$(SONIC_ZTP))) \
$(addprefix $(PYTHON_DEBS_PATH)/,$(SONIC_UTILS)) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_PY_COMMON_PY2)) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_PY_COMMON_PY3)) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_CONFIG_ENGINE)) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_PLATFORM_COMMON_PY2)) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(REDIS_DUMP_LOAD_PY2)) \
Expand Down Expand Up @@ -835,6 +837,8 @@ $(addprefix $(TARGET_PATH)/, $(SONIC_INSTALLERS)) : $(TARGET_PATH)/% : \
export installer_debs="$(addprefix $(IMAGE_DISTRO_DEBS_PATH)/,$($*_INSTALLS))"
export lazy_installer_debs="$(foreach deb, $($*_LAZY_INSTALLS),$(foreach device, $($(deb)_PLATFORM),$(addprefix $(device)@, $(IMAGE_DISTRO_DEBS_PATH)/$(deb))))"
export installer_images="$(addprefix $(TARGET_PATH)/,$($*_DOCKERS))"
export sonic_py_common_py2_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_PY_COMMON_PY2))"
export sonic_py_common_py3_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_PY_COMMON_PY3))"
export config_engine_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_CONFIG_ENGINE))"
export swsssdk_py2_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(SWSSSDK_PY2))"
export swsssdk_py3_wheel_path="$(addprefix $(PYTHON_WHEELS_PATH)/,$(SWSSSDK_PY3))"
Expand Down
39 changes: 39 additions & 0 deletions src/sonic-py-common/setup.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
from setuptools import setup

dependencies = [
'natsort',
'pyyaml',
'swsssdk>=2.0.1',
]

high_performance_deps = [
'swsssdk[high_perf]>=2.0.1',
]

setup(
name='sonic-py-common',
version='1.0',
description='Common Python libraries for SONiC',
license='Apache 2.0',
author='SONiC Team',
author_email='linuxnetdev@microsoft.com',
url='https://github.com/Azure/SONiC',
maintainer='Joe LeVeque',
maintainer_email='jolevequ@microsoft.com',
install_requires=dependencies,
extras_require={
'high_perf': high_performance_deps,
},
packages=[
'sonic_py_common',
],
classifiers=[
'Intended Audience :: Developers',
'Operating System :: Linux',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python',
],
keywords='SONiC sonic PYTHON python COMMON common',
)

Empty file.
81 changes: 81 additions & 0 deletions src/sonic-py-common/sonic_py_common/daemon_base.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
import imp
import signal
import sys

from . import device_info
from .logger import Logger

#
# Constants ====================================================================
#
REDIS_TIMEOUT_MSECS = 0

EEPROM_MODULE_NAME = 'eeprom'
EEPROM_CLASS_NAME = 'board'

#
# Helper functions =============================================================
#

def db_connect(db_name):
from swsscommon import swsscommon
return swsscommon.DBConnector(db_name, REDIS_TIMEOUT_MSECS, True)

#
# DaemonBase ===================================================================
#

class DaemonBase(Logger):
def __init__(self, log_identifier):
super(DaemonBase, self).__init__(log_identifier, Logger.LOG_FACILITY_DAEMON)

# Register our default signal handlers, unless the signal already has a
# handler registered, most likely from a subclass implementation
if not signal.getsignal(signal.SIGHUP):
signal.signal(signal.SIGHUP, self.signal_handler)
if not signal.getsignal(signal.SIGINT):
signal.signal(signal.SIGINT, self.signal_handler)
if not signal.getsignal(signal.SIGTERM):
signal.signal(signal.SIGTERM, self.signal_handler)

# Default signal handler; can be overridden by subclass
def signal_handler(self, sig, frame):
if sig == signal.SIGHUP:
log_info("DaemonBase: Caught SIGHUP - ignoring...")
elif sig == signal.SIGINT:
log_info("DaemonBase: Caught SIGINT - exiting...")
sys.exit(128 + sig)
elif sig == signal.SIGTERM:
log_info("DaemonBase: Caught SIGTERM - exiting...")
sys.exit(128 + sig)
else:
log_warning("DaemonBase: Caught unhandled signal '{}'".format(sig))

# Loads platform specific platform module from source
def load_platform_util(self, module_name, class_name):
platform_util = None

# Get path to platform and hwsku
(platform_path, hwsku_path) = device_info.get_path_to_platform_and_hwsku()

try:
module_file = "/".join([platform_path, "plugins", module_name + ".py"])
module = imp.load_source(module_name, module_file)
except IOError as e:
raise IOError("Failed to load platform module '%s': %s" % (module_name, str(e)))

try:
platform_util_class = getattr(module, class_name)
# board class of eeprom requires 4 paramerters, need special treatment here.
if module_name == EEPROM_MODULE_NAME and class_name == EEPROM_CLASS_NAME:
platform_util = platform_util_class('','','','')
else:
platform_util = platform_util_class()
except AttributeError as e:
raise AttributeError("Failed to instantiate '%s' class: %s" % (class_name, str(e)))

return platform_util

# Runs daemon
def run(self):
raise NotImplementedError()
Loading