Skip to content

Commit

Permalink
Add Checkpoint and Rollback for Multi ASIC. (sonic-net#3299)
Browse files Browse the repository at this point in the history 
#### What I did

Add `config` `checkpoint`, `rollback`, `replace`, `list-checkpoints`, `delete-checkpoint` support of Multi ASIC

#### How I did it
Add namespace for each of operation to support Multi ASIC.

#### How to verify it
1. Single ASIC
```admin@str2-msn2700-spy-1:~/gcu$ sudo config checkpoint 20240522-xincun
Config Rollbacker: Config checkpoint starting.
Config Rollbacker: Checkpoint name: 20240522-xincun.
Config Rollbacker: Getting current config db.
Config Rollbacker: Getting checkpoint full-path.
Config Rollbacker: Ensuring checkpoint directory exist.
Config Rollbacker: Saving config db content to /etc/sonic/checkpoints/20240522-xincun.cp.json.
Config Rollbacker: Config checkpoint completed.
Checkpoint created successfully.

admin@str2-msn2700-spy-1:~/gcu$ sudo config list-checkpoints 
[
    "20240522-xincun"
]

admin@str2-msn2700-spy-1:~/gcu$ sudo config rollback 20240522-xincun
Config Rollbacker: Config rollbacking starting.
Config Rollbacker: Checkpoint name: 20240522-xincun.
Config Rollbacker: Verifying '20240522-xincun' exists.
Config Rollbacker: Loading checkpoint into memory.
Config Rollbacker: Replacing config using 'Config Replacer'.
Config Replacer: Config replacement starting.
Config Replacer: Target config length: 71214.
Config Replacer: Getting current config db.
Config Replacer: Generating patch between target config and current config db.
Config Replacer: Applying patch using 'Patch Applier'.
Patch Applier: localhost: Patch application starting.
Patch Applier: localhost: Patch: []
Patch Applier: localhost getting current config db.
Patch Applier: localhost: simulating the target full config after applying the patch.
Patch Applier: localhost: validating all JsonPatch operations are permitted on the specified fields
Patch Applier: localhost: validating target config does not have empty tables,
                               since they do not show up in ConfigDb.
Patch Applier: localhost: sorting patch updates.
Patch Applier: The localhost patch was converted into 0 changes.
Patch Applier: localhost: applying 0 changes in order.
Patch Applier: localhost: verifying patch updates are reflected on ConfigDB.
Patch Applier: localhost patch application completed.
Config Replacer: Verifying config replacement is reflected on ConfigDB.
Config Replacer: Config replacement completed.
Config Rollbacker: Config rollbacking completed.
Config rolled back successfully.

admin@str2-msn2700-spy-1:~/gcu$ sudo config delete-checkpoint 20240522-xincun
Config Rollbacker: Deleting checkpoint starting.
Config Rollbacker: Checkpoint name: 20240522-xincun.
Config Rollbacker: Checking checkpoint exists.
Config Rollbacker: Deleting checkpoint.
Config Rollbacker: Deleting checkpoint completed.
Checkpoint deleted successfully.

admin@str2-msn2700-spy-1:~/gcu$ sudo config list-checkpoints 
[]
```

2. Multi ASIC
```
stli@str2-7250-2-lc01:~/gcu$ sudo config checkpoint 20240522-xincun
MultiASICConfigRollbacker: Config checkpoint starting.
MultiASICConfigRollbacker: Checkpoint name: 20240522-xincun.
MultiASICConfigRollbacker: Getting current  config db.
MultiASICConfigRollbacker: Getting current asic0 config db.
MultiASICConfigRollbacker: Getting current asic1 config db.
MultiASICConfigRollbacker: Getting checkpoint full-path.
MultiASICConfigRollbacker: Ensuring checkpoint directory exist.
MultiASICConfigRollbacker: Saving config db content to /etc/sonic/checkpoints/20240522-xincun.cp.json.
MultiASICConfigRollbacker: Config checkpoint completed.
Checkpoint created successfully.
stli@str2-7250-2-lc01:~/gcu$ sudo config list-checkpoints 
[
    "20240522-xincun"
]

stli@str2-7250-2-lc01:~/gcu$ sudo config rollback 20240522-xincun
MultiASICConfigRollbacker: Config rollbacking starting.
MultiASICConfigRollbacker: Checkpoint name: 20240522-xincun.
MultiASICConfigRollbacker: Verifying '20240522-xincun' exists.
MultiASICConfigRollbacker: Loading checkpoint '20240522-xincun' into memory.
MultiASICConfigRollbacker: Replacing config '20240522-xincun' using 'Config Replacer'.
Config Replacer: Config replacement starting.
Config Replacer: Target config length: 38147.
Config Replacer: Getting current config db.
Config Replacer: Generating patch between target config and current config db.
Config Replacer: Applying patch using 'Patch Applier'.
Patch Applier: localhost: Patch application starting.
Patch Applier: localhost: Patch: []
Patch Applier: localhost getting current config db.
Patch Applier: localhost: simulating the target full config after applying the patch.
Patch Applier: localhost: validating all JsonPatch operations are permitted on the specified fields
Patch Applier: localhost: validating target config does not have empty tables,
                               since they do not show up in ConfigDb.
Patch Applier: localhost: sorting patch updates.
Patch Applier: The localhost patch was converted into 0 changes.
Patch Applier: localhost: applying 0 changes in order.
Patch Applier: localhost: verifying patch updates are reflected on ConfigDB.
Patch Applier: localhost patch application completed.
Config Replacer: Verifying config replacement is reflected on ConfigDB.
Config Replacer: Config replacement completed.
Config Replacer: Config replacement starting.
Config Replacer: Target config length: 97546.
Config Replacer: Getting current config db.
Config Replacer: Generating patch between target config and current config db.
Config Replacer: Applying patch using 'Patch Applier'.
Patch Applier: asic0: Patch application starting.
Patch Applier: asic0: Patch: []
Patch Applier: asic0 getting current config db.
Patch Applier: asic0: simulating the target full config after applying the patch.
Patch Applier: asic0: validating all JsonPatch operations are permitted on the specified fields
Patch Applier: asic0: validating target config does not have empty tables,
                               since they do not show up in ConfigDb.
Patch Applier: asic0: sorting patch updates.
Patch Applier: The asic0 patch was converted into 0 changes.
Patch Applier: asic0: applying 0 changes in order.
Patch Applier: asic0: verifying patch updates are reflected on ConfigDB.
Patch Applier: asic0 patch application completed.
Config Replacer: Verifying config replacement is reflected on ConfigDB.
Config Replacer: Config replacement completed.
Config Replacer: Config replacement starting.
Config Replacer: Target config length: 97713.
Config Replacer: Getting current config db.
Config Replacer: Generating patch between target config and current config db.
Config Replacer: Applying patch using 'Patch Applier'.
Patch Applier: asic1: Patch application starting.
Patch Applier: asic1: Patch: []
Patch Applier: asic1 getting current config db.
Patch Applier: asic1: simulating the target full config after applying the patch.
Patch Applier: asic1: validating all JsonPatch operations are permitted on the specified fields
Patch Applier: asic1: validating target config does not have empty tables,
                               since they do not show up in ConfigDb.
Patch Applier: asic1: sorting patch updates.
Patch Applier: The asic1 patch was converted into 0 changes.
Patch Applier: asic1: applying 0 changes in order.
Patch Applier: asic1: verifying patch updates are reflected on ConfigDB.
Patch Applier: asic1 patch application completed.
Config Replacer: Verifying config replacement is reflected on ConfigDB.
Config Replacer: Config replacement completed.
MultiASICConfigRollbacker: Config rollbacking completed.
Config rolled back successfully.

stli@str2-7250-2-lc01:~/gcu$ sudo config delete-checkpoint 20240522-xincun
MultiASICConfigRollbacker: Deleting checkpoint starting.
MultiASICConfigRollbacker: Checkpoint name: 20240522-xincun.
MultiASICConfigRollbacker: Checking checkpoint: 20240522-xincun exists.
MultiASICConfigRollbacker: Deleting checkpoint: 20240522-xincun.
MultiASICConfigRollbacker: Deleting checkpoint: 20240522-xincun completed.
Checkpoint deleted successfully.
stli@str2-7250-2-lc01:~/gcu$ sudo config list-checkpoints 
[]

stli@str2-7250-2-lc01:~/gcu$ sudo config replace 20240522-xincun.cp.json 
Config Replacer: Config replacement starting.
Config Replacer: Target config length: 38147.
Config Replacer: Getting current config db.
Config Replacer: Generating patch between target config and current config db.
Config Replacer: Applying patch using 'Patch Applier'.
Patch Applier: localhost: Patch application starting.
Patch Applier: localhost: Patch: []
Patch Applier: localhost getting current config db.
Patch Applier: localhost: simulating the target full config after applying the patch.
Patch Applier: localhost: validating all JsonPatch operations are permitted on the specified fields
Patch Applier: localhost: validating target config does not have empty tables,
                               since they do not show up in ConfigDb.
Patch Applier: localhost: sorting patch updates.
Patch Applier: The localhost patch was converted into 0 changes.
Patch Applier: localhost: applying 0 changes in order.
Patch Applier: localhost: verifying patch updates are reflected on ConfigDB.
Patch Applier: localhost patch application completed.
Config Replacer: Verifying config replacement is reflected on ConfigDB.
Config Replacer: Config replacement completed.
Config Replacer: Config replacement starting.
Config Replacer: Target config length: 97546.
Config Replacer: Getting current config db.
Config Replacer: Generating patch between target config and current config db.
Config Replacer: Applying patch using 'Patch Applier'.
Patch Applier: asic0: Patch application starting.
Patch Applier: asic0: Patch: []
Patch Applier: asic0 getting current config db.
Patch Applier: asic0: simulating the target full config after applying the patch.
Patch Applier: asic0: validating all JsonPatch operations are permitted on the specified fields
Patch Applier: asic0: validating target config does not have empty tables,
                               since they do not show up in ConfigDb.
Patch Applier: asic0: sorting patch updates.
Patch Applier: The asic0 patch was converted into 0 changes.
Patch Applier: asic0: applying 0 changes in order.
Patch Applier: asic0: verifying patch updates are reflected on ConfigDB.
Patch Applier: asic0 patch application completed.
Config Replacer: Verifying config replacement is reflected on ConfigDB.
Config Replacer: Config replacement completed.
Config Replacer: Config replacement starting.
Config Replacer: Target config length: 97713.
Config Replacer: Getting current config db.
Config Replacer: Generating patch between target config and current config db.
Config Replacer: Applying patch using 'Patch Applier'.
Patch Applier: asic1: Patch application starting.
Patch Applier: asic1: Patch: []
Patch Applier: asic1 getting current config db.
Patch Applier: asic1: simulating the target full config after applying the patch.
Patch Applier: asic1: validating all JsonPatch operations are permitted on the specified fields
Patch Applier: asic1: validating target config does not have empty tables,
                               since they do not show up in ConfigDb.
Patch Applier: asic1: sorting patch updates.
Patch Applier: The asic1 patch was converted into 0 changes.
Patch Applier: asic1: applying 0 changes in order.
Patch Applier: asic1: verifying patch updates are reflected on ConfigDB.
Patch Applier: asic1 patch application completed.
Config Replacer: Verifying config replacement is reflected on ConfigDB.
Config Replacer: Config replacement completed.
Config replaced successfully.
```
  • Loading branch information
@xincunli-sonic
xincunli-sonic authored Jun 14, 2024
1 parent c2370f8 commit d0856af
Show file tree
Hide file tree
Showing 10 changed files with 420 additions and 132 deletions.
2 changes: 1 addition & 1 deletion config/main.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1185,7 +1185,7 @@ def apply_patch_for_scope(scope_changes, results, config_format, verbose, dry_ru
scope_for_log = scope if scope else HOST_NAMESPACE
try:
# Call apply_patch with the ASIC-specific changes and predefined parameters
GenericUpdater(namespace=scope).apply_patch(jsonpatch.JsonPatch(changes),
GenericUpdater(scope=scope).apply_patch(jsonpatch.JsonPatch(changes),
config_format,
verbose,
dry_run,
Expand Down
32 changes: 14 additions & 18 deletions generic_config_updater/change_applier.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@

print_to_console = False


def set_verbose(verbose=False):
global print_to_console, logger

Expand All @@ -34,11 +35,12 @@ def log_error(m):
logger.log(logger.LOG_PRIORITY_ERROR, m, print_to_console)


def get_config_db(namespace=multi_asic.DEFAULT_NAMESPACE):
config_db = ConfigDBConnector(use_unix_socket_path=True, namespace=namespace)
def get_config_db(scope=multi_asic.DEFAULT_NAMESPACE):
config_db = ConfigDBConnector(use_unix_socket_path=True, namespace=scope)
config_db.connect()
return config_db


def set_config(config_db, tbl, key, data):
config_db.set_entry(tbl, key, data)

Expand All @@ -61,11 +63,9 @@ class DryRunChangeApplier:
def __init__(self, config_wrapper):
self.config_wrapper = config_wrapper


def apply(self, change):
self.config_wrapper.apply_change_to_config_db(change)


def remove_backend_tables_from_config(self, data):
return data

Expand All @@ -74,9 +74,9 @@ class ChangeApplier:

updater_conf = None

def __init__(self, namespace=multi_asic.DEFAULT_NAMESPACE):
self.namespace = namespace
self.config_db = get_config_db(self.namespace)
def __init__(self, scope=multi_asic.DEFAULT_NAMESPACE):
self.scope = scope
self.config_db = get_config_db(self.scope)
self.backend_tables = [
"BUFFER_PG",
"BUFFER_PROFILE",
Expand All @@ -86,7 +86,6 @@ def __init__(self, namespace=multi_asic.DEFAULT_NAMESPACE):
with open(UPDATER_CONF_FILE, "r") as s:
ChangeApplier.updater_conf = json.load(s)


def _invoke_cmd(self, cmd, old_cfg, upd_cfg, keys):
# cmd is in the format as <package/module name>.<method name>
#
Expand All @@ -98,7 +97,6 @@ def _invoke_cmd(self, cmd, old_cfg, upd_cfg, keys):

return method_to_call(old_cfg, upd_cfg, keys)


def _services_validate(self, old_cfg, upd_cfg, keys):
lst_svcs = set()
lst_cmds = set()
Expand All @@ -124,7 +122,6 @@ def _services_validate(self, old_cfg, upd_cfg, keys):
log_debug("service invoked: {}".format(cmd))
return 0


def _upd_data(self, tbl, run_tbl, upd_tbl, upd_keys):
for key in set(run_tbl.keys()).union(set(upd_tbl.keys())):
run_data = run_tbl.get(key, None)
Expand All @@ -135,20 +132,17 @@ def _upd_data(self, tbl, run_tbl, upd_tbl, upd_keys):
upd_keys[tbl][key] = {}
log_debug("Patch affected tbl={} key={}".format(tbl, key))


def _report_mismatch(self, run_data, upd_data):
log_error("run_data vs expected_data: {}".format(
str(jsondiff.diff(run_data, upd_data))[0:40]))


def apply(self, change):
run_data = self._get_running_config()
upd_data = prune_empty_table(change.apply(copy.deepcopy(run_data)))
upd_keys = defaultdict(dict)

for tbl in sorted(set(run_data.keys()).union(set(upd_data.keys()))):
self._upd_data(tbl, run_data.get(tbl, {}),
upd_data.get(tbl, {}), upd_keys)
self._upd_data(tbl, run_data.get(tbl, {}), upd_data.get(tbl, {}), upd_keys)

ret = self._services_validate(run_data, upd_data, upd_keys)
if not ret:
Expand All @@ -168,9 +162,9 @@ def remove_backend_tables_from_config(self, data):

def _get_running_config(self):
_, fname = tempfile.mkstemp(suffix="_changeApplier")
if self.namespace:
cmd = ['sonic-cfggen', '-d', '--print-data', '-n', self.namespace]

if self.scope:
cmd = ['sonic-cfggen', '-d', '--print-data', '-n', self.scope]
else:
cmd = ['sonic-cfggen', '-d', '--print-data']

Expand All @@ -181,7 +175,9 @@ def _get_running_config(self):
return_code = result.returncode
if return_code:
os.remove(fname)
raise GenericConfigUpdaterError(f"Failed to get running config for namespace: {self.namespace}, Return code: {return_code}, Error: {err}")
raise GenericConfigUpdaterError(
f"Failed to get running config for scope: {self.scope}," +
f"Return code: {return_code}, Error: {err}")

run_data = {}
try:
Expand Down
Loading

0 comments on commit d0856af

Please sign in to comment.