Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Management Framework HLD PR #436

Merged
merged 78 commits into from
Oct 22, 2019
Merged
Changes from 1 commit
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
ac4ea94
Add gnmi doc
seiferteric Jul 30, 2019
81a3d92
Fix formatting, change target
seiferteric Jul 30, 2019
66c43bb
BRCM+Dell initial contribution of Mgmt-framework doc
Jul 30, 2019
4c351cf
Merge pull request #1 from project-arlo/eric-wip
seiferteric Jul 30, 2019
7337b14
Push latest changes from brcm
Aug 5, 2019
7fcdcac
Merge branch 'mgmt-framework' of https://github.com/project-arlo/SONi…
Aug 5, 2019
702f9af
Added Transformer section to HLD
jeff-yin Aug 5, 2019
308a985
Revert "Added Transformer section to HLD"
jeff-yin Aug 5, 2019
21e97a1
Un-revert "Added Transformer section to HLD"
jeff-yin Aug 5, 2019
4779f96
Updated GNMI information
skg-net Aug 6, 2019
1bc3f63
Added the GNMI Server Design
skg-net Aug 6, 2019
0c398b8
Merge pull request #2 from project-arlo/gnmi_upd
PrabhuSreenivasan Aug 6, 2019
5db33a7
few cosmetic changes
Aug 6, 2019
9256725
Update Mgmt_Frmk_Arch.jpg
jeff-yin Aug 6, 2019
412d2f5
Merge branch 'mgmt-framework' of https://github.com/project-arlo/SONi…
jeff-yin Aug 6, 2019
006daf6
Update management framework HLD
jeff-yin Aug 6, 2019
ed7c378
updated gnmi and cli section with some minor changes, and removal of …
jghalam Aug 7, 2019
80a5f03
BRCM changes
Aug 7, 2019
7e5b24a
fixed hyper link for Appendix
Aug 7, 2019
45add3b
removed section 11
Aug 7, 2019
a62dcbd
Management Framework HLD: Updated for
a-barboza Aug 7, 2019
442283d
HLD Management Framework:
a-barboza Aug 7, 2019
23871d1
HLD Management Framework: Fixed indents on last commit
a-barboza Aug 7, 2019
5a3ad84
Update Management Framework.md
kwangsuk Aug 8, 2019
27c7da5
update the transformer components diagram
kwangsuk Aug 8, 2019
9e71314
update the components diagram
kwangsuk Aug 8, 2019
3fff54a
updated developer work flow
Aug 8, 2019
41d3bed
fixed hyper links
Aug 8, 2019
48712dd
fixed hyper links
Aug 8, 2019
7fc4a13
replaced CISCO Style CLI with IS CLI
Aug 8, 2019
ea0cfbe
replaced the components diagram w/ larger version
kwangsuk Aug 8, 2019
680310a
Delete obsolete diagram
kwangsuk Aug 8, 2019
2ba87f9
Update management Framework HLD
jeff-yin Aug 9, 2019
201ee15
Updated Management Framework.md
dutta-partha Aug 9, 2019
01fb105
Merge pull request #3 from dutta-partha/patch-1
sachin-brm Aug 9, 2019
95123d2
Merge pull request #4 from project-arlo/mgmt-framework
a-barboza Aug 9, 2019
9db3778
Review comments incorporated.
Aug 22, 2019
469866b
Merge pull request #5 from project-arlo/master
PrabhuSreenivasan Aug 22, 2019
ac23706
Merge branch 'mgmt-framework' of https://github.com/project-arlo/SONi…
Aug 22, 2019
e22f70c
Merge pull request #6 from project-arlo/mgmt-framework
PrabhuSreenivasan Aug 22, 2019
16ceb13
Changes by broadcom team
Aug 27, 2019
5798ee9
Merge pull request #7 from project-arlo/mgmt-framework
PrabhuSreenivasan Aug 29, 2019
efcf7f4
Add files via upload
kwangsuk Sep 5, 2019
2f8ea6b
Delete transformer spec diagram.vsdx
kwangsuk Sep 5, 2019
2d7c87b
Add files via upload
kwangsuk Sep 5, 2019
ae5d8f2
Delete transformer_components.png
kwangsuk Sep 5, 2019
c5ab609
Delete transformer components.png
kwangsuk Sep 5, 2019
09858f0
Add files via upload
kwangsuk Sep 5, 2019
7b763ed
Add files via upload
kwangsuk Sep 5, 2019
8fbc809
Add files via upload
kwangsuk Sep 5, 2019
ff680c0
Update Management Framework.md
kwangsuk Sep 5, 2019
9c9b86b
update the Transformer section
kwangsuk Sep 5, 2019
e1265bc
Update Management Framework.md
kwangsuk Sep 25, 2019
3275f4b
Update Management Framework.md
kwangsuk Sep 26, 2019
b717a74
Update Management Framework.md
kwangsuk Sep 26, 2019
7efcaa7
Update the Transformer section
kwangsuk Sep 26, 2019
4434143
Merge pull request #18 from project-arlo/Update-HLD
kwangsuk Sep 26, 2019
411365e
Resize the sequence diagrams
kwangsuk Sep 26, 2019
c41d9e3
resize the diagrams
kwangsuk Sep 26, 2019
1e596c5
Update Management Framework.md
kwangsuk Sep 28, 2019
535c451
updated images
kwangsuk Sep 29, 2019
6200a7a
update architecture diagram
kwangsuk Sep 29, 2019
e045691
update image
kwangsuk Sep 29, 2019
c23769f
Merge branch 'Update-HLD' of https://github.com/project-arlo/SONiC in…
kwangsuk Sep 29, 2019
38cfdb0
fix conflicts
kwangsuk Sep 29, 2019
60e12f7
Fix merging conflicts
kwangsuk Sep 30, 2019
a451026
Merge pull request #21 from project-arlo/Update-HLD
kwangsuk Sep 30, 2019
8accb18
Update based on SONiC YANG
dutta-partha Sep 30, 2019
ab4be29
Merge pull request #22 from dutta-partha/patch-1
PrabhuSreenivasan Sep 30, 2019
dc11ccf
redo the contents
kwangsuk Oct 2, 2019
2642b18
Update Management Framework.md
kwangsuk Oct 2, 2019
10fe5d9
Merge pull request #24 from project-arlo/Update-HLD
kwangsuk Oct 2, 2019
9d77b02
RADIUS Management User HLD 0.6
a-barboza Oct 15, 2019
354e75b
Incorporating review comments from the community
anand-kumar-subramanian Oct 15, 2019
0c1f3e9
Incorporating review comments from the community phase 2
anand-kumar-subramanian Oct 17, 2019
70e9a17
Address Review comment - Libraries used in sonic-telemetry
Oct 20, 2019
5a1a1fd
Merge pull request #30 from project-arlo/mgmtfr_spec_comments
dutta-partha Oct 21, 2019
7c51962
RADIUS Management User HLD 0.6
a-barboza Oct 21, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update Management Framework.md
kwangsuk committed Sep 26, 2019
commit b717a745455bbf11c8216db534557f8f7a2650c5
99 changes: 53 additions & 46 deletions doc/mgmt/Management Framework.md
Original file line number Diff line number Diff line change
@@ -1198,78 +1198,73 @@ e.g. in case that the sonic-acl.yang used by NBI and the payload with CRAETE ope

The translation hints are defined as YANG extensions to support simple table/field name mapping or more complex data translation by overloading the default methods.

| Extensions | Usage |
| ---------- | ----- |
| `sonic-ext:table-name [string]` | Map a YANG container/list to TABLE name |
| `sonic-ext:field-name [string]` | Map a YANG leafy - leaf or leaf-list - node to FIELD name |
| `sonic-ext:key-delimiter [string]` | Override the default delimiter, “|” |
| `sonic-ext:key-name [string]` | Fixed key name, used for YANG container mapped to TABLE with a fixed key |
| `sonic-ext:key-transformer [function]` | Overloading default method to generate DB keys(s) |
| `sonic-ext:field-transformer [function]` | Overloading default method to generate FIELD value |
| `sonic-ext:subtree-transformer [function]` | Overloading default method for the current subtree |
| `sonic-ext:db-name [string]` | DB name to access data – “APPL_DB”, “ASIC_DB”, “COUNTERS_DB”, “CONFIG_DB”, “FLEX_COUNTER_DB”, “STATE_DB”. The default db-name is CONFIG_DB |
| `sonic-ext:post-transformer [function]` | A special hook to update the DB requests right before passing to common-app |
| `sonic-ext:table-transformer [function]` | Dynamically map a YANG container/list to TABLE name(s) |
| `sonic-ext:get-validate [function]` | A special hook to validate YANG nodes, to populate data read from database |

Note:
1. sonic-ext:table-name [string]
----------

1. `sonic-ext:table-name [string]`: Map a YANG container/list to TABLE name
Processed by the default transformer method. Argument is a table name statically mapped to the given YANG container or list node
The table-name is inherited to all descendant nodes unless another one is defined.
2. sonic-ext:field-name [string]

2. `sonic-ext:field-name [string]`: Map a YANG leafy - leaf or leaf-list - node to FIELD name
Processed by the default transformer method
3. sonic-ext:key-delimiter [string]

3. `sonic-ext:key-delimiter [string]`: Override the default delimiter, “|”
Processed by the default transformer method. Used to concatenate multiple YANG keys of a YANG list into a single DB key
Note that the default delimiter “|” is used when keys are concatenated between current node and the nested nodes, i.e. container/list
4. sonic-ext:key-name [string]

4. `sonic-ext:key-name [string]`: Fixed key name, used for YANG container mapped to TABLE with a fixed key
Processed by the default transformer method. Used to define a fixed key, mainly for container mapped to TABLE key,
e.g. Redis can have a hash “STP|GLOBAL”
```YANG
container global
sonic-ext:table-name “STP”
sonic-ext:key-name “GLOBAL”
5. sonic-ext:key-transformer [function]
```
5. `sonic-ext:key-transformer [function]`: Overloading default method to generate DB keys(s)
Used when the key values in a YANG list are different from ones in DB TABLE
A pair of callbacks should be implemented to support 2 way translation - YangToDBxxx, Db2Yangxxx
6. sonic-ext:field-transformer [function]
A pair of callbacks should be implemented to support 2 way translation - **YangToDB***function*, **DbToYang***function*

6. `sonic-ext:field-transformer [function]`: Overloading default method to generate FIELD value
Used when the leaf/leaf-list values defined in a YANG list are different from the field values in DB
A pair of callbacks should be implemented to support 2 way translation - YangToDBxxx, Db2Yangxxx
7. sonic-ext:subtree-transformer [function]
A pair of callbacks should be implemented to support 2 way translation - **YangToDB***function*, **DbToYang***function*

7. `sonic-ext:subtree-transformer [function]`: Overloading default method for the current subtree
Allows the sub-tree transformer to take full control of translation. Note that, if any other extensions are annotated to the nodes on the subtree, they are not effective.
The subtree-transformer is inherited to all descendant nodes unless another one is defined, i.e. the scope of subtree-transformer callback is limited to the current and descendant nodes along the YANG path until a new subtree transformer is annotated.
A pair of callbacks should be implemented to support 2 way translation - YangToDBxxx, DbToYangxxx
8. sonic-ext:db-name [string]
A pair of callbacks should be implemented to support 2 way translation - **YangToDB***function*, **DbToYang***function*

8. `sonic-ext:db-name [string]`: DB name to access data – “APPL_DB”, “ASIC_DB”, “COUNTERS_DB”, “CONFIG_DB”, “FLEX_COUNTER_DB”, “STATE_DB”. The default db-name is CONFIG_DB
Used for GET operation to non CONFIG_DB, applicable only to SONiC YANG
Processed by Transformer core to traverse database
The db-name is inherited to all descendant nodes unless another one. Must be defined with the table-name
9. sonic-ext:post-transformer [function]

9. `sonic-ext:post-transformer [function]`: A special hook to update the DB requests right before passing to common-app
Analogous to the postponed YangToDB subtree callback that is invoked at the very end by the Transformer.
Used to add/update additional data to the maps returned from Transformer before passing to common-app, e.g. add a default acl rule
Note that the post-transformer can be annotated only to the top-level container(s) within each module, and called once for the given node during translation
10. sonic-ext:table-transformer [function]

10. `sonic-ext:table-transformer [function]`: Dynamically map a YANG container/list to TABLE name(s)
Allows the table-transformer to map a YANG list/container to table names
Used to dynamically map a YANG list/container to table names based on URI and payload
The table-transformer is inherited to all descendant nodes unless another one is defined
11. sonic-ext:get-validate [function]

11. `sonic-ext:get-validate [function]`: A special hook to validate YANG nodes, to populate data read from database
Allows developers to instruct Transformer to choose a YANG node among multiple nodes, while constructing the response payload
Typically used to check the “when” condition to validate YANG node among multiple nodes to choose only valid nodes from sibling nodes.

----------


Note that the key-transformer, field-transformer and subtree-transformer have a pair of callbacks associated with 2 way translation using a prefix - YangToDBxxx, Db2Yangxxx. It is not mandatory to implement both functions. E.g. if you need a translation for GET operation only, you can implement only Db2Yangxxx.
Note that the key-transformer, field-transformer and subtree-transformer have a pair of callbacks associated with 2 way translation using a prefix - **YangToDB***function*, **DbToYang***function*. It is not mandatory to implement both functions. E.g. if you need a translation for GET operation only, you can implement only **DbToYang***function*.

The template annotation file can be generated and used by the developers to define extensions to the yang paths as needed to translate data between YANG and ABNF format. Refer to the 3.2.2.7.8 Utilities.

Here is the general guide you can check to find which extensions can be annotated in implementing your model.
```YANG
1) If the translation is simple mapping between YANG container/list and TABLE, consider using the extensions - table-name, field-name, optionally key-delimiter

2) If the translation requires a complex translation with your codes, consider the following transformer extensions - key-transformer, field-transformer, subtree-transformer to take a control during translation. Note that multiple subtree-transformers can be annotated along YANG path to divide the scope

3) If multiple tables are mapped to a YANG list, e.g. openconfig-interface.yang, use the table-transformer to dynamically choose tables based on URI/payload

4) In Get operation access to non CONFIG_DB, you can use the db-name extension

5) In Get operation, you can annotate the subtree-transformer on the node to implement your own data access and translation with DbToYangxxx function

6) In case of mapping a container to TABLE/KET, you can use the key-name along with the table-name extension
```

@@ -1292,50 +1287,62 @@ func GetAndXlateFromDB(xpath string, uri *ygot.GoStruct, dbs [db.MaxDB]*db.DB) (
Overloaded transformer methods are prepended with ‘YangToDb’ or ‘DbToYang’ to support bi-directional data transfer. The function prototypes defined in the following-

```go
type XfmrParams struct {
d *db.DB
dbs [db.MaxDB]*db.DB
curDb db.DBNum
ygRoot *ygot.GoStruct
uri string
oper int
key string
dbDataMap *map[db.DBNum]map[string]map[string]db.Value
param interface{}
}

/**
* KeyXfmrYangToDb type is defined to use for conversion of Yang key to DB Key
* Transformer function definition.
* Param: Database info, YgotRoot, operation, Xpath
* Param: XfmrParams structure having Database info, YgotRoot, operation, Xpath
* Return: Database keys to access db entry, error
**/
type KeyXfmrYangToDb func (*db.DB, *ygot.GoStruct, int, string) (string, error)
type KeyXfmrYangToDb func (inParams XfmrParams) (string, error)
/**
* KeyXfmrDbToYang type is defined to use for conversion of DB key to Yang key
* Transformer function definition.
* Param: Database info, operation, Database keys to access db entry
* Param: XfmrParams structure having Database info, operation, Database keys to access db entry
* Return: multi dimensional map to hold the yang key attributes of complete xpath, error
**/
type KeyXfmrDbToYang func (*db.DB, int, string) (map[string]map[string]string, error)
type KeyXfmrDbToYang func (inParams XfmrParams) (map[string]interface{}, error)

/**
* FieldXfmrYangToDb type is defined to use for conversion of yang Field to DB field
* Transformer function definition.
* Param: Database info, YgotRoot, operation, Xpath
* Return: multi dimensional map to hold the DB data, error
**/
type FieldXfmrYangToDb func (*db.DB, *ygot.GoStruct, int, string, interface {}) (map[string]string, error)
type FieldXfmrYangToDb func (inParams XfmrParams) (map[string]string, error)
/**
* FieldXfmrDbtoYang type is defined to use for conversion of DB field to Yang field
* Transformer function definition.
* Param: Database info, operation, DB data in multidimensional map, output param YgotRoot
* Param: XfmrParams structure having Database info, operation, DB data in multidimensional map, output param YgotRoot
* Return: error
**/
type FieldXfmrDbtoYang func (*db.DB, int, map[string]map[string]db.Value, *ygot.GoStruct) (error)
type FieldXfmrDbtoYang func (inParams XfmrParams) (map[string]interface{}, error)

/**
* SubTreeXfmrYangToDb type is defined to use for handling the yang subtree to DB
* Transformer function definition.
* Param: Database info, YgotRoot, operation, Xpath
* Param: XfmrParams structure having Database info, YgotRoot, operation, Xpath
* Return: multi dimensional map to hold the DB data, error
**/
type SubTreeXfmrYangToDb func (*db.DB, *ygot.GoStruct, int, string) (map[string]map[string]db.Value, error)
type SubTreeXfmrYangToDb func (inParams XfmrParams) (map[string]map[string]db.Value, error)
/**
* SubTreeXfmrDbToYang type is defined to use for handling the DB to Yang subtree
* Transformer function definition.
* Param : Database info, operation, DB data in multidimensional map, output param YgotRoot
* Param : XfmrParams structure having Database pointers, current db, operation, DB data in multidimensional map, output param YgotRoot, uri
* Return : error
**/
type SubTreeXfmrDbToYang func (*db.DB, int, map[string]map[string]db.Value, *ygot.GoStruct, string) (error)
type SubTreeXfmrDbToYang func (inParams XfmrParams) (error)
/**
* ValidateCallpoint is used to validate a YANG node during data translation back to YANG as a response to GET
* Param : XfmrParams structure having Database pointers, current db, operation, DB data in multidimensional map, output param YgotRoot, uri