Skip to content

Commit

Permalink
Update SONiC_static_route_bfd_hld.md
Browse files Browse the repository at this point in the history
Minor editorial and format updates
  • Loading branch information
jamesan47 authored Jan 6, 2023
1 parent 0fd7a6e commit 3d3170f
Showing 1 changed file with 45 additions and 44 deletions.
89 changes: 45 additions & 44 deletions doc/static-route/SONiC_static_route_bfd_hld.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ This document describes the design of using BFD to detect static route nexthop a

One important assumption is that the BFD session (for a static route nexthop) in this BFD Static Route application is not shared with any other applications (i.e., creating or deleting static route BFD session has no impact on other applications).<br />

<span style="color:blue"> *Notes:*<br>
*This document describes how to create local BFD session for a static route based on static route configuration. Creating BFD session in its peer system is out of the scope of this document.*<br />
<span style="color:blue"> <br>
*Note: This document describes how to create local BFD session for a static route based on static route configuration. Creating BFD session in its peer system is out of the scope of this document.*<br />
</span>


Expand Down Expand Up @@ -36,71 +36,72 @@ Two optional fields are introduced to STATIC_ROUTE_TABLE: <br>
2. "refreshable"

StaticRouteMgr(config_db) and BfdRouteMgr check "bfd" field in config_db STATIC_ROUTE_TABLE.<br>
BfdRouteMgr sets "refreshable"="false" in appl_db STATIC_ROUTE_TABLE. StaticRouteTimer checks "refreshable" field to skip the static route entry timeout checking.<br>
[*Link to STATIC_ROUTE_TABLE schema:*
BfdRouteMgr sets "refreshable"="false" in appl_db STATIC_ROUTE_TABLE.
StaticRouteTimer checks "refreshable" field to skip the static route entry timeout checking.<br>

[*Reference: STATIC_ROUTE_TABLE schema:*
[STATIC_ROUTE table in CONFIG_DB](https://github.com/Azure/SONiC/blob/master/doc/static-route/SONiC_static_route_hdl.md#3211-static_route).]


## Internal tables in BfdRouteMgr
Four tables (dictionary/map) are needed to use BFD session to monitor nexthop and update static route.<br>
Four tables (i.e., dictionary, map, etc) are needed to use BFD session to monitor nexthop and update static route.<br>

1. TABLE_CONFIG: config_db STATIC_ROUTE_TABLE cache (for the route with "bfd"="true" only)
2. TABLE_NEXTHOP: different prefixes may have same nexthop, this table is used to track which prefix is using the nexthop
3. TABLE_BFD: bfd session created by BfdRouteMgr, contents are part of appl_db BFD_SESSION_TABLE (for the session its peer IP is in nexthop table) <br>
*Assumption: BFD session is not shared with other applications*
4. TABLE_SR: the static routes written to appl_db STATIC_ROUTE_TABLE by BfdRouteManager (with "refreshable"="false"), it's nexthop list might be different from the configuration depends on BFD session state.<br>
2. TABLE_NEXTHOP: different prefixes may have same nexthop. This table is used to track which prefix is using the nexthop.
3. TABLE_BFD: bfd session created by BfdRouteMgr. The contents are part of appl_db BFD_SESSION_TABLE (for the session its peer IP is in nexthop table). *Assumption: BFD session is not shared with other applications*
4. TABLE_SR: the static routes written to appl_db STATIC_ROUTE_TABLE by BfdRouteManager (with "refreshable"="false"). It's nexthop list might be different from the configuration depends on BFD session state.<br>

<img src="static_rt_bfd_table.png" width="400">
<br />

## Adding/updating static route flow
when a new static route is added to config_db STATIC_ROUTE_TABLE,
When a new static route is added to config_db STATIC_ROUTE_TABLE, the following steps are taken.

* 1\. BfdRouteMgr is notified and get the new static route, skip this route if there is no "bfd" field or "bfd"="false".
* 1\. BfdRouteMgr is notified of the new static route. Skip this route if there is no "bfd" field or "bfd"="false".
* 2\. BfdRouteMgr check TABLE_CONFIG to see if the route is already in this table
* a\. if the route is NOT in the TABLE_CONFIG:
* i\. a new route will be added to TABLE_CONFIG.
* ii\. a new route will be added to TABLE_SR, but make it's nexthop list empty (BFD state update will modify the nexthop list).
* iii\. For each next hop, need to check TABLE_NEXTHOP,
* If the route is NOT in the TABLE_CONFIG:
* A new route will be added to TABLE_CONFIG.
* A new route will be added to TABLE_SR, but make it's nexthop list empty (BFD state update will modify the nexthop list).
* For each next hop, need to check TABLE_NEXTHOP,
1. if the entry already exist, add prefix to the existing nexthop entry, break.
2. create a new entry with prefix if there is no such entry yet, update TABLE_BFD and write it to redis appl_db BFD_SESSION_TABLE to create BFD session
* b\. if the route is in the TABLE_CONFIG already, update the corresponding fields. for nexthop list, compare it to identify which nexthop is new added and which is deleted.
* i\. for the new added nexthop, do the above (a.ii)
* ii\. for the deleted nexthop, look up the TABLE_NEXTHOP to get nexthop entry, remove the prefix from the nexthop entry.
* If the route is in the TABLE_CONFIG already, update the corresponding fields. For nexthop list, compare it to identify which nexthop is new added and which is deleted.
* For the new added nexthop, add to TABLE_SR.
* For the deleted nexthop, look up the TABLE_NEXTHOP to get nexthop entry. Remove the prefix from the nexthop entry.
1. if the is no prefix in that nexthop entry, delete the corresponding BFD sessions (from redis appl_db and state_db) and delete that nexhop entry

## Deleting static route flow
* 1\. BfdRouteMgr is notified and get the prefix, skip this route if there is no "bfd" field or "bfd"="false".
* 1\. BfdRouteMgr is notified and gets the prefix. Skip this route if there is no "bfd" field or "bfd"="false".
* 2\. BfdRouteMgr check TABLE_CONFIG to see if the route is already in this table
* a\. if the route is NOT in the table, done (should not happen)
* b\. if the route is in the table, For each nexthop in the static route entry.
* i\. look up the TABLE_NEXTHOP to get nexthop entry, remove the prefix from the nexthop entry.
* ii\. if the is no prefix in that nexthop entry, delete the corresponding BFD sessions (from redis appl_db and state_db) and delete that nexhop entry
* iii\. delete the entry from TABLE SR and delete it from appl_db STATIC_ROUTE_TABLE
* If the route is NOT in the table, done (should not happen)
* If the route is in the table, For each nexthop in the static route entry:
* Look up the TABLE_NEXTHOP to get nexthop entry, remove the prefix from the nexthop entry.
* If there is no prefix in that nexthop entry, delete the corresponding BFD sessions (from redis appl_db and state_db) and delete that nexhop entry
* Delete the entry from TABLE SR and delete it from appl_db STATIC_ROUTE_TABLE


## BFD session state update flow
BfdRouteMgr will be notified if there is any update in state_db BFD_SESSION_TABLE.<br>

* 1\. look up TABLE_BFD, ignore the event if session if not found in local table. otherwise get the nexthop
* 2\. look up TABLE_NEXTHOP using nexthop from step #1, get nexthop entry
* 3\. for each prefix in the nexthop entry, lookup TABLE_SR table to get the static route entry.
* a\. If the BFD session state is UP, and this nexthop is in the static route entry's nexthop list. no action needed, break;
* b\. If the BFD session state is UP, and this nexthop is NOT in the static route entry's nexthop list:
* i\. add this nexthop to the static route entry's nexthop list, set "refreshable": "false", write this static route to redis appl_db STATIC_ROUTE_TABLE.
* c\. If the BFD session state is DOWN, and this nexthop is NOT in the static route entry's nexthop list, no action needed
* d\. If the BFD session state is DOWN, and this nexthop is in the static route entry's nexthop list ,
* i\. delete this nexthop from the static route entry's nexthop list
* 1\. if the static route entry's nexthop list is empty, delete this static route from redis appl_db, break;
* 2\. if the static route entry's nexthop list is NOT empty, write it to redis appl_db STATIC_ROUTE_TABLE with "refreshable"="false".
* 1\. Look up TABLE_BFD, ignore the event if session is not found in local table. Otherwise get the nexthop
* 2\. Look up TABLE_NEXTHOP using nexthop from step #1, get nexthop entry
* 3\. For each prefix in the nexthop entry, lookup TABLE_SR table to get the static route entry.
* If the BFD session state is UP and this nexthop is in the static route entry's nexthop list, no action needed, break;
* If the BFD session state is UP and this nexthop is NOT in the static route entry's nexthop list:
* Add this nexthop to the static route entry's nexthop list, set "refreshable": "false", write this static route to redis appl_db STATIC_ROUTE_TABLE.
* If the BFD session state is DOWN and this nexthop is NOT in the static route entry's nexthop list, no action needed
* If the BFD session state is DOWN and this nexthop is in the static route entry's nexthop list,
* Delete this nexthop from the static route entry's nexthop list
* if the static route entry's nexthop list is empty, delete this static route from redis appl_db, break;
* if the static route entry's nexthop list is NOT empty, write it to redis appl_db STATIC_ROUTE_TABLE with "refreshable"="false".
<br>
<br>

## Table reconciliation after BfdRouteMgr crash/restart
When BfdRouteMgr crashes or restart, it loses all the internal table contents. Need to rebuild the internal tables from redis DB (configuration DB, application DB and state DB), and do cross checking between these tables and redis db to make the information consistent among these table, and then start to process the events received from redis DB.
When BfdRouteMgr crashes or restarts, it loses all the internal table contents. Need to rebuild the internal tables from redis DB (i.e., configuration DB, application DB, state DB, etc), and do cross checking between these tables and redis db to make the information consistent among these tables, and then start to process the events received from redis DB.

### Start event listening
start event listening before the following table and redis DB reconciliation, so the event won't be missing during the reconciliation.
Start event listening before the following table and redis DB reconciliation, so the event won't be missing during the reconciliation.
<br>

### Build TABLE_CONFIG
Expand All @@ -121,22 +122,22 @@ Loop each entry in TABLE_CONFIG, build TABLE_NEXTHOP
### Build TABLE_SR and sync-up with redis appl_db STATIC_ROUTE_TABLE
* 1\. loop TABLE_CONFIG to build TABLE_SR with empty nexthop list
* 2\. read redis state_db BFD_SESSION_TABLE, for each BFD session in TABLE_BFD and the BFD session state is UP, get nexthop
* a\. get prefix list from TABLE_NEXTHOP, and loop this list. For each prefix lookup TABLE_SR to get prefix entry
* i\. add the above nexthop to this prefix's nexthop list
* Get prefix list from TABLE_NEXTHOP, and loop this list. For each prefix lookup TABLE_SR to get prefix entry
* Add the above nexthop to this prefix's nexthop list
* 3\. read redis appl_db STATIC_ROUTE_TABLE, collect all the entries with "refreshable"="false" (BfdRouteMgr created static route entry)
* 4\. loop TABLE_SR table, for each static route, compare with the route in above step #3,
* a\. skip this entry if the static route matches (same nexthop list)
* b\. delete the redis appl_db static route if the nexthop list is empty in TABLE_SR
* c\. update static route in redis appl_db, "refreshable"="false".
* Skip this entry if the static route matches (same nexthop list)
* Delete the redis appl_db static route if the nexthop list is empty in TABLE_SR
* Update static route in redis appl_db, "refreshable"="false".
<br>

### Start event processing
start event processing after the above table and redis DB reconciliation.<br>
Start event processing after the above table and redis DB reconciliation.<br>
In the rare case that, data were read from DB but also get an event later, cause rewrite BFD session or Static Route to appl_DB with same contents should not be a problem.
<br>

## Examples
a few examples, for the cases that adding/deleting static route, and also different prefixes may include same nexthop in their nexthop list.
A few examples for the cases that adding/deleting static route, and also different prefixes may include same nexthop in their nexthop list.

### Add static route for prefix1 with 3 nexthop (nh_a, nh_b and nh_c)

Expand Down

0 comments on commit 3d3170f

Please sign in to comment.