How to handle adding/modifying flows for Extending EVCs

[Ktmi]

One feature of Kytos which we wish to see explored further is standardizing the way in which we extend the functionality of EVCs. EVCs, may be extended to have there traffic mirrored (mirroring), or have telemetry data attached (telemetry_int), or to filter out specific types of traffic. However, at this moment, we have yet to develop a standardized methodology on how to do so. Various methods have different tradeoffs, so this issue is intended as a discussion on how to proceed.

Currently, with telemetry_int, the way we handle extending EVCs is by adding a higher priority flow, which will then attach the telemetry data before it is routed to the next switch. When mef_line updates or disables the EVC, then telemetry_int must handle convergence on the INT flows. However, this has the downside of additional flows, which take up additional space on switches, as well as tear down and installation still needing to be handled by the extending NApp.

An alternative method that's been discussed, is to have the NApp extending the EVC, to modify the EVCs flows. This reduces the total number of flows on the switch, however it makes it more difficult for mef_eline and multiple NApps extending an EVC to coordinate. This does come with the advantage that mef_eline would naturally tear down these extensions with the EVC, when it comes time to update or remove the EVC.

A third method, a sorta hybrid of the above two, is to still use higher priority flows, but to use flow cookies to add context information about the flows, while also making them classified as being part of the EVC. That way, mef_eline can tear them down automatically, while retaining the flexibility of multiple priority levels for extensions to use. This retains the downsize of taking up additional table space, but that ultimately shouldn't be too big a deal.

EDIT:

According to @viniarck, we should be going for the following plan:

Let's try to summarize and settle on what we'll recommend, so far I think we should only support:

1. **Higher priority flows (in new tables if desired) while network handling convergence**.

2. **If it's not a generic service, potentially consider to build or augment as a new feature in the NApp responsible for it**, an example here would be, for instance, the vlan range feature on `mef_eline`. Notice that vlan range weren't created a new NApp to manage mef_eline flows since that one made sense for a given EVC and mef_eline was responsible for it.

Now regarding the downsides of handling network convergence with higher priority flows:

* Notice that not all the time the flows being handled are necessarily in the hot path. The worst/critical case is `telemetry_int` since it's "shadowing" the service path entirely end-to-end from the ingress part, so when the less priority flows change, consequently the higher ones need to do it similarly. But, if it's not "shadowing" the entire path, then not necessarily it impacts on the failover for example, so it's just a matter of traditional clean up and installing again (where/if applicable, if that flows need to move at all)

See this topology here for an example:

* If a NApp is only dealing with blocking ingress flows in an UNI and not even output ports, then it's even simpler, since no matter what it won't change.

* If a NApp is mirroring traffic in a link, doing so in UNIs are simpler too, but even if they need to it on NNIs, if you're monitoring one segment/link, then it's just a matter of subscribing to the service and making sure that that NApp is matching correctly the service vlans if you just want to also mirror one EVC, but if the mirror service also doesn't care about which service(s), and just work at a link ingress level, then it's just a matter of making sure that the input/output ports are being followed correctly. Also, if the link only has one potential egress that becomes simpler too.

The general thing is, if you're dealing with higher priority flows, you need to be aware of how network convergence works, but then if you strategically only deal with certain paths (or sometimes if you're only working at a interface or link level) the NApp can be simpler. Higher priority flows also compose well, and considering how the platform has evolve this is the least disruptive way of doing so. Optimizing for flow removals can be discussed too, but I don't expect that removals would be a bottle neck. If fact there are many other optimizations at mef_eline that we still didn't prioritize such as reusing existing paths (but that's a major undertaking that likely won't be prioritized in the medium term). Higher priority flows will still keep most of the NApps responsibilities independently, facilitating for potential consistency checks. It should be a core part of the design to ensure that you aim for a reasonable convergence, that's part of writing network applications, and then you measure what users can expect for your NApp, if it's not good enough then you can rethink about the design, and/or consider whether or not it needs to be a feature in the main NApp in the first place.

[viniarck]

Adjacent cookies removals only work well (for one part of the removal convergence) when flow matches are equal or when it's removing entirely from a switch. Example where it doesn't work well is when only a set is being removed (current or failover path), and the other NApp might also have different flow matches (pretty rare case, but that's telemetry_int for example). So, good to document when it's not suitable too.

[viniarck]

Fourth method (what we currently have a general rule): don't change other NApps flows. If flows need to be augmented but there's a very closely tight coupled functionality (and it's not a generic type of relationship), it can potentially also just be a feature in the NApp. So, in this case users can propose a new feature on a NApp, and if that's considered useful without bloating too much but simplifying things than it might be accepted, otherwise they should stick with other solutions. telemetry_int could be this way, it could've been entirely implemented on mef_eline for example. This also implies that when extending it to be done in a modular way (but then also finding a balance and questioning if it might be too big to be just a feature or not), this is kind of a feature flag type of extension.

[italovalcy]

@viniarck, if I understood correctly, the suggestion here is to duplicate the feature implementation in the Napp, right? For instance, failover convergence will be handled by telemetry_int and eventually any other Napp besides being handled by mef_eline, right?

Please note that creating new flows is already part of the new Napp responsibilities; that is fine. But the main point here is fast reacting to network changes to avoid service disruption. I think only mef_eline should concentrate on that. Other Napps will react to the new EVC current_path accordingly when it gets deployed.

My idea here would be something like the following workflow:

1. The network operator creates the EVC on mef_eline (circuit_id=123)
2. mef_eline provision the current_path and failover_path, flows are created on the switches with cookie_id=0xaa123
3. The network operator enables telemetry_int or mirroring on the circuit_id=123
4. telemetry_int or mirroring Napps gets the current_path from mef_eline and creates new flows with higher priority and cookie_id=0xad123
5. telemetry_int or mirroring notifies mef_eline saying: "hey the circuit_id=123 was extended to have additional flows with cookie_id=0xad123, please take note"
6. When a link failure occurs, mef_eline will identify circuits with failover_path and not. For circuits with failover_path, mef_eline will send two flowmod removal on the UNI switches with cookie_id=0xad123/0xfffff + two flow_mod to change the ingress traffic leveraging the failover_path pre-provisioned (the order here can change, not sure which one is the best: fist remove then modify, or first modify then remove)
7. mef_eline will generate KytosEvents to inform a circuit was redeployed
8. telemetry_int or mirroring upon receiving the KytosEvent from step 7, will clean up the flows used on the old path (also based on the flowmod removal with cookie_id=0xad123/0xfffff) and then establish a new path based on the new evc's current_path
9. mef_eline's consistency check has two situations: flows validation (currently being discussed) and sdntrace validation. For flows validation, that won't change because mef_eline flows will remain untouched; for sdntrace, changes will be required.

[viniarck]

the suggestion here is to duplicate the feature implementation in the Napp, right? For instance, failover convergence will be handled by telemetry_int and eventually any other Napp besides being handled by mef_eline, right?

No. This suggestion here is that for the feature to be implemented on NApp who owns the flows, just so it still own them all without any other NApps changing them (by changing here I mean overwriting existing flows actions). For instance, let's take this future mirroring example, it could be a feature on mef_eline, where mef_eline would also be apple to parametrize in an EVC where mirror happens (also accepting updates in new endpoints). But, this is only suitable for features that are not generic over any kind of transport service and aren't expected to be used by many other NApps. Otherwise, it'd duplicate functionality, and it wouldn't be suitable or viable to maintain. If mirroring were to be used on both mef_eline and of_l2ls (or any other new NApp that we expect to have in the future which will also provision paths that need to be mirrored), then only being a feature of mef_eline wouldn't be suitable for example. If one of the goals of mirroing is to be able to mirror any flows over a given link then it'd be a new NApp makes a lot of sense.

failover convergence will be handled by telemetry_int and eventually any other Napp besides being handled by mef_eline, right?

NApps who are using higher priority flows, managing their own flows, that also cover network paths, yes. NApps who don't need to handle convergence on NNIs don't need to handle them all #471 (comment)

[viniarck]

@italovalcy, the idea and steps you pointed out it's also a solution and makes a lot of sense (and it's the one we're using on telemetry_int as you know), that one being the solution where each NApp own their own flows with higher priorities.

[viniarck]

General comment about network convergence events:

• If your NApp doesn't need to change NNI related flows, flows associated to a network path and has a 1-to-1 relationship with UNI flows, and it only installs higher priority flows, then it becomes simpler to handle the rest of network related events, since that part is already a single point of failure, there's no other network path optimizations there (other than masked wildcard flows removals), and then the NApp might only need to handle interface going up or down (if applicable to represent any sort of status) and/or other services events related to that UNI (like the service being disabled or deleted and if the flows depend on other services). So, if the scope of your NApp only needs to deal with UNI flows it can simplify other challenges. When designing your NApp, if only working on UNI interfaces is sufficient for the use cases then you also minimize other challenges and/or the solutions you can pick.

Examples: A NApp that blocks traffic with higher priority flows in a given UNI.

[viniarck]

A point we'll need to sort out to see which solutions/options will be supported is how mef_eline consistency will evolve. One of the biggest challenges there is that it'll inevitably need to compare expected runtime flows with that's in the DB (since there's orphan flows or other unhandled corner cases that might leave something behind in an extreme case), and that's already a problem that requires concurrency safety to avoid false positives to impact on unwanted network convergence, this can become hairy real quick to also ensure concurrency safety from other NApps, although solvable if keeping track wish pub/sub about the modifications, but then again it'd require also persistence which in turn can also have original problems of potential failures in worst extreme cases (so kind of only moving part of the problem to a different place). Generally speaking, I don't think it's a good idea to split flows ownership changing existing flows from a different NApp, especially with the new requirements with extra consistencies checks. We'll need provide good guidelines when it also needs to be a feature on an existing NApp or a new associated higher priority service/resource (like telemetry is as of May 2024).

[italovalcy]

I think we should not concentrate on the number of flows at this point. Even though this is very important, migrating to P4 will bring many new possibilities for optimization in this sense in the near future, so optimizing this now wouldn't benefit us much.

[viniarck]

Agreed. Many things will change with P4 too, including data models, although many will likely be equivalent-ish where applicable.

As of now, I suppose what we can add as a non functional requirement, Italo, is that for NApps who need to also ensure fast convergence is to provide metrics on how well and at which scalability it's been tested (and then who will use the NApp can make a judgement if it's suitable for their use cases - and if it's not, then keep evolving it to meet the requiements). If let's say it's handling well concurrently while not add too much extra latency let's say two or low three digits of ms latency, then it's reasonable. And then if that doesn't cut it for production usage (assuming that the goal is indeed in prod and not just an experiment proof of concept), then back to the drawing board to see where it can be optimized/moved/changed. Also, not every NApp as well need to be handling convergence optimally all the time, I don't know yet how you plan to use mirroring, but if it's something that is only enabled for some time temporarily and won't be constantly in the data path, then it might not bee too bad if it's not optimal to the full extent, although it's desirable to be where possible.

[Ktmi]

For EVCs, the switch is doing two things, identifying what service the packet is part of, then update any routing headers and send it off to the next switch, or its destination. What if we instead have the EVC opts into these extensions, by telling the switch which of these extensions it opts into.

To do so, we can use packet metadata. We have the first table identify the EVC, and add metadata containing an identifier, which will later be used for outputing the packet, plus flags for the various types of extensions to apply to the packet. The extensions, which the EVC subscribes to, always exist separate from the EVC. This means that extensions don't need any cleanup when it comes to removing any specific EVC, as the flows for the extension are meant for all EVCs on that switch.

If there is an extension that needs to know about a specific EVC in order to work correctly, then it can use that EVC's identifier for that switch.

[Ktmi]

So if we where to implement this, the way it should work is that:

1. A feature providing NApp allocates a flag on a switch, and installs a feature to use that flag on the switch on a feature table.
2. The NApp advertises that the feature is available on the switch by using the given flag.
3. When an EVC is created, part of the options will be a set of features, as strings, which can then be mapped to each switch's flags.
4. When the EVC is installed it installs a flow to a capture table, and a flow to a routing table. The flow on the capture table identifies the flow as being part of the evc.

As for table allocations, I mention three in the above explanation, but it could be more. Now that I think about it you would need a new table for each feature...

[Ktmi]

As for how this would work with P4, its fundamentally not that far off from what I've already written. The switch manager, would subscribe to have the feature, which then the NApp responsible for the feature would provide either a segment of P4 code, or something similar in order to build the P4 program.

[viniarck]

This is far off from how the current design and NApps responsibilities, each NApp flows provision their flows without core shared state or required extra subscriptions. Sharing state via core can be acceptable in certain cases, but here, it's forcing that would require extra persistence in that one too, while also fundamentally changing how it's computed. At the moment, I don't think this help much as a solution that wouldn't require too much changes or refactoring.

[Ktmi]

The thing is, I don't necessarily understand how you reach convergence succinctly with other implementations. The problem with implementing higher priority flows is that they override the lower priority flows. In the case of multiple NApps extending, this means that if you have one NApp overriding to provide feature A, and another with a higher priority to provide feature B, then you only get feature B. Even if you have the NApps identify lower priority flows to integrate their features into their own, you will run into a race condition when both try to add their flows at the same time.

[viniarck]

A higher priority flow "A" won't overwrite a lower "B" one, a switch table lookup is ordered and executed by priority and their match. Looks like you're referring to a case where the flow would be overwritten (same priority same match), with high priorities the main point is to not overwrite in the first place. There's more information also on this comment here #471 (comment) for more information. Let me know if you meant something else.

[viniarck]

Let's try to summarize and settle on what we'll recommend, so far I think we should only support:

1. Higher priority flows (in new tables if desired) while network handling convergence.
2. If it's not a generic service, potentially consider to build or augment as a new feature in the NApp responsible for it, an example here would be, for instance, the vlan range feature on mef_eline. Notice that vlan range weren't created a new NApp to manage mef_eline flows since that one made sense for a given EVC and mef_eline was responsible for it.

Now regarding the downsides of handling network convergence with higher priority flows:

• Notice that not all the time the flows being handled are necessarily in the hot path. The worst/critical case is telemetry_int since it's "shadowing" the service path entirely end-to-end from the ingress part, so when the less priority flows change, consequently the higher ones need to do it similarly. But, if it's not "shadowing" the entire path, then not necessarily it impacts on the failover for example, so it's just a matter of traditional clean up and installing again (where/if applicable, if that flows need to move at all)

See this topology here for an example:

• If a NApp is only dealing with blocking ingress flows in an UNI and not even output ports, then it's even simpler, since no matter what it won't change.
• If a NApp is mirroring traffic in a link, doing so in UNIs are simpler too, but even if they need to it on NNIs, if you're monitoring one segment/link, then it's just a matter of subscribing to the service and making sure that that NApp is matching correctly the service vlans if you just want to also mirror one EVC, but if the mirror service also doesn't care about which service(s), and just work at a link ingress level, then it's just a matter of making sure that the input/output ports are being followed correctly. Also, if the link only has one potential egress that becomes simpler too.

The general thing is, if you're dealing with higher priority flows, you need to be aware of how network convergence works, but then if you strategically only deal with certain paths (or sometimes if you're only working at a interface or link level) the NApp can be simpler. Higher priority flows also compose well, and considering how the platform has evolve this is the least disruptive way of doing so. Optimizing for flow removals can be discussed too, but I don't expect that removals would be a bottle neck. If fact there are many other optimizations at mef_eline that we still didn't prioritize such as reusing existing paths (but that's a major undertaking that likely won't be prioritized in the medium term). Higher priority flows will still keep most of the NApps responsibilities independently, facilitating for potential consistency checks. It should be a core part of the design to ensure that you aim for a reasonable convergence, that's part of writing network applications, and then you measure what users can expect for your NApp, if it's not good enough then you can rethink about the design, and/or consider whether or not it needs to be a feature in the main NApp in the first place.