feat: added support for batching FlowMods

[viniarck]

Closes #171

Summary

• See updated changelog file
• Version of the API wasn't bumped since it'll need to be bumped in the future for the rest of validations on v3, so we're saving a bit of refactoring on endpoint parametrization, since it's breaking compatibility it's a reasonable tradeoff for now.

Local Tests

• Made a request with force: false with batching, it didn't block the request

2023-08-31 16:42:52,272 - INFO [kytos.napps.kytos/flow_manager] (AnyIO worker thread) Send FlowMod from request dpid: 00:00:00:00:00:00:00:01, command: add, force: False, batch_size: 2
, batch_interval: 5, flows[0, 3]: [{'priority': 4000, 'cookie': 100, 'match': {'in_port': 1, 'dl_vlan': 1}, 'actions': [{'action_type': 'output', 'port': 2}]}, {'priority': 4000, 'cook
ie': 100, 'match': {'in_port': 1, 'dl_vlan': 2}, 'actions': [{'action_type': 'output', 'port': 2}]}, {'priority': 4000, 'cookie': 100, 'match': {'in_port': 1, 'dl_vlan': 3}, 'actions':
 [{'action_type': 'output', 'port': 2}]}]
2023-08-31 16:42:52,281 - INFO [kytos.napps.kytos/flow_manager] (AnyIO worker thread) Sending FlowMods slice[0: 2], iteration: 0, flows[0, 2]: [{'switch': '00:00:00:00:00:00:00:01', 't
able_id': 0, 'match': {'in_port': 1, 'dl_vlan': 1}, 'priority': 4000, 'idle_timeout': 0, 'hard_timeout': 0, 'cookie': 100, 'id': '2e371cca3dcfe383a5058c4410920c1e', 'stats': {}, 'cooki
e_mask': 0, 'instructions': [{'instruction_type': 'apply_actions', 'actions': [{'port': 2, 'action_type': 'output'}]}]}, {'switch': '00:00:00:00:00:00:00:01', 'table_id': 0, 'match': {
'in_port': 1, 'dl_vlan': 2}, 'priority': 4000, 'idle_timeout': 0, 'hard_timeout': 0, 'cookie': 100, 'id': '8f4ebd12d31dba67b36dadfad7ba2c84', 'stats': {}, 'cookie_mask': 0, 'instructio
ns': [{'instruction_type': 'apply_actions', 'actions': [{'port': 2, 'action_type': 'output'}]}]}]
2023-08-31 16:42:52,282 - INFO [kytos.napps.kytos/flow_manager] (AnyIO worker thread) Send FlowMods batching will sleep for 5 seconds before sending slice[2:2]
kytos $>                                                                                                                                                                                

kytos $> 2023-08-31 16:42:57,285 - INFO [kytos.napps.kytos/flow_manager] (AnyIO worker thread) Sending FlowMods slice[2: 4], iteration: 1, flows[0, 1]: [{'switch': '00:00:00:00:00:00:0
0:01', 'table_id': 0, 'match': {'in_port': 1, 'dl_vlan': 3}, 'priority': 4000, 'idle_timeout': 0, 'hard_timeout': 0, 'cookie': 100, 'id': '6f614c8654066e827cce5dc6b3e785cb', 'stats': {
}, 'cookie_mask': 0, 'instructions': [{'instruction_type': 'apply_actions', 'actions': [{'port': 2, 'action_type': 'output'}]}]}]
2023-08-31 16:42:57,291 - INFO [uvicorn.access] (MainThread) 127.0.0.1:52294 - "POST /api/kytos/flow_manager/v2/flows/00%3A00%3A00%3A00%3A00%3A00%3A00%3A01 HTTP/1.1" 202

• Made a request with force: true with batching, it blocked the request as expected

kytos $> 2023-08-31 16:43:14,817 - INFO [kytos.napps.kytos/flow_manager] (AnyIO worker thread) Send FlowMod from request dpid: 00:00:00:00:00:00:00:01, command: add, force: True, batch
_size: 2, batch_interval: 5, flows[0, 3]: [{'priority': 4000, 'cookie': 100, 'match': {'in_port': 1, 'dl_vlan': 1}, 'actions': [{'action_type': 'output', 'port': 2}]}, {'priority': 400
0, 'cookie': 100, 'match': {'in_port': 1, 'dl_vlan': 2}, 'actions': [{'action_type': 'output', 'port': 2}]}, {'priority': 4000, 'cookie': 100, 'match': {'in_port': 1, 'dl_vlan': 3}, 'a
ctions': [{'action_type': 'output', 'port': 2}]}]
2023-08-31 16:43:14,852 - INFO [uvicorn.access] (MainThread) 127.0.0.1:45808 - "POST /api/kytos/flow_manager/v2/flows/00%3A00%3A00%3A00%3A00%3A00%3A00%3A01 HTTP/1.1" 202
2023-08-31 16:43:14,857 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_app_12) Sending FlowMods slice[0: 2], iteration: 0, flows[0, 2]: [{'switch': '00:00:00:00:00:00:00:01', 'ta
ble_id': 0, 'match': {'in_port': 1, 'dl_vlan': 1}, 'priority': 4000, 'idle_timeout': 0, 'hard_timeout': 0, 'cookie': 100, 'id': '2e371cca3dcfe383a5058c4410920c1e', 'stats': {}, 'cookie
_mask': 0, 'instructions': [{'instruction_type': 'apply_actions', 'actions': [{'port': 2, 'action_type': 'output'}]}]}, {'switch': '00:00:00:00:00:00:00:01', 'table_id': 0, 'match': {'
in_port': 1, 'dl_vlan': 2}, 'priority': 4000, 'idle_timeout': 0, 'hard_timeout': 0, 'cookie': 100, 'id': '8f4ebd12d31dba67b36dadfad7ba2c84', 'stats': {}, 'cookie_mask': 0, 'instruction
s': [{'instruction_type': 'apply_actions', 'actions': [{'port': 2, 'action_type': 'output'}]}]}]
2023-08-31 16:43:14,872 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_app_12) Send FlowMods batching will sleep for 5 seconds before sending slice[2:2]
kytos $>                                                                                                                                                                                

kytos $> 2023-08-31 16:43:19,878 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_app_12) Sending FlowMods slice[2: 4], iteration: 1, flows[0, 1]: [{'switch': '00:00:00:00:00:00:00
:01', 'table_id': 0, 'match': {'in_port': 1, 'dl_vlan': 3}, 'priority': 4000, 'idle_timeout': 0, 'hard_timeout': 0, 'cookie': 100, 'id': '6f614c8654066e827cce5dc6b3e785cb', 'stats': {}
, 'cookie_mask': 0, 'instructions': [{'instruction_type': 'apply_actions', 'actions': [{'port': 2, 'action_type': 'output'}]}]}]

• Sent event with 100 flows with batching, it worked as expected (I truncated the output here):

kytos $> def create_flows(dpid: str, n: int) -> dict[list]: 
    ...:     """docstring.""" 
    ...:     flows = [ 
    ...:         { 
    ...:             "priority": i, 
    ...:             "cookie": i, 
    ...:             "match": {"in_port": 1, "dl_vlan": i}, 
    ...:             "actions": [{"action_type": "output", "port": 2}], 
    ...:         } 
    ...:         for i in range(1, n + 1) 
    ...:     ] 
    ...:     return { 
    ...:         "flow_dict": { 
    ...:             "flows": flows,
    ...:         }, 
    ...:         "dpid": dpid, 
                 "force": True, 
    ...:         "batch_interval": 1, 
    ...:         "batch_size": 50
    ...:     } 
    ...:  
    ...:  
    ...: data = create_flows(dpid, 100) 
    ...: ev = KytosEvent("kytos.flow_manager.flows.install", content=data) 
    ...: controller.buffers.app.put(ev)                                                                                                                                                 

2023-08-31 16:49:04,584 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_app_19) Sending FlowMods slice[0: 50], iteration: 0, flows[0, 50]: ...
2023-08-31 16:49:04,660 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_app_19) Send FlowMods batching will sleep for 1 seconds before sending slice[50:50]
kytos $>                                                                                                                                                                                

kytos $> 2023-08-31 16:49:05,662 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_app_19) Sending FlowMods slice[50: 100], iteration: 1, flows[0, 50]: ...

End-to-End Tests

tests/test_e2e_01_kytos_startup.py ..                                    [  0%]
tests/test_e2e_05_topology.py ..................                         [  8%]
tests/test_e2e_10_mef_eline.py ..........ss.....x.....x................  [ 24%]
tests/test_e2e_11_mef_eline.py ......                                    [ 27%]
tests/test_e2e_12_mef_eline.py .....Xx.                                  [ 30%]
tests/test_e2e_13_mef_eline.py .....xs.s......xs.s.XXxX.xxxx..X......... [ 47%]
...                                                                      [ 48%]
tests/test_e2e_14_mef_eline.py x                                         [ 49%]
tests/test_e2e_15_mef_eline.py ..                                        [ 50%]
tests/test_e2e_20_flow_manager.py .....................                  [ 58%]
tests/test_e2e_21_flow_manager.py ...                                    [ 60%]
tests/test_e2e_22_flow_manager.py ...............                        [ 66%]
tests/test_e2e_23_flow_manager.py ..............                         [ 72%]
tests/test_e2e_30_of_lldp.py ....                                        [ 73%]
tests/test_e2e_31_of_lldp.py ...                                         [ 75%]
tests/test_e2e_32_of_lldp.py ...                                         [ 76%]
tests/test_e2e_40_sdntrace.py .............                              [ 81%]
tests/test_e2e_41_kytos_auth.py ........                                 [ 85%]
tests/test_e2e_50_maintenance.py ........................                [ 95%]
tests/test_e2e_60_of_multi_table.py .....                                [ 97%]
tests/test_e2e_70_kytos_stats.py .......                                 [100%]
=============================== warnings summary ===============================
------------------------------- start/stop times -------------------------------
= 219 passed, 6 skipped, 11 xfailed, 5 xpassed, 867 warnings in 11382.39s (3:09:42) =

[Ktmi]

Everything seems to work as expected, beyond allowing multiple overlapping requests to execute.

[Ktmi]

Nothing is stopping multiple requests from sending in flow_mods to the same switch, which could potentially overload the switch. Perhaps instead of performing batching here, we could use a per switch queue for the flow_mods received to rate limit.
Additionally, batch size doesn't make much sense as a per request parameter, but makes sense as a per switch parameter.

[viniarck]

Yes, @Ktmi that's a good idea. We might as well go ahead and start queueing per switch on flow_manager. Initially this was just adding the same client per request behavior.

It's worth also noting that we also have in our backlog to have per message type to support rate limiting on core queue but hasn't been prioritized kytos-ng/kytos#245

[viniarck]

@Ktmi check out this thread kytos-ng/kytos#245 (comment), I'll need your help with this one if we'll go for the ideal solution, appreciated your review, help, and suggestions. In the meantime, I'll keep making progress on telemetry_int. Otherwise, I can still handle it in the future. If you confirm that you can help out with that one on version 2023.2, of course following the current planned priorities, we can close this PR here. Let me know.

[viniarck]

Closing this PR in favor of kytos-ng/kytos#412. David is helping out with it. Thanks, David.