Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: parametrized timeout on KytosEventBuffer.put and exposing buffers queue size config #429

Merged
merged 11 commits into from
Jan 25, 2024

Conversation

viniarck
Copy link
Member

@viniarck viniarck commented Nov 2, 2023

Closes #423

This PR is on top of #438 since it depends on event_buffer_conf.

Summary

See updated changelog file (it also explains how to use it)

Local Tests

  • I ran kytosd with my local env, still worked as it used to.

  • I've also explored configurations on kytos.conf it's behaving as expected

  • I then simulated a NApp misbehaving by sending too many events during setup() with mef_eline. Notice below that you can see exactly which producer couldn't put the event anymore, and then in the tracebacks it also includes who may events it had, which other events ended up filling the queue, in this case ``kytos/mef_eline.xyz'` had more than a 1000:

Full KytosEventBuffers counters: {'app': Counter({'kytos/mef_eline.xyz': 1003, 'kytos/topology.port.created': 19, 'kytos/topology.topology_loaded': 1, 'kytos/mef_eline.evcs_loaded': 1})
Kytos couldn't start because of KytosNAppSetupException: NApp kytos/mef_eline exception   Traceback (most recent call last):
  File "/home/viniarck/repos/kytos/kytos/core/controller.py", line 817, in load_napp
    napp = napp_module.Main(controller=self)
  File "/home/viniarck/repos/kytos/kytos/core/napps/base.py", line 194, in __init__
    self.setup()
  File "/home/viniarck/repos/napps/napps/kytos/mef_eline/main.py", line 72, in setup
    self.send_events(1024)
  File "/home/viniarck/repos/napps/napps/kytos/mef_eline/main.py", line 76, in send_events
    self.controller.buffers.app.put(KytosEvent("kytos/mef_eline.xyz"), timeout=1)
  File "/home/viniarck/repos/kytos/kytos/core/buffers/buffers.py", line 40, in put
    self._queue.sync_q.put(event, timeout=timeout)
  File "/home/viniarck/repos/kytos/.direnv/python-3.9/lib/python3.9/site-packages/janus/__init__.py", line 388, in put
    raise SyncQueueFull
queue.Full

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/viniarck/repos/kytos/kytos/core/controller.py", line 263, in start
    self.start_controller()
  File "/home/viniarck/repos/kytos/kytos/core/controller.py", line 360, in start_controller
    self.load_napps()
  File "/home/viniarck/repos/kytos/kytos/core/controller.py", line 852, in load_napps
    self.load_napp(napp.username, napp.name)
  File "/home/viniarck/repos/kytos/kytos/core/controller.py", line 820, in load_napp
    raise KytosNAppSetupException(msg) from exc
kytos.core.exceptions.KytosNAppSetupException: KytosNAppSetupException: NApp kytos/mef_eline exception  

Full KytosEventBuffers counters: {'app': Counter({'kytos/mef_eline.xyz': 1003, 'kytos/topology.port.created': 19, 'kytos/topology.topology_loaded': 1, 'kytos/mef_eline.evcs_loaded': 1})
}
Shutting down Kytos...
[1]    36965 terminated  kytosd -f -n /home/viniarck/repos/napps/napps -E --database mongodb

End-to-End Tests

I'll dispatch e2e execution with this branch and the other NApps PR which are using timeout=1 when putting events during setup():

+ python3 -m pytest tests/ --reruns 2 -r fEr
============================= test session starts ==============================
platform linux -- Python 3.9.2, pytest-7.2.0, pluggy-1.3.0
rootdir: /builds/amlight/kytos-end-to-end-tester/kytos-end-to-end-tests
plugins: rerunfailures-10.2, timeout-2.1.0, anyio-3.6.2
collected 255 items
tests/test_e2e_01_kytos_startup.py ..                                    [  0%]
tests/test_e2e_05_topology.py ...................                        [  8%]
tests/test_e2e_10_mef_eline.py ..........ss.....x.....x................  [ 23%]
tests/test_e2e_11_mef_eline.py ......                                    [ 26%]
tests/test_e2e_12_mef_eline.py .....Xx.                                  [ 29%]
tests/test_e2e_13_mef_eline.py ....Xs.s.....Xs.s.XXxX.xxxx..X........... [ 45%]
.                                                                        [ 45%]
tests/test_e2e_14_mef_eline.py x                                         [ 46%]
tests/test_e2e_15_mef_eline.py .....                                     [ 48%]
tests/test_e2e_20_flow_manager.py .....................                  [ 56%]
tests/test_e2e_21_flow_manager.py ...                                    [ 57%]
tests/test_e2e_22_flow_manager.py ...............                        [ 63%]
tests/test_e2e_23_flow_manager.py ..............                         [ 69%]
tests/test_e2e_30_of_lldp.py ....                                        [ 70%]
tests/test_e2e_31_of_lldp.py ...                                         [ 71%]
tests/test_e2e_32_of_lldp.py ...                                         [ 72%]
tests/test_e2e_40_sdntrace.py ..............                             [ 78%]
tests/test_e2e_41_kytos_auth.py ........                                 [ 81%]
tests/test_e2e_42_sdntrace.py ..                                         [ 82%]
tests/test_e2e_50_maintenance.py ........................                [ 91%]
tests/test_e2e_60_of_multi_table.py .....                                [ 93%]
tests/test_e2e_70_kytos_stats.py ........                                [ 96%]
tests/test_e2e_80_pathfinder.py ss......                                 [100%]
=============================== warnings summary ===============================
../../../../usr/local/lib/python3.9/dist-packages/kytos/core/config.py:235
------------------------------- start/stop times -------------------------------
= 231 passed, 8 skipped, 9 xfailed, 7 xpassed, 1109 warnings in 12224.88s (3:23:44) =

@viniarck viniarck requested a review from a team as a code owner November 2, 2023 17:41
@viniarck viniarck marked this pull request as draft November 2, 2023 17:41
@viniarck viniarck changed the base branch from master to feat/event_buffer_config January 17, 2024 15:52
@viniarck viniarck marked this pull request as ready for review January 19, 2024 14:53
Base automatically changed from feat/event_buffer_config to master January 24, 2024 20:56
@viniarck viniarck merged commit 7a2afa8 into master Jan 25, 2024
2 checks passed
@viniarck viniarck deleted the feat/qsize branch January 25, 2024 13:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

MainThread can get blocked when enqueuing events if the queue capacity is reached
1 participant