Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: topology with 100+ evcs usually spike CPU up to 100% during network convergence #478

Open
viniarck opened this issue May 24, 2024 · 1 comment
Labels
epic_high_cpu_usage future_release Planned for the next release priority_medium Medium priority

Comments

@viniarck
Copy link
Member

viniarck commented May 24, 2024

kytosd is spiking up to 100% CPU usage (measured on a i7-12700H 2.3 GHz)

Locally, I'm using a topology with 100 EVCs a performing one link flap on both current_path and in the failover_path, during flow mod removals and installations it's spiking up to 100% during that period, this is supposed to be mostly IO-bound, this needs investigation to profile and see if we can spot the main sources cause that, maybe msg serialization/deserialization might be struggling, still needs to be profiled:

cpu

cpu1

On APM these were the top ranked traced handlers running at the time:

20240524_121259

@viniarck viniarck added priority_medium Medium priority future_release Planned for the next release labels May 24, 2024
@Ktmi
Copy link

Ktmi commented Jun 10, 2024

So it took some effort to recreate, but here's what I got. First I created a few hundred evcs on Kytos, then I deleted all the created flows on open vswitch. I then restarted kytos, with consistency check disabled on flow manager, to allow for resending of all stored flows to execute. This then causes a brief period where the kytos process is at 100% cpu utilization.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
epic_high_cpu_usage future_release Planned for the next release priority_medium Medium priority
Projects
None yet
Development

No branches or pull requests

2 participants