investigate px4fmu-v5 cpu spikes #9852
Conversation
|
Logging on the bench. https://logs.px4.io/plot_app?log=317643e0-7cb3-4d4a-87ba-dd1ade7097f4
|
|
Questions
|
We are not running round-robin. So I think is is not an issue. We do not terminate on system clock ticks. So something running long is logged as such.
Why not? Can we at mode to top to track and log? |
I meant in a we could live with on by default. The concern being triggering something relatively expensive as a result of abnormally high cpu usage. |
src/modules/load_mon/load_mon.cpp
Outdated
| @@ -207,26 +190,31 @@ void LoadMon::_compute() | |||
| } | |||
|
|
|||
| /* compute system load */ | |||
| const hrt_abstime interval_idletime = system_load.tasks[0].total_runtime - _last_idle_time; | |||
| const float interval = hrt_elapsed_time(&_last_idle_time_sample); | |||
There was a problem hiding this comment.
I would try to keep to integer math as much as possible.
src/modules/load_mon/load_mon.cpp
Outdated
| // Run it at 1 Hz. | ||
| const unsigned LOAD_MON_INTERVAL_US = 1000000; | ||
| // Run it at 10 Hz. | ||
| const unsigned LOAD_MON_INTERVAL_US = 100000; |
There was a problem hiding this comment.
Good for bench-testing but I would not increase it in general.
We can add spike-detection instrumentation to the top command. Then we can also do something that is a bit more expensive.
dagar
force-pushed
the
pr-loadmon
branch
5 times, most recently
from
9e98fa9 to
d397720
Compare
|
@PX4/testflights could you please do a quick flight test on any px4fmu-v5 platform with a comparison to master? The only difference should be in the cpu logging on flight review. |
dagar
changed the title
|
Tested on Pixhawk 4 mini. good flight, there is no difference with the master. |
|
Thanks @Avysuarez Still seeing cpu spikes. @davids5
|
dagar
changed the title
dagar
added
bug
Board: Pixhawk fmu-v5
|
These don't seem to be present in similar stm32f4 logs. |
|
Here's a possibly interesting data point. The AV-X board (another stm32f7 board) doesn't appear to suffer from this. In this test flight it wasn't using i2c for anything. https://review.px4.io/plot_app?log=273dd3d5-c530-4ab1-bdac-f36b8a816421
|
|
@dagar that would confirm the suspicion I had. I will instrument it today. |
|
@dagar I started looking at this and no obvious smoking gun on the I2C. More on Monday,... |
|
Seems a bit more inconsistent than I initially thought. Here are 2 more logs from earlier today on a pixhawk 4 and 4 mini. Otherwise these are nearly identical setups, flying the exact same firmware revision, same vehicle, same place, same time. Here's a pixhawk 4 mini flight from today. https://review.px4.io/plot_app?log=8b8adf6b-0bcb-49c0-a684-0f23685c4e71 Here's a pixhawk 4 flight from today. https://review.px4.io/plot_app?log=a01ee82e-fe67-4755-98d8-649cc51ecd02 EDIT: I went through a random sampling of old flights for both of those boards and the presence of cpu spikes seems consistent per board. |
|
Related #11013 (comment) |
|
@davids5 good to merge? The only functional change here is using the actual measured interval rather than the LOAD_MON_INTERVAL_US constant for calculating the load. |
There was a problem hiding this comment.
@dagar - looks good, I think there are 2 schools of thought on the task_stack_info becoming automatic. It has all the info in one place that is now not persistent. But if needed the we can make it static to see the deltas in the debugger so this is an OK change.
|
@dagar are we still seeing the spikes post nuttx upgrade? |
For fmu-v5 testing.