EKF2 stops, causing a flight termination

[Thijs----]

Describe the bug
During flight, the ekf2 module stops publishing all messages, causing a flight termination.
It is remarkable that before it's stops publishing, it publishes one more time all messages at the exact same moment and with exact same timestamp.
Sensor data (sensor_combined, vehicle_magnetometer, etc) keeps publishing as expected.

To Reproduce
This occurred twice, no idea if it's possible to trigger it.

Expected behavior
ekf2 should keep working en publishing it's messages.

Log Files and Screenshots

• first log (17:28)
• second log (8:12)
  How can I delete location (gps) information before uploading the Ulogs?

Screenshot from plotted timestamps (sensors + ekf2), added delay to view them separately.

Drone:

• generic quad-copter, with onboard companion computer
• Pixhawk 1
• Firmware PX4 v10.0.0 with small modifications

Additional context
Flight is mostly done in auto modes (takeoff, loiter, mission)

[Thijs----]

I just noticed in both log files that CPU & RAM (cpuload msg) also stops publishing, this happens before the ekf2 stops.
In the first log ~3 minutes before (14:26), in the second ~20 s (7:52).
In my other log files this does not happen.

Edit:
camera_trigger stops also with cpuload,
vehicle_land_detected stops with ekf.

So it seems that the ekf2 isn't really the problem. But why are those modules stopping?

[bkueng]

Hi

Can you provide the changes you made? It's impossible to asses w/o knowing what was changed in the code.
Alternatively you can try to reproduce the problem with a commit from upstream.

[Thijs----]

Hi,

Sure, two remarkable changes have been made:

1. I added an extra driver. This driver is based on px4flow driver. The sensor is called exopulse and measures distances between itself and another exopules and sends it via i2c. This distance I use in the landing_target_estimator.
   To use this driver and precision_landing, I added the following to ROMFS/px4fmu_common/init.d/rcS:

	# Check for irlock sensor - Exo
	irlock start &
	# Check for exopulse sensor - Exo
	exopulse start &
	# Start up Landing Target Estimator module - Exo
	landing_target_estimator start &

2. I added a new camera_trigger interface. On triggering it publishes to actuator_controls_2 (gimbal), so I can use it in a mixer file to assign an aux output for triggering (instead of using aux 5 and 6).

[bkueng]

Any added code can interfere with the whole system (e.g. if there are memory bugs).
Is it something you want to contribute? Might be worth opening a pull request then, so others can have a look.
Otherwise I suggest you try to reproduce the issue on master.