Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory and mavlink problems pixhawk v2 #12502

Open
tuloski opened this issue Jul 17, 2019 · 18 comments
Open

Memory and mavlink problems pixhawk v2 #12502

tuloski opened this issue Jul 17, 2019 · 18 comments
Labels
Admin: More Information Needed bug stale

Comments

@tuloski
Copy link

tuloski commented Jul 17, 2019 •
edited
Loading

I noticed that my pixhawk v2 was having some problems, so I hooked up the console in telem 4/5.

What I see at random are different problems:

  • Memory problems: this warning comes out "WARNING WARNING WARNING!
    Revision A has a silicon errata:
    This device can only utilize a maximum of 1MB flash safely!
    https://docs.px4.io/en/flight_controller/silicon_errata.html" but I havn't found any way to fix it.
    And it's not only a warning because later this happens:

"nsh: mount: mount failed: Out of memory
nsh: mkfatfs: mkfatfs failed: Out of memory"

  • Mavlink problems: I have a custom "extras" script to launch mavlink instances. In particular the script is the following:
mavlink stop-all
#ESP
#mavlink start -r 20000 -b 921600 -d /dev/ttyS0
#USB
mavlink start -d /dev/ttyACM0 -b 1000000 -m custom
mavlink stream -d /dev/ttyACM0 -s HEARTBEAT -r 2
mavlink stream -d /dev/ttyACM0 -s ATTITUDE_QUATERNION -r 50
mavlink stream -d /dev/ttyACM0 -s ATTITUDE -r 10
mavlink stream -d /dev/ttyACM0 -s HIGHRES_IMU -r 50
mavlink stream -d /dev/ttyACM0 -s GPS_RAW_INT -r 5
mavlink stream -d /dev/ttyACM0 -s GLOBAL_POSITION_INT -r 50

what happens sometimes at random is this error:
"INFO [mavlink] all instances stopped
INFO [mavlink] mode: Custom, data rate: 50000 B/s on /dev/ttyACM0 @ 1000000B
ERROR [mavlink] could not open /dev/ttyACM0
WARN [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN [mavlink] mavlink for device /dev/ttyACM0 is not running
"

I really don't know where to start fixing it. Can it be all related to memory issues? Why isn't it able to open /dev/ttyACM0 sometimes?

PS: I tried to update the bootloader with https://docs.px4.io/v1.9.0/en/advanced_config/parameter_reference.html#SYS_BL_UPDATE without any success.
@mhkabir
Copy link
Member

mhkabir commented Jul 17, 2019

You're running out of RAM. You need to either stop some modules (unused mavlink instances for example), or get newer hardware with more RAM (e.g Pixhawk 4)

The Silicon Errata warning is about Flash, and is unrelated to the issues you're seeing.

@tuloski
Copy link
Author

tuloski commented Jul 17, 2019

But it's standard px4 1.9.2 code.
There should be a warning that it is unusable on pixhawk v2.

@tuloski
Copy link
Author

tuloski commented Jul 17, 2019 •
edited
Loading

BTW "free" returns:

nsh> free
total used free largest
Umem: 238064 170320 67744 64576

It seems there is some free RAM.

@mhkabir
Copy link
Member

mhkabir commented Jul 17, 2019

Can you post logs from before and after your extras script runs?

@dagar FYI

@tuloski
Copy link
Author

tuloski commented Jul 17, 2019

Case 1 (happening at random, can't open ACM0):

FMUv2 ver 0x8 : Rev 0 V2
[boot] Fault Log info File No 4 Length 3177 flags:0x01 state:1
[boot] Fault Log is Armed
sercon: Registering CDC/ACM serial driver
sercon: Successfully registered the CDC/ACM serial driver
HW arch: PX4_FMU_V2
HW type: V2
HW version: 0x00090008
HW revision: 0x00000000
FW git-hash: 106905871d374c125730fe675266d67c7f869469
FW version: Release 1.9.2 (17367807)
OS: NuttX
OS version: Release 7.28.0 (119275775)
OS git-hash: 423371c7d4012e725ac4ca51323a18df64e581b3
Build datetime: Jul  9 2019 13:47:06
Build uri: localhost
Toolchain: GNU GCC, 7.2.1 20170904 (release) [ARM/embedded-7-branch revision 255                          204]
PX4GUID: 000100000000313235333432470a00280019
MCU: STM32F42x, rev. A

WARNING   WARNING   WARNING!
Revision A has a silicon errata:
This device can only utilize a maximum of 1MB flash safely!
https://docs.px4.io/en/flight_controller/silicon_errata.html

[hardfault_log] Fault Log is Armed
INFO  [tune_control] Publishing standard tune 1
INFO  [param] selected parameter default file /fs/mtd_params
Board defaults: /etc/init.d/rc.board_defaults
rgbled on I2C bus 1 at 0x55 (bus: 100 KHz, max: 100 KHz)
nsh: rgbled_ncp5623c: command not found
nsh: rgbled_pwm: command not found
nsh: blinkm: command not found
Board sensors: /etc/init.d/rc.board_sensors
HMC5883_I2C on I2C bus 1 at 0x1e (bus: 100 KHz, max: 400 KHz)
nsh: lis3mdl: command not found
nsh: ist8310: command not found
WARN  [hmc5883] no device on bus 2 (type: 1)
WARN  [mpu6000] no device on bus #3 (SPI1)
MPU6000 on SPI bus 1 at 4 (1000 KHz)
WARN  [mpu6000] no device on bus #5 (SPI4)
INFO  [mpu9250] Bus probed: 3
WARN  [mpu9250] probe failed! 104
WARN  [mpu9250] no device on bus 3
INFO  [mpu9250] Bus probed: 5
WARN  [mpu9250] probe failed! 0
WARN  [mpu9250] no device on bus 5
L3GD20 on SPI bus 1 at 1 (11000 KHz)
LSM303D on SPI bus 1 at 2 (11000 KHz)
MS5611_SPI on SPI bus 4 at 3 (20000 KHz)
WARN  [ms5611] no device on bus 4
MS5611_SPI on SPI bus 1 at 3 (20000 KHz)
nsh: bst: command not found
ERROR [param] Parameter SENS_EN_BATT not found
ERROR [param] Parameter SENS_EN_MB12XX not found
ERROR [param] Parameter SENS_EN_PGA460 not found
ERROR [param] Parameter SENS_EN_SF1XX not found
ERROR [param] Parameter SENS_EN_TRANGER not found
ERROR [param] Parameter TRIG_MODE not found
ERROR [param] Parameter UAVCAN_ENABLE not found
INFO  [mavlink] mode: Config, data rate: 800000 B/s on /dev/ttyACM0 @ 57600B
Starting Main GPS on /dev/ttyS3
Starting MAVLink on /dev/ttyS1
INFO  [mavlink] mode: Normal, data rate: 1200 B/s on /dev/ttyS1 @ 57600B
INFO  [px4io] default PWM output device
INFO  [init] Mixer: /etc/mixers/quad_x.main.mix on /dev/pwm_output0
INFO  [init] Mixer: /etc/mixers/pass.aux.mix on /dev/pwm_output1
ERROR [param] Parameter CAM_CAP_FBACK not found
ERROR [param] Parameter WEST_EN not found
px4flow [370:100]
INFO  [px4flow] scanning I2C buses for device..
Addons script: /fs/microsd/etc/extras.txt
INFO  [mavlink] waiting for instances to stop
.INFO  [mavlink] exiting channel 0

INFO  [mavlink] all instances stopped
INFO  [mavlink] mode: Custom, data rate: 50000 B/s on /dev/ttyACM0 @ 1000000B
ERROR [mavlink] could not open /dev/ttyACM0
WARN  [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN  [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN  [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN  [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN  [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN  [mavlink] mavlink for device /dev/ttyACM0 is not running
ERROR [param] Parameter UAVCAN_ENABLE not found
INFO  [logger] logger started (mode=all)

NuttShell (NSH)

Case 2 (happening at random, out of memory):

FMUv2 ver 0x8 : Rev 0 V2
[boot] Fault Log info File No 4 Length 3177 flags:0x01 state:1
[boot] Fault Log is Armed
sercon: Registering CDC/ACM serial driver
sercon: Successfully registered the CDC/ACM serial driver
HW arch: PX4_FMU_V2
HW type: V2
HW version: 0x00090008
HW revision: 0x00000000
FW git-hash: 106905871d374c125730fe675266d67c7f869469
FW version: Release 1.9.2 (17367807)
OS: NuttX
OS version: Release 7.28.0 (119275775)
OS git-hash: 423371c7d4012e725ac4ca51323a18df64e581b3
Build datetime: Jul  9 2019 13:47:06
Build uri: localhost
Toolchain: GNU GCC, 7.2.1 20170904 (release) [ARM/embedded-7-branch revision 255204]
PX4GUID: 000100000000313235333432470a00280019
MCU: STM32F42x, rev. A

WARNING   WARNING   WARNING!
Revision A has a silicon errata:
This device can only utilize a maximum of 1MB flash safely!
https://docs.px4.io/en/flight_controller/silicon_errata.html

nsh: mount: mount failed: Out of memory
INFO  [tune_control] Publishing standard tune 16
nsh: mkfatfs: mkfatfs failed: Out of memory
INFO  [tune_control] Publishing standard tune 2
INFO  [param] selected parameter default file /fs/mtd_params
Board defaults: /etc/init.d/rc.board_defaults
WARN  [dataman] Could not open data manager file /fs/microsd/dataman
ERROR [dataman] dataman start failed
rgbled on I2C bus 1 at 0x55 (bus: 100 KHz, max: 100 KHz)
nsh: rgbled_ncp5623c: command not found
nsh: rgbled_pwm: command not found
nsh: blinkm: command not found
Board sensors: /etc/init.d/rc.board_sensors
HMC5883_I2C on I2C bus 1 at 0x1e (bus: 100 KHz, max: 400 KHz)
nsh: lis3mdl: command not found
nsh: ist8310: command not found
WARN  [hmc5883] no device on bus 2 (type: 1)
WARN  [mpu6000] no device on bus #3 (SPI1)
MPU6000 on SPI bus 1 at 4 (1000 KHz)
WARN  [mpu6000] no device on bus #5 (SPI4)
INFO  [mpu9250] Bus probed: 3
WARN  [mpu9250] probe failed! 104
WARN  [mpu9250] no device on bus 3
INFO  [mpu9250] Bus probed: 5
WARN  [mpu9250] probe failed! 0
WARN  [mpu9250] no device on bus 5
L3GD20 on SPI bus 1 at 1 (11000 KHz)
LSM303D on SPI bus 1 at 2 (11000 KHz)
MS5611_SPI on SPI bus 4 at 3 (20000 KHz)
WARN  [ms5611] no device on bus 4
MS5611_SPI on SPI bus 1 at 3 (20000 KHz)
nsh: bst: command not found
ERROR [param] Parameter SENS_EN_BATT not found
ERROR [param] Parameter SENS_EN_MB12XX not found
ERROR [param] Parameter SENS_EN_PGA460 not found
ERROR [param] Parameter SENS_EN_SF1XX not found
ERROR [param] Parameter SENS_EN_TRANGER not found
ERROR [param] Parameter TRIG_MODE not found
ERROR [param] Parameter UAVCAN_ENABLE not found
INFO  [mavlink] mode: Config, data rate: 800000 B/s on /dev/ttyACM0 @ 57600B
Starting Main GPS on /dev/ttyS3
Starting MAVLink on /dev/ttyS1
INFO  [mavlink] mode: Normal, data rate: 1200 B/s on /dev/ttyS1 @ 57600B
ERROR [mavlink] DM_KEY_MISSION_STATE lock failed
ERROR [mavlink] offboard mission init failed (-1)
INFO  [px4io] default PWM output device
INFO  [init] Mixer: /etc/mixers/quad_x.main.mix on /dev/pwm_output0
INFO  [init] Mixer: /etc/mixers/pass.aux.mix on /dev/pwm_output1
ERROR [param] Parameter CAM_CAP_FBACK not found
ERROR [param] Parameter WEST_EN not found
px4flow [350:100]
INFO  [px4flow] scanning I2C buses for device..
ERROR [param] Parameter UAVCAN_ENABLE not found
INFO  [logger] logger started (mode=all)
INFO  [logger] log root dir created: /fs/microsd/log

NuttShell (NSH)

@dagar
Copy link
Member

dagar commented Jul 17, 2019

What's on you SD card? Can you format it and try again?

@tuloski
Copy link
Author

tuloski commented Jul 17, 2019

What's on you SD card? Can you format it and try again?

There are only extras and logs.
Formatted it now with no luck.

@dagar
Copy link
Member

dagar commented Jul 17, 2019

All the errors involve the sdcard. What size is it? How is it formatted? Do you have another you can try?

  • nsh: mount: mount failed: Out of memory
  • mkfatfs: mkfatfs failed: Out of memory
  • WARN [dataman] Could not open data manager file /fs/microsd/dataman
  • ERROR [dataman] dataman start failed

@tuloski
Copy link
Author

tuloski commented Jul 17, 2019

All the errors involve the sdcard. What size is it? How is it formatted? Do you have another you can try?

  • nsh: mount: mount failed: Out of memory
  • mkfatfs: mkfatfs failed: Out of memory
  • WARN [dataman] Could not open data manager file /fs/microsd/dataman
  • ERROR [dataman] dataman start failed

It's a Kingstone 4Gb that ships with pixhawk.
BTW I switched it with a newer one 16 Gb, and it seems solved.
I'll keep you updated if it happens again. Thanks for the input.

@tuloski
Copy link
Author

tuloski commented Jul 17, 2019

mmm I said hurray too early. The problem is still there. It worked 3 reboots and now again at random.
I'm starting to think it's some hardware failure on the pixhawk.

@julianoes
Copy link
Contributor

@tuloski can you see the log output when it fails?

@tuloski
Copy link
Author

tuloski commented Jul 18, 2019

@tuloski can you see the log output when it fails?

In the SD?

@julianoes
Copy link
Contributor

No, that would be with a console connected: https://dev.px4.io/en/debug/system_console.html

@tuloski
Copy link
Author

tuloski commented Jul 18, 2019

Yes, actually what I wrote in Case1 and Case2 are console outputs. Also when it fails I see it.

@julianoes
Copy link
Contributor

Oh, sorry I missed that.

@weekly-digest weekly-digest bot mentioned this issue Jul 21, 2019
Closed
@tuloski
Copy link
Author

tuloski commented Jul 29, 2019

Adding some usleep of 0.5s here and there in the extras.txt seems to solve the issue.
Still not clear what the real issue is.

@julianoes
Copy link
Contributor

Adding some usleep of 0.5s here and there in the extras.txt seems to solve the issue.
Still not clear what the real issue is.

That's odd though because when it fails it doesn't even mount the SD card (from what I see in the log above).

@stale
Copy link

stale bot commented Oct 28, 2019

This issue has been automatically marked as stale because it has not had recent activity. Thank you for your contributions.

@stale stale bot added the stale label Oct 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Admin: More Information Needed bug stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@dagar @julianoes @mhkabir @tuloski