Random crashes when using WiFi (IDFGH-18)

[kluverp]

Environment

• Development Kit: [none]
• Core (if using chip or module): [ESP-WROOM32]
• IDF version: 26f44d781bc5e9638a7222b1b3abba3905751b76 (v3.1)
• Development Env: [Eclipse]
• Operating System: [Ubuntu]
• Power Supply: [external 5V]

Problem Description

During runtime the ESP seems to crash unexpectedly. I believe this has something to do with the WiFi library. Although I am not sure.
The situation: ESP32 is connected with WiFi in station (STA) mode, and sends sensor data every 10 sec. This runs fine for hours, until it crashes.
Sometimes it runs 8 hours, sometime it only runs 15 min.

Where to look for this problem?

All I got is the following core dump:

==================== ESP32 CORE DUMP START ====================

================== CURRENT THREAD REGISTERS ===================
pc 0x4013f9e0 0x4013f9e0 <ram_tx_pwr_backoff+412>
lbeg 0x4000c46c 1073792108
lend 0x4000c477 1073792119
lcount 0x0 0
sar 0x1b 27
ps 0x60320 394016
threadptr
br
scompare1
acclo
acchi
m0
m1
m2
m3
expstate
f64r_lo
f64r_hi
f64s
fcr
fsr
a0 0x4013d38f 1075041167
a1 0x3ffc51b0 1073500592
a2 0x1 1
a3 0x200000 2097152
a4 0x3ffba578 1073456504
a5 0x200000 2097152
a6 0x43422c7c 1128410236
a7 0x3ffbc874 1073465460
a8 0x1fffffff 536870911
a9 0x3ffc5190 1073500560
a10 0x6001ccb8 1610730680
a11 0x1604270 23085680
a12 0x60320 394016
a13 0x3ff000e4 1072693476
a14 0x3ffb3b28 1073429288
a15 0x60 96

==================== CURRENT THREAD STACK =====================
#0 0x4013f9e0 in ram_tx_pwr_backoff (target_pwr=0x0, linear_flag=0x0) at phy_chip_v7_cal.c:1432
#1 0x4013d38f in rf_cal_data_backup (rf_cal_data=0x0) at phy_chip_v7.c:2903
#2 0x4013d6e3 in bb_init () at phy_chip_v7.c:2396
#3 0x4010df68 in system_event_ap_start_handle_default (event=0x0) at /home/peter/esp/esp-idf-v3.1/components/esp32/event_default_handlers.c:161
#4 0x4010e166 in system_event_eth_connected_handle_default (event=0x1) at /home/peter/esp/esp-idf-v3.1/components/esp32/event_default_handlers.c:101
#5 0x40118adc in wifi_hw_stop ()
#6 0x4012c4bc in lmacProcessTxSuccess ()
#7 0x4012c8b4 in lmac_stop_hw_txq ()
#8 0x4012f650 in ppMapWaitTxq ()
#9 0x4012f6d8 in ppProcessWaitingQueue ()
#10 0x40088165 in ppTask ()

======================== THREADS INFO =========================
Id Target Id Frame
11 process 10 0x4008c188 in xQueueGiveFromISR (xQueue=0x3ffaff04, pxHigherPriorityTaskWoken=0x0) at /home/peter/esp/esp-idf-v3.1/components/freertos/queue.c:1348
10 process 9 0x4008c188 in xQueueGiveFromISR (xQueue=0x3ffaea78, pxHigherPriorityTaskWoken=0x0) at /home/peter/esp/esp-idf-v3.1/components/freertos/queue.c:1348
9 process 8 0x4008c188 in xQueueGiveFromISR (xQueue=0x3ffbddc4 <emac_dma_tx_buf+14300>, pxHigherPriorityTaskWoken=0x3ffbef30 <emac_dma_rx_buf+2760>) at /home/peter/esp/esp-idf-v3.1/components/freertos/queue.c:1348
8 process 7 0x4008c188 in xQueueGiveFromISR (xQueue=0x3ffc2c48 <dns_pcbs+8>, pxHigherPriorityTaskWoken=0x3ffc3ca0 <g_ic+412>) at /home/peter/esp/esp-idf-v3.1/components/freertos/queue.c:1348
7 process 6 0x4008c188 in xQueueGiveFromISR (xQueue=0x3ffaff58, pxHigherPriorityTaskWoken=0x0) at /home/peter/esp/esp-idf-v3.1/components/freertos/queue.c:1348
6 process 5 0x40081dfa in esp_crosscore_int_send (core_id=0, reason_mask=1073418360) at /home/peter/esp/esp-idf-v3.1/components/esp32/crosscore_int.c:100
5 process 4 0x4008c188 in xQueueGiveFromISR (xQueue=0x3ffc1d0c <emac_dma_rx_buf+14500>, pxHigherPriorityTaskWoken=0x3ffc2950 <dns_table+420>) at /home/peter/esp/esp-idf-v3.1/components/freertos/queue.c:1348
4 process 3 0x4008cee0 in vTaskDelete (xTaskToDelete=) at /home/peter/esp/esp-idf-v3.1/components/freertos/tasks.c:1229
3 process 2 0x4017537e in SHA1Transform (state=0x0, buffer=) at /home/peter/esp/esp-idf-v3.1/components/wpa_supplicant/src/crypto/sha1-internal.c:208
2 process 1 0x4017537e in SHA1Transform (state=0x0, buffer=) at /home/peter/esp/esp-idf-v3.1/components/wpa_supplicant/src/crypto/sha1-internal.c:208

• 1 0x4013f9e0 in ram_tx_pwr_backoff (target_pwr=0x0, linear_flag=0x0) at phy_chip_v7_cal.c:1432

======================= ALL MEMORY REGIONS ========================
Name Address Size Attrs
.rtc.text 0x400c0000 0x0 RW
.rtc_noinit 0x50000000 0x0 RW
.iram0.vectors 0x40080000 0x400 R XA
.iram0.text 0x40080400 0x118c4 RWXA
.dram0.data 0x3ffb0000 0x3a88 RW A
.noinit 0x3ffb3a88 0x0 RW
.flash.rodata 0x3f400020 0x2ef88 RW A
.flash.text 0x400d0018 0xaaa74 R XA
.coredump.tasks 0x3ffc5470 0x164 RW
.coredump.tasks 0x3ffc50f0 0x378 RW
.coredump.tasks 0x3ffbf6ac 0x164 RW
.coredump.tasks 0x3ffbf530 0x174 RW
.coredump.tasks 0x3ffbfc18 0x164 RW
.coredump.tasks 0x3ffbfaa0 0x170 RW
.coredump.tasks 0x3ffce570 0x164 RW
.coredump.tasks 0x3ffcbcb0 0x28b8 RW
.coredump.tasks 0x3ffc29f0 0x164 RW
.coredump.tasks 0x3ffc27f0 0x1f8 RW
.coredump.tasks 0x3ffc0678 0x164 RW
.coredump.tasks 0x3ffc0500 0x170 RW
.coredump.tasks 0x3ffbdae4 0x164 RW
.coredump.tasks 0x3ffbd950 0x18c RW
.coredump.tasks 0x3ffc3d58 0x164 RW
.coredump.tasks 0x3ffc3ba0 0x1b0 RW
.coredump.tasks 0x3ffbf0bc 0x164 RW
.coredump.tasks 0x3ffbee20 0x294 RW
.coredump.tasks 0x3ffafa88 0x164 RW
.coredump.tasks 0x3ffaf900 0x180 RW
.coredump.tasks 0x3ffbd578 0x164 RW
.coredump.tasks 0x3ffbd3f0 0x180 RW

===================== ESP32 CORE DUMP END =====================

[zhangyanjiaoesp]

@kluverp
I have tested on the latest release/v3.1, but I can't reproduce your problem. I have several questions to ask :

1. Can you confirm the IDF version? I can't find the branch 26f44d781bc5e9638a7222b1b3abba3905751b76 in release/v3.1.
2. Can you show more details on how to reproduce your problem? For example, you said sends sensor data every 10 sec, what's the size of the data? where did these data be sent to ?
3. If it's possible, can you provide capture packets?
4. I see there is a log system_event_eth_connected_handle_default (event=0x1) in your core dump, do you use ethernet also?

[kluverp]

@kluverp
I have tested on the latest release/v3.1, but I can't reproduce your problem. I have several questions to ask :

1. Can you confirm the IDF version? I can't find the branch 26f44d781bc5e9638a7222b1b3abba3905751b76 in release/v3.1.
2. Can you show more details on how to reproduce your problem? For example, you said sends sensor data every 10 sec, what's the size of the data? where did these data be sent to ?
3. If it's possible, can you provide capture packets?
4. I see there is a log system_event_eth_connected_handle_default (event=0x1) in your core dump, do you use ethernet also?

Thanks for looking into this!

1. I made a mistake here. The revision is 22da5f6de9ff3657fa7613fc06ad2eca1d1fe14a. Seems I took the revision of my user code, instead of IDF framework.
2. The data is sensor data in JSON format that gets send every 10 seconds. The body is around 4400 chars or 4.3 kb. I use a RTOS task to make a HTTP post request to our server every 10 sec.
3. Unfortunately not, since the device is on a remote location.
4. Yes, we also use Ethernet, but only one of Ethernet or WiFi is initialized. I did not notice this event earlier. I have a block like so:

`
switch(W_conn_type.val)
{
case 0:
// no internet
break;
case 1:
// enable Ethernet if connection parameter is set to "ethernet"
ethernet_init();
break;
case 2:
// enable WiFi if connection parameter is set to "WiFi"
wifi_init();
break;
}

`

So either one of the two should be initialized. I also see a system_event_ap_start_handle_default. But we only use the wifi in station mode, so I expect only to see a system_event_sta_start_handle_default.

I cannot for sure say, what happened here.

[zhangyanjiaoesp]

@kluverp Please update your IDF version to have a test.

[zhangyanjiaoesp]

You can update to the latest v3.1

[Alvin1Zhang]

@kluverp Thanks for reporting, we have fixed on IDF v3.1. Would you please help try latest IDF v3.1 and help share whether the issue gets resolved? Thanks.

[kluverp]

@kluverp Thanks for reporting, we have fixed on IDF v3.1. Would you please help try latest IDF v3.1 and help share whether the issue gets resolved? Thanks.

Thanks for the effort. My project however has moved on to the IDF v3.2 so I cannot reproduce this anymore. I'd say this issue can be closed.

[Alvin1Zhang]

@kluverp Thanks for sharing the updates.