ESP32 crash backtraces: decoding Guru Meditation errors from the field

On your desk a crash prints a backtrace to the serial monitor and you fix it. In the field the same crash prints to a port nothing is listening on, and the device restarts as if nothing happened.

What the device knows when it crashes

When the firmware panics, ESP-IDF's panic handler prints the reason, the registers and a backtrace:

Guru Meditation Error: Core  0 panic'ed (StoreProhibited). Exception was unhandled.

Backtrace: 0x4008de70:0x3ffc1e40 0x400affcf:0x3ffc1e60 0x400b117d:0x3ffc1e80

Those addresses are positions in your code. With the ELF file of that exact build they turn into function names and lines:

xtensa-esp32s2-elf-addr2line -pfiaC -e build/app.elf 0x4008de70 0x400affcf 0x400b117d

On the RISC-V chips (C3, C6) the tool is riscv32-esp-elf-addr2line, and the panic output is a register dump and a stack dump instead of a ready backtrace.

The two things that go missing

The output. Nobody is reading the serial port. After the restart the only trace is the reset reason, ESP_RST_PANIC, which says that it crashed and nothing about where.

The ELF. Addresses mean something only against the build they came from. Six months later, with three firmware versions in the field, you need to know which build a device was running and still have that build's ELF.

Keeping the crash across the reboot

ESP-IDF can save a core dump to a flash partition (CONFIG_ESP_COREDUMP_ENABLE_TO_FLASH). It holds every task's stack and is the most complete record there is. It needs a partition of its own, typically 64 KB, and then a way to get that dump off the device, and idf.py coredump-info with the right ELF to read it.

A lighter way is to keep only what a person reads first: the reason, the task, the program counter and the backtrace. That is a few hundred bytes, and it fits in RTC memory, which survives a restart without touching flash. On the next start the firmware finds it and sends it.

That is what Pinscope does. It wraps the panic handler, keeps the summary in RTC memory, and reports it on the next connection, together with the hash of the build that crashed. If you have uploaded that build's ELF, the dashboard shows

StoreProhibited in task httpd
0x4008de70: valve_set at main/valve.c:57
0x400affcf: on_post at main/api.c:112

and if you have not, it shows the addresses and the addr2line command to run. A crash is decoded only with the ELF it came from, matched by hash, so a line number is never from the wrong build. An alert can go to your email or a webhook when it happens.

Habits that make crashes readable

Keep the ELF of every build that leaves the building. It is the only thing that turns addresses into lines.
Put a version in the firmware (PROJECT_VER) and report it, so you know what each device runs.
Watch the free stack of each task before it becomes a crash. Most field crashes on an ESP32 are a stack that was nearly full on the desk and overflowed in the field.
Report restarts by reason. A device that restarts from the watchdog every night has a bug, even if it never panics.

Try it

Three devices are free with every feature, and no card is needed. Create an account, or read the five-minute guide first.