Reading a Zephyr fatal error

A Zephyr fatal error on a Cortex-M chip prints a fault type, a register dump, the fault status registers, sometimes a call trace, and a reason number. Each part answers a different question; this is what they mean and how to get to the line that crashed.

An example

This is a real one, from a Cortex-M3 running Zephyr 4.5:

<err> os: ***** USAGE FAULT *****
<err> os:   Attempt to execute undefined instruction
<err> os: r0/a1:  0x20000240  r1/a2:  0x00000009  r2/a3:  0x20000238
<err> os: r3/a4:  0x20000120 r12/ip:  0x00000010 r14/lr:  0x00004c6f
<err> os:  xpsr:  0x61000000
<err> os: CFSR: 0x00010000
<err> os: HFSR: 0x00000000
<err> os: Faulting instruction address (r15/pc): 0x00004c62
<err> os: call trace:
<err> os:      0: lr: 0x00004c62
<err> os:      1: lr: 0x00004c6f
<err> os:      2: lr: 0x00000fa5
<err> os: >>> ZEPHYR FATAL ERROR 36: Unknown error on CPU 0
<err> os: Current thread: 0x20000120 (main)

1. The fault type and its detail

***** USAGE FAULT ***** and the indented line under it are the short answer. Usage faults are about the instruction (undefined, division by zero, unaligned, a bad function pointer), bus faults about an address with nothing behind it, MPU faults about an access the memory protection forbids (NULL pointers and stack guards), and a hard fault is where the others end up when they cannot be handled.

2. The reason number

FATAL ERROR 36 is a k_fatal_error_reason. 0 to 4 are the kernel's own: 0 a CPU exception, 1 a spurious interrupt, 2 a stack canary, 3 k_oops() or a failed assert, 4 k_panic(). From 16 on they are the ARM fault reasons, in the order of arch/arm/arch.h: 36 is K_ERR_ARM_USAGE_UNDEFINED_INSTRUCTION, 30 division by zero, 32 a stack overflow. "Unknown error" only means the generic printer has no text for it.

3. CFSR and HFSR

The Configurable Fault Status Register holds one bit per cause. 0x00010000 is bit 16, UNDEFINSTR. When MMFAR or BFAR is marked valid, it holds the address that was accessed; an address near 0 is a NULL pointer, and an address just below a thread's stack is a stack overflow.

The fault status bits

Bit	Fault	Usually means
`DACCVIOL`	MemManage	A read or write the MPU forbids: a NULL pointer (MMFAR near 0) or a stack overflow into the guard.
`IACCVIOL`	MemManage	Code run from a region that may not execute: a jump through a bad function pointer.
`MSTKERR`, `STKERR`	MemManage, BusFault	The registers could not be saved on exception entry: a stack overflow.
`PRECISERR`	BusFault	The instruction at PC touched an address with nothing behind it; BFAR says which.
`IMPRECISERR`	BusFault	A buffered write failed; the culprit is a few instructions before PC. Often a peripheral with its clock off.
`UNDEFINSTR`	UsageFault	Bytes that are not an instruction: a corrupt pointer or return address, or `__builtin_trap()`.
`INVSTATE`	UsageFault	A branch to an even address: a call through a NULL or uninitialised function pointer.
`NOCP`	UsageFault	A floating-point instruction with the FPU off.
`UNALIGNED`, `DIVBYZERO`	UsageFault	An unaligned access or an integer division by zero, when trapping is on.
`FORCED` (HFSR)	HardFault	One of the above escalated to HardFault; read the CFSR bits.
`VECTTBL` (HFSR)	HardFault	The vector table could not be read: an image linked for the wrong address, often behind a bootloader.

4. The program counter and the call trace

Faulting instruction address is where it happened; the call trace (printed with CONFIG_EXTRA_EXCEPTION_INFO on Cortex-M, with CONFIG_EXCEPTION_STACK_TRACE on by default) is how it got there. Each entry is a return address: subtract one before looking it up, because it points after the call. With the ELF of that build:

arm-zephyr-eabi-addr2line -fe build/zephyr/zephyr.elf 0x4c62 0x4c6e 0xfa4

Or paste the whole output into the Zephyr fatal error decoder with zephyr.elf: it explains every part and gives each frame as a function and line, in the browser.

For the example above that gives store_reading at main.c:9, called from sample at main.c:24, called from main.

5. Thread names

Current thread: ... (unknown) means the names are not kept; CONFIG_THREAD_NAME=y adds them, and names every thread you create with k_thread_name_set.

Crashes on devices that are not on your desk

The console output above goes to a UART nobody reads once the device is installed. Pinscope's Zephyr module keeps the reason, the thread and the addresses across the reboot and sends them on the next connection, decoded with the ELF you uploaded.

Try it

Three devices are free, and no card is needed. Create an account, or read the guide for your platform: ESP32, Zephyr, Raspberry Pi Pico, Linux, ESP8266.