Zephyr devices in the field: logs, crashes and OTA over the internet

Zephyr gives you a network stack, a logging subsystem and MCUboot. What it does not give you is a way to see the logs of a device on someone else's network, to know it crashed, or to send it new firmware. This is how to add that.

What the pieces are

Logs: Zephyr's logging subsystem has backends for UART, RTT and a few network protocols, but a backend that sends logs over UDP or MQTT still needs a server, and a device behind NAT cannot be reached for more.
Crashes: a fatal error prints the fault and a call trace to the console, then halts or reboots. In the field nobody is reading the console.
Updates: MCUboot swaps and verifies images, and mcumgr can send them, but over serial, BLE or a UDP port someone has to reach.

All three need the device to keep a connection out to something you can reach. That is what Pinscope is: a Zephyr module that keeps one outbound connection, and a dashboard on the other end.

Adding it

Add the module to west.yml and run west update:

- name: pinscope
  url: https://github.com/pinscope-dev/pinscope-zephyr
  path: modules/lib/pinscope

Then, in prj.conf:

CONFIG_PINSCOPE=y
CONFIG_PINSCOPE_PROJECT_KEY="psk_..."

No code. The module switches on what it needs, as defaults your configuration can still change: IPv4, TCP, sockets through the POSIX API, DNS, DHCP, enough network contexts, settings in NVS, and the CSPRNG. It starts by itself at boot and starts DHCP unless the interface already has an address. When something cannot be switched on for you, the build stops and the message says what to add: a storage partition in the devicetree, a random number source, more connections.

Logs

Everything logged with LOG_INF, LOG_WRN, LOG_ERR and the rest goes to the dashboard as it happens, with its level and module, and is stored. With CONFIG_LOG_PRINTK, so does printk. Nothing else in your logging changes: the UART keeps working.

Crashes

When the firmware hits a fatal error, the reason (K_ERR_*), the thread, the program counter and the code addresses on the stack are kept in RAM across the reboot and sent on the next connection. Builds are named by their GNU build ID, which the module keeps in flash; upload zephyr.elf and every crash of that build shows functions and lines. For a crash on your desk, the Zephyr fatal error decoder reads the console output.

Updates with MCUboot

Build with sysbuild and MCUboot (SB_CONFIG_BOOTLOADER_MCUBOOT=y) and the dashboard shows an Update firmware button. Upload zephyr.signed.bin; the server checks it is an MCUboot image, the device writes it to the second slot and requests a test swap. The new image confirms itself once it reaches Pinscope again; if it does not, MCUboot swaps back at the next reset.

Check the signing key before you ship: by default sysbuild signs with MCUboot's development key, whose private half is public. The MCUboot image inspector tells you, and how to change it.

The device's own web server

If the firmware serves HTTP (Zephyr's HTTP server, or your own on sockets), it gets an HTTPS address in the dashboard. The request is relayed over the device's connection to its own address, so the server does not have to know about Pinscope.

Memory

About 8.6 KB of RAM, all of it static: 4.2 KB of buffers, queue and state, the same as on ESP32, half a kilobyte for the Zephyr port and its log backend, and the module's thread with a 4 KB stack (CONFIG_PINSCOPE_STACK_SIZE), of which it uses about 1.3 KB on Cortex-M; the dashboard's stack metric shows it on your build. Nothing is allocated at run time. No TLS library: the Noise protocol with X25519 and ChaCha20-Poly1305 needs neither mbedTLS nor certificates.

Boards

Anything Zephyr gives a network to. The sample is set up and built on every change for thirteen: Nordic's nRF7002 DK (WiFi), nRF5340 DK (Ethernet shield) and nRF9151 DK (LTE-M and NB-IoT, in the nRF Connect SDK); ST's NUCLEO-F429ZI and NUCLEO-H563ZI (Ethernet) and B-L475E-IOT01A (WiFi); Espressif's ESP32-S3, ESP32-C3 and ESP32-C6 (WiFi); NXP's FRDM-RW612 (Ethernet); the Raspberry Pi Pico 2 W (WiFi); the WIZnet W55RP20 (Ethernet); and QEMU, to try it with no board at all. Run on hardware, with updates, rollback and crash reports: the W55RP20 over Ethernet and the ESP32-C6 over WiFi. The Zephyr docs have the command for each.

Try it

Three devices are free, and no card is needed. Create an account, or read the guide for your platform: ESP32, Zephyr, Raspberry Pi Pico, Linux, ESP8266.