Debugging with ELF Core Dumps on i.MX RT Targets

1. Overview

This application note describes how to collect and analyse ELF core dumps from user-space applications running on the Emcraft uClinux BSP for NXP i.MX RT targets.

When a user-space process on the target terminates from an uncaught fatal signal (SIGSEGV, SIGABRT, SIGBUS, SIGFPE, ...), the kernel can be configured to write an ELF core file capturing the process's register state, memory layout, and loaded shared-library map. The core file is later opened on the development host with the BSP's cross gdb, which reconstructs the full backtrace and variable state as they were at the moment of the crash.

This workflow is the primary tool for diagnosing crashes that cannot be reproduced under a live gdbserver session -- field failures, timing-sensitive bugs, and crashes inside third-party libraries.

The i.MX RT targets are ARM Cortex-M7 microcontrollers without an MMU. User-space binaries run in the FDPIC (Function Descriptor Position-Independent Code) format, and the kernel uses fs/binfmt_elf_fdpic.c rather than the mainline ELF loader. All core-dump mechanics described below are the FDPIC variants of the familiar Linux tooling.

2. Target-Side Prerequisites

2.1. Kernel Configuration

Three Linux kernel options control core-dump support:

Inspect the kernel configuration of your project for these options and enable any that are not set:

In the resulting menu, navigate to:

Save the configuration, then rebuild the kernel:

2.2. Run-Time Settings

Two kernel knobs control when and where a core file is produced. Both have to be set per boot unless you persist them in an init script under etc/:

1. Allow cores of any size (defaults to 0, i.e. no core).

2. Choose where the core goes. A plain filename places it in the crashing process's current working directory. An absolute path centralises cores in one place. (The %e (executable name) and %p (PID) substitutions in core_pattern are documented in core(5); use them to keep multiple cores from overwriting one another.)

2.3. Memory Budget

No-MMU Linux does not use demand paging; a process's virtual address space is backed one-to-one by physical RAM. The core file therefore contains the actual memory image of the process and is the size of its resident footprint. On targets with limited RAM (the 1050/1060 ship with 32 MB or less), make sure core_pattern points at a filesystem with enough free space -- an SD card mount or a network mount -- rather than tmpfs, which would consume SDRAM a second time.

2.4. Keep an Unstripped Copy on the Host

Stripping a binary does not affect core-dump generation on the target: a stripped executable produces exactly the same core as its unstripped counterpart, and the GNU build-id (.note.gnu.build-id) that GDB uses to match the two is preserved by strip. The target-side copy can therefore remain stripped -- which is the normal configuration for size-constrained i.MX RT rootfs images.

What matters is the host-side copy that you load into gdb for analysis. GDB's post-mortem backtrace on FDPIC relies on:

• ARM EHABI unwind tables (.ARM.exidx / .ARM.extab), which survive strip -- they are allocatable sections required at runtime by the C++ exception machinery;

• function prologue scanning for frames that EHABI does not describe -- works on stripped code;

• DWARF debug info for file names, line numbers, local variables and function arguments -- removed by strip.

In other words, a stripped binary still back-traces correctly on the host but yields hex PCs only. Source listings (list), local variables (info locals), and function arguments require the unstripped copy. Keep the unstripped build output of each user-space application on the host alongside its core files.

For BSP-provided packages this is already taken care of. The pre-built toolchain under tools/arm-buildroot-uclinuxfdpiceabi_sdk-buildroot/ ships a full sysroot containing unstripped copies of the C library, the dynamic loader, and every user-space binary of every enabled package. ACTIVATE.sh exports the sysroot path as $TOOLS_LIBS; pointing gdb set sysroot at it (§4.2) gives post-mortem sessions full debug info for all BSP-provided code without any extra setup.

If you rebuild the toolchain from the BSP's Buildroot tree with make buildroot-sdk, the freshly built sysroot appears under buildroot/output_<defconfig>/host/arm-buildroot-uclinuxfdpiceabi/sysroot/
(for example buildroot/output_arm_cortexm_sdk_lite/host/arm-buildroot-uclinuxfdpiceabi/sysroot/). Point set sysroot there instead of $TOOLS_LIBS.

For your own applications, build with -g and keep the unstripped build output on the host. Only strip the copy that you install into the target image.

3. Triggering and Retrieving a Core

3.1. A Minimal Reproducer

Any process that terminates from an uncaught fatal signal produces a core, so the reproducer can be very small. Save the following as crash_fdpic.c on the host:

The two __attribute__((noinline)) markers keep level1 and level2 visible as separate frames even at higher optimisation levels, which makes the backtrace in §4.3 easier to read.

Build it with the BSP cross-compiler after sourcing the activation script:

-g embeds DWARF debug information (used by GDB on the host for source listings and local variables); -O0 disables optimisations so that stack frames and variables line up with the source code.
The toolchain defaults to the FDPIC binary format required by the no-MMU kernel -- no extra flag is needed.

Keep the resulting crash_fdpic binary on the host unchanged; it is the unstripped host-side copy GDB will later match to the core by build-id (§2.4).

Copy the binary to the target via scp:

3.2. Capturing a Core

On the target:

The default BSP shell is /bin/hush, which prints just Segmentation fault without the (core dumped) suffix that bash and busybox ash append. The shell message is therefore not a reliable indicator of whether a core was written. Confirm success by the ls -l /var/cores/ output above -- if no file appears there, re-check the prerequisites from §2, most commonly a missing ulimit -c unlimited or a read-only / full filesystem pointed at by core_pattern.

3.3. Copying the Core to the Host

The rootfs project ships dropbear (SSH server), so scp from the host work out of the box:

Only the core file has to travel from the target to the host. The executable passed to gdb is the unstripped host-side build output from §3.1 -- the crash_fdpic produced by the cross-compiler, still sitting in your working directory. GDB matches it to the core by build-id, which strip preserves on the target copy, so the host-side build output and the stripped target copy are a valid pair.

If you no longer have the matching build output on the host (for example because the tree has since been rebuilt from modified sources, which changes the build-id), GDB will warn with warning: exec file is newer than core file and symbol resolution will be unreliable. In that case, rebuild from the exact revision that was on the target, or pull the binary back from the target with a second scp -- but remember that the target copy may be stripped, so source listings and local variables will not be available.

4. Host-Side Analysis with Cross-GDB

4.1. Selecting the Right GDB

Use the BSP cross debugger, accessible after sourcing the BSP activation script:

If you have rebuilt the toolchain from the BSP's Buildroot tree (see Using Buildroot to Customize the Toolchain and Applications), the equivalent binary lives at buildroot/output_<defconfig>/host/bin/arm-buildroot-uclinuxfdpiceabi-gdb. Both are functionally identical.

This BSP-specific build of gdb contains four FDPIC core-dump patches (tracked as RM-7623) without which post-mortem sessions either fail to open the core or produce an incorrect backtrace. A stock arm-linux-gnueabi-gdb downloaded from a distribution package repository or the ARM Developer site will not work for these cores.

4.2. Opening a Core

From the BSP top-level directory, with ACTIVATE.sh sourced:

Breaking that down:

4.3. A Typical Session

bt full, info locals, info args, list, and disassemble all work on frames that carry DWARF information. Frames inside stripped libraries still resolve PC to symbol name but do not show source.

5. Known Cosmetic Messages

The following diagnostics are expected on a healthy FDPIC core-dump session and can be ignored:

• warning: shared library handler failed to enable breakpoint - GDB tries to plant a breakpoint in the dynamic loader to observe future dlopen() calls. In a post-mortem session there are no future events to observe, so the failure is harmless.

• cpsr 0x01000000 ... - The core target synthesises a generic ARM register set that names the status register cpsr. On a Cortex-M target this field actually holds the xPSR; bit 24 is the T (Thumb) bit rather than bit 5. The BSP cross-GDB is aware of the difference and interprets the value correctly for unwinding; only the label remains cpsr.

• Empty .eh_frame - FDPIC executables carry ARM EHABI tables (.ARM.exidx / .ARM.extab) rather than DWARF .eh_frame. The cross-GDB selects EHABI automatically; no extra option is needed.

• fpscr <unavailable> in info registers - The FDPIC core dumper in the current kernel writes only the NT_PRSTATUS and (legacy FPA) NT_PRFPREG register notes. The Cortex-M VFP state, which would be carried in an NT_ARM_VFP note, is not emitted; GDB's target description still includes fpscr and therefore reports it as unavailable. Integer registers, PC, LR, SP, and stack memory are fully captured, so backtrace and variable inspection are unaffected. If VFP state is specifically required, attach live with gdbserver (which uses ptrace and returns the full VFP register set).

6. Troubleshooting

7. Further Reading

• core(5) - kernel manual page on core file format and /proc/sys/kernel/core_pattern.

• GDB User Manual, chapter "Core Files" -- generic post-mortem workflow.

• Using Buildroot to Customize the Toolchain and Applications -- BSP-level overview of the Buildroot integration and how to rebuild the cross gdb used in this note.

• For live-attach debugging, gdbserver is installed in the rootfs at /usr/bin/gdbserver and complements, rather than replaces, post-mortem core dumps.