---
title: "Kernel and system traces"
---

# Kernel and system traces

Kernel traces recorded on a board with trace-cmd or tracefs and LTTng sessions with kernel and user-space events, read in one trace editor with per-CPU and per-thread timelines, latency hot spots, system calls and every reported loss.

Both recorders are in the Targets view's board menu. Traces are copied off the board with a SHA-256 check and read on the build host or this computer, in the **Kernel Trace** editor.

## ftrace and trace-cmd

**State:** modelled on QEMU arm64 boards with an Ubuntu 6.8.0 kernel and trace-cmd 3.2, and a 6.8.0 kernel without function tracers (tracefs alone); trace-cmd 3.2.0 and 2.9.6 on this machine and on a build host behind OpenSSH 9.6p1; trace.dat versions 6 and 7; wakeup, interrupt and softirq figures equal to `trace-cmd report --profile`; overruns and drops reported; the running app. Not run on a real board or a BitBake-built image, with real latencies (QEMU timing inflates them), on Windows, on Wind River Linux, or with a non-root board account.

Add `trace-cmd` to the image (and `CONFIG_FUNCTION_GRAPH_TRACER` for function graphs). On the build host install trace-cmd 3.x or set `neuralInverse.embeddedLinux.traceCmd.path`.

1. Run **Record Kernel Trace on Target...**. Pick the tool (the image's `trace-cmd record`, or tracefs alone when the image has no trace-cmd) and what to trace: **Scheduling and Interrupts**, **Choose Events...** from the board's `available_events`, **Function Tracer...** with function filters, or **Function Graph...** with a graph depth; every CPU, a process or a program; a duration and ring buffer size.
2. Events and filters are checked on the board before anything is traced (trace-cmd 3.2 silently traces every function when a filter matches nothing).
3. The confirmation names the board and the exact command. **Stop Kernel Trace Recording** stops early; cancel is in the notification.
4. The file is copied to the build host or this computer (`neuralInverse.embeddedLinux.trace.localFolder`) and read with `trace-cmd report` and `dump` (`neuralInverse.embeddedLinux.traceCmd.localPath` on Linux), or by the module itself for tracefs on any computer.
5. The editor shows a per-CPU **Timeline** of tasks, interrupts and softirqs, **Events** with filters, **Latency Hot Spots** (wakeup to switch, handler and function graph durations) and **Trace Information**: kernel, clock, CPUs, trace.dat version and every reported loss. **Export Timing Record** writes Markdown or CSV.

**Open Kernel Trace...** opens an earlier trace. Agent tools: four read-only; recording asks first.

## LTTng

**State:** modelled on a QEMU arm64 board with kernel 6.8.0, lttng-tools and lttng-ust 2.13 and lttng-modules 2.14, read with babeltrace2: kernel tracepoints and system calls, lttng-ust events, snapshot mode, discarded events, PID tracking, an unknown event refused, stop and cancel, the copied trace's SHA-256, the running app. Not run on a physical board. On Windows, copying a trace to this computer is refused with the reason; reading on the build host works.

Install lttng-tools, lttng-ust and lttng-modules on the board and set its clock (without it lttng-ust writes a negative clock offset and babeltrace2 refuses the trace). On the build host install babeltrace2 2.0 or later, or set `neuralInverse.embeddedLinux.babeltrace2.path` (`babeltrace2.localPath` on this computer).

1. Run **Record LTTng Trace on Target...** and pick **Recording Session** (every event while the session runs, for a number of seconds) or **Snapshot** (overwrite mode, written once at the end).
2. Pick kernel tracepoints, system calls and lttng-ust user-space events, every process or one PID, and buffer sizes. The vpid, vtid and procname contexts are added. Unknown events are refused before anything is traced.
3. The confirmation names the board and the exact commands. **Stop LTTng Recording** stops; cancel leaves no session behind.
4. The CTF folder is copied with a SHA-256 check of every file and read with babeltrace2. The trace editor shows **Resources** (per CPU), **Control Flow** (per thread), **CPU Usage** per process, **System Calls** with latencies, user-space events beside kernel events, and a header stating lost and discarded events. The timing record exports as above.

Agent tools: probing and reading are read-only; recording asks first.

Next: [AMD PetaLinux](https://neuralinverse.com/docs/modules/embedded-linux/amd-petalinux/).