Build resource monitoring

Build resource monitoring#

The cross container builds are long and QEMU-heavy, and the failure that hurts most is silent resource exhaustion — an OOM-killed cc1plus at hour six, or a full disk mid-export. linux/scripts/01-core/resource-monitor.sh records what the system was doing throughout a build so you can answer, after the fact:

When, and during which build activity, did RAM / disk / CPU headroom run out?

What it captures#

A CSV time-series (resources-<run-id>.csv), one row per tick (default every 15 s), plus a computed resources-<run-id>.summary.txt on exit.

Column	Meaning
`epoch`, `iso`, `elapsed_s`	sample time (absolute + seconds since start)
`load1`, `cpu_pct`	load average and CPU-busy % (from `/proc/stat` deltas)
`mem_used_mb`, `mem_avail_mb`, `mem_pct`	`MemTotal-MemAvailable`, `MemAvailable`, % used
`swap_used_mb`	swap in use — any value >0 means real memory pressure
`disk_avail_gb`, `disk_pct`	free space / % used on the build filesystem
`compilers`	live `cc1plus`/`cc1`/`cc1objplus`/`clang(++)`/`rustc`/`lto1`/`go`/`ld` count
`stage`, `context`	coarse build stage + the exact build log line active then

The context column is the key one: at every peak-pressure moment the summary tells you which file/target was compiling, so you can trace an OOM or disk spike straight to the offending stage (e.g. torch’s aten TUs).

Overhead#

Negligible. Each tick is a handful of /proc reads plus ~3 short-lived forks (df, pgrep, tail|grep). At 15 s that is far below the noise floor of a 32-core build and never competes with the compilers.

Automatic use#

build-cross-chain.sh and build-cross-stage.sh start it automatically for the whole run (start_resource_monitor in lib-orchestrator.sh, gated by RESOURCE_MONITOR, default on) and it self-terminates when the orchestrator exits (via --watch-pid), writing the summary. Output lands in --log-dir.

# default: monitoring on, 15s interval, CSV+summary in the run's log dir
bash linux/scripts/build-cross-chain.sh --from-stage media --log-dir ~/build-logs

# disable it
RESOURCE_MONITOR=0 bash linux/scripts/build-cross-chain.sh ...

Manual use (any other build)#

# start in the background, following whichever log is newest, self-ending with
# the build process:
setsid bash linux/scripts/01-core/resource-monitor.sh \
  --out-dir ~/build-logs --run-id my-run --interval 15 \
  --stage-log-dir ~/build-logs --disk-path / --watch-pid "$BUILD_PID" &

# ...or re-analyze a finished run's CSV at any time:
bash linux/scripts/01-core/resource-monitor.sh summarize ~/build-logs/resources-my-run.csv

Thresholds for the summary’s “near-OOM” / “low-disk” sample counts are tunable with --near-oom-mb (default 4096) and --low-disk-gb (default 20).

Reading the results#

swap_used_mb climbing or mem_avail_mb near zero → you are at the OOM edge. Cross-reference the context at that row and lower the relevant *_MB_PER_JOB per build-parallelism-memory-tuning.md, or add RAM.
disk_avail_gb dropping toward the low-disk threshold → reclaim disk before the next run in the order of Linux Host Setup § B7: prune-safe.sh first, the regenerable cache-export dir ~/.cache/kata-buildcache after it.
compilers well below core count while mem_avail is low → the stage is RAM-bound (expected for torch); more RAM, not more -j, is the lever.
The CSV plots directly in any spreadsheet / gnuplot if you want a timeline.

The CSV is rewritten on each start, so give each build a distinct --run-id (the orchestrator passes CROSS_RUN_ID) to preserve history.

Mining a build log for the actual failure#

The monitoring above tells you what a build consumed. When one fails, the complementary problem is finding the real error in tens of thousands of lines, where the last line printed is usually a downstream symptom rather than the cause.

Capture the log in the first place — build-cross-chain.sh and build-cross-stage.sh take --log-dir; every other build, including a hand-run one, should be teed:

nerdctl build -t <tag> -f linux/Dockerfile.<stage> . 2>&1 | tee out/build-logs/build.log

Then pull out the lines that actually mark failures:

grep -nE '(^| )FAILED: |ninja: build stopped|error:|fatal error:|undefined reference|collect2: error|ld\.lld: error|permission denied|no space left on device|exit code [1-9]' out/build-logs/build.log

Reading the hits, in order of what they mean:

Match	What it usually means
`FAILED:`	The ninja edge that broke — start here, it names the target
`fatal error:` / `error:`	The compiler diagnostic; the first one is the cause, later ones are fallout
`undefined reference` / `collect2: error` / `ld.lld: error`	Link stage — a missing library or an ABI mismatch, not a source bug
`no space left on device`	Host disk, not the build. See Linux Host Setup § B3
`permission denied`	Usually a bind-mount uid mismatch rather than a real permission problem
`ninja: build stopped`	Terminator line only — the cause is above it

-n matters: it gives you the line number to seek to, which is the point of the exercise on a log too large to read.