# Build resource monitoring

The cross container builds are long and QEMU-heavy, and the failure that hurts
most is silent resource exhaustion — an OOM-killed `cc1plus` at hour six, or a
full disk mid-export. `linux/scripts/01-core/resource-monitor.sh` records what
the system was doing throughout a build so you can answer, after the fact:

> **When, and during which build activity, did RAM / disk / CPU headroom run out?**

## What it captures

A CSV time-series (`resources-<run-id>.csv`), one row per tick (default every
15 s), plus a computed `resources-<run-id>.summary.txt` on exit.

| Column | Meaning |
|---|---|
| `epoch`, `iso`, `elapsed_s` | sample time (absolute + seconds since start) |
| `load1`, `cpu_pct` | load average and CPU-busy % (from `/proc/stat` deltas) |
| `mem_used_mb`, `mem_avail_mb`, `mem_pct` | `MemTotal-MemAvailable`, `MemAvailable`, % used |
| `swap_used_mb` | swap in use — **any** value >0 means real memory pressure |
| `disk_avail_gb`, `disk_pct` | free space / % used on the build filesystem |
| `compilers` | live `cc1plus`/`cc1`/`clang(++)`/`rustc`/`lto1`/`ld` count |
| `stage`, `context` | coarse build stage + the exact build log line active then |

The **`context`** column is the key one: at every peak-pressure moment the
summary tells you *which file/target was compiling*, so you can trace an OOM or
disk spike straight to the offending stage (e.g. torch's `aten` TUs).

## Overhead

Negligible. Each tick is a handful of `/proc` reads plus ~3 short-lived forks
(`df`, `pgrep`, `tail|grep`). At 15 s that is far below the noise floor of a
32-core build and never competes with the compilers.

## Automatic use

`build-cross-chain.sh` starts it automatically for the whole run (gated by
`RESOURCE_MONITOR`, default on) and it self-terminates when the orchestrator
exits (via `--watch-pid`), writing the summary. Output lands in `--log-dir`.

```bash
# default: monitoring on, 15s interval, CSV+summary in the run's log dir
bash linux/scripts/build-cross-chain.sh --from-stage media --log-dir ~/build-logs

# disable it
RESOURCE_MONITOR=0 bash linux/scripts/build-cross-chain.sh ...
```

## Manual use (any build, or a surgical `build-cross-stage.sh`)

```bash
# start in the background, following whichever log is newest, self-ending with
# the build process:
setsid bash linux/scripts/01-core/resource-monitor.sh \
  --out-dir ~/build-logs --run-id my-run --interval 15 \
  --stage-log-dir ~/build-logs --disk-path / --watch-pid "$BUILD_PID" &

# ...or re-analyze a finished run's CSV at any time:
bash linux/scripts/01-core/resource-monitor.sh summarize ~/build-logs/resources-my-run.csv
```

Thresholds for the summary's "near-OOM" / "low-disk" sample counts are tunable
with `--near-oom-mb` (default 4096) and `--low-disk-gb` (default 20).

## Reading the results

- **`swap_used_mb` climbing or `mem_avail_mb` near zero** → you are at the OOM
  edge. Cross-reference the `context` at that row and lower the relevant
  `*_MB_PER_JOB` per [build-parallelism-memory-tuning.md](build-parallelism-memory-tuning.md),
  or add RAM.
- **`disk_avail_gb` dropping toward the low-disk threshold** → prune the local
  buildkit cache (`~/.cache/kata-buildcache`) before the next run.
- **`compilers` well below core count while `mem_avail` is low** → the stage is
  RAM-bound (expected for torch); more RAM, not more `-j`, is the lever.
- The CSV plots directly in any spreadsheet / `gnuplot` if you want a timeline.

> The CSV is **rewritten** on each start, so give each build a distinct
> `--run-id` (the orchestrator passes `CROSS_RUN_ID`) to preserve history.