Build parallelism & per-job memory calibration

Build parallelism & per-job memory calibration#

Audience: anyone (human or AI agent) tempted to “speed up the build” by raising job counts or lowering the per-job memory estimates in linux/scripts/01-core/parallelism.sh.

Read this first. The per-job memory numbers look conservative but are calibrated to the host’s RAM. Lowering them the naive way OOM-kills a multi-hour build. This doc explains the model, the one trap that makes it counter-intuitive, and how to tune it safely with data.

TL;DR — the one rule#

Every memory-capped stage obeys one formula:

jobs = min( cores ,  usable_RAM / BUILD_MEM_DIVISOR / PEAK_per-TU_RAM )

The config knob is mb_per_job, which is PEAK_per-TU_RAM.

  • Lowering mb_per_job raises jobs → faster only if RAM allows it.

  • If jobs × PEAK exceeds RAM, cc1plus gets OOM-killed and the build dies.

  • BUILD_MEM_DIVISOR (default 1) is the only concurrency knob: set it to the number of builds sharing the host RAM at once (see Max speed below).

Tune against the PEAK per-TU footprint, never the average. See the trap below.

The helpers (linux/scripts/01-core/parallelism.sh)#

All logic lives in one core function; the named helpers just pick a profile:

mem_capped_jobs <peak_mb> [requested]        # the core: jobs = min(cores, usable/peak)

Named wrapper

Profile / used for

peak (normal)

peak (aggressive)

compute_jobs

pure CPU-bound, no mem cap

— (just cores)

—

compute_jobs_with_mem_cap

generic — C/C++ (OpenCV, ONNX, …)

DEFAULT_MB_PER_JOB:-2000

DEFAULT_MB_PER_JOB:-800

compute_rust_jobs

rust — Cargo (gst-plugins-rs)

RUST_MB_PER_JOB:-2500

RUST_MB_PER_JOB:-1200

compute_cpp_heavy_jobs

heavy — PyTorch / torch_cpu / LTO

CPP_HEAVY_MB_PER_JOB:-4096

4096 (floor kept)

Peak values come from _profile_mb <profile>; the wrappers are one-liners over mem_capped_jobs. Where:

  • cores = detect_available_cores() (respects cgroup CPU quota).

  • usable_RAM = _usable_mem_mb() = _mem_available_mb() (MemAvailable, or the cgroup memory limit remaining — smaller wins) divided by BUILD_MEM_DIVISOR.

Each workload gets a different estimate because its peak translation unit is different — torch’s aten/autograd TUs peak ~4 GB per cc1plus, while a typical OpenCV TU is a few hundred MB.

AGGRESSIVE_PARALLELISM auto-enables at ≥ 16 GB RAM#

_auto_aggressive_parallelism() sets AGGRESSIVE_PARALLELISM=true automatically when usable_RAM ≥ 16 GB (unless you export it false). In aggressive mode the generic and rust estimates drop (2000→800, 2500→1200), so on any real-size host those stages run at full core count — they are not RAM-capped and have no headroom to reclaim.

compute_cpp_heavy_jobs deliberately ignores aggressive mode and keeps the 4 GB floor: torch TUs do not get cheaper on a big host, so the only thing more RAM buys is more of them at once — which the usable_RAM / mb_per_job formula already grants automatically.

Override knobs#

Env var

Effect

PARALLEL_JOBS=N

Hard override — bypasses all auto-detection for every helper.

AGGRESSIVE_PARALLELISM=true|false

Force aggressive on/off (auto ≥16 GB).

BUILD_MEM_DIVISOR=N

Divide usable RAM by N. Set = #concurrent builds. See Max speed.

DEFAULT_MB_PER_JOB

Per-job estimate for generic C/C++.

RUST_MB_PER_JOB

Per-job estimate for Rust.

CPP_HEAVY_MB_PER_JOB

Per-job estimate for torch/heavy C++. See warning.

🚀 Max speed — the real lever is --parallel-archs, not the per-job numbers#

On any ≥16 GB host the per-job math is already optimal: generic and rust run at full core count (aggressive mode), and torch is RAM-bound at 4 GB/TU (can’t go faster without more RAM — see below). There is no idle throughput to reclaim by tuning *_MB_PER_JOB for a single build.

The wall-clock win is at the orchestration layer. build-cross-chain.sh builds the three arches of each per-arch stage (sdk / media / android) sequentially by default. --parallel-archs runs them concurrently — close to a 3× speedup on those stages.

The catch it used to have: each per-arch build runs its own parallelism.sh, which reads the full host RAM. Three concurrent torch builds would each launch RAM/4096 jobs → 3× overcommit → OOM at hour six.

The fix (now built in): BUILD_MEM_DIVISOR. When the orchestrator runs N arches in parallel it passes BUILD_MEM_DIVISOR=N into each build; every helper then sizes against usable_RAM / N, so the sum of all N builds’ jobs still fits one host. Sequential builds pass nothing (divisor 1) → unchanged.

# 62 GB host, 3 arches in parallel:
#   without divisor:  3 × (62000/4096)=15 jobs × 4 GB = 184 GB  → OOM
#   with DIVISOR=3:   3 × (20666/4096)= 5 jobs × 4 GB =  61 GB  → fits

Because a container can’t know how many siblings share the host, the divisor must be injected — it is the one number auto-detection can’t supply. Wiring: the orchestrator computes min(--max-parallel-archs, #arches) and forwards it as a build-arg → ENV BUILD_MEM_DIVISOR in each heavy Dockerfile.

Before the first --parallel-archs run, validate the divisor actually reaches the container (grep a build log for the job count on a cheap stage), and watch dmesg | grep -i oom. Trading 3× disk + bandwidth too — the parallel arches all push large images at once.

⚠️ The trap: average RAM usage lies#

free//proc/meminfo during a heavy build shows low average usage and lots of “free” RAM — because heavy TUs are staggered, so most samples catch the light ones. This is not slack you can reclaim.

Worked example (this host, 2026-07, riscv64 torch build):

  • Host: 32 cores, 62 GB RAM.

  • compute_cpp_heavy_jobs @ 4096 MB/job → ~14 concurrent jobs.

  • free snapshots during the build: ~14–18 GB used, ~44 GB free. 🚩 looks wasteful.

  • But torch’s aten/autograd TUs peak ~4 GB each. Worst case when the heavy ones align: 14 × 4 GB = 56 GB — fits under 62 GB with a thin margin. ✅

  • Lower it to 2500 “because 44 GB is free” → ~24 jobs × 4 GB = 96 GB → OOM, build dies at hour six. ❌

This exact failure is already recorded in code: at the generic 2 GB/job estimate, 27 jobs × 4 GB overcommitted a 60 GB host and the OOM-killer terminated cc1plus (see the comment above compute_cpp_heavy_jobs). 4096 is the tightest value that keeps the worst case under 62 GB. It is not padding.

How to tune safely (procedure for an agent)#

  1. Identify the binding stage. Only stages using compute_cpp_heavy_jobs are RAM-capped on a ≥16 GB host. Generic-C++ and Rust already run at core count — no config change speeds them up; don’t touch their estimates.

  2. Measure the PEAK, not the average. While the target stage compiles, sample the largest single cc1plus/rustc RSS, not total used:

    # peak RSS (MB) of the biggest compiler process, sampled repeatedly
for _ in $(seq 60); do
  ps -eo rss,comm --sort=-rss | awk '/cc1plus|cc1|rustc|lto/{print $1/1024; exit}'
  sleep 5
done | sort -n | tail -1

  3. Apply the invariant. Only lower mb_per_job if:

    (target_jobs) × (measured_peak_MB) ≤ usable_RAM_MB × 0.9   # keep ~10% headroom

    Solve for the max safe target_jobs, then set mb_per_job = usable_RAM_MB / target_jobs. Never set it below the measured peak.

  4. Verify on a real build. Watch dmesg -w | grep -i oom and the compiler logs for Killed/signal 9 on the first run after a change. If anything is OOM-killed, revert immediately.

RAM upgrade guidance (corrected)#

  • torch build time is peak-RAM-bound. You can only fit usable_RAM / 4 GB heavy TUs at once. On 62 GB that’s ~14 jobs on 32 cores — cores sit idle not because of a bad estimate, but because RAM can’t feed more 4 GB TUs. → More RAM genuinely speeds up the torch stage (e.g. 128 GB → ~30 concurrent → roughly 2× torch parallelism). This is the only real lever for torch; the config estimate must stay at 4 GB.

  • Everything else is already core-bound (aggressive mode), so more RAM does not help OpenCV / ONNX / Rust / GStreamer.

  • Bottlenecks that more RAM does not fix at all: disk headroom (the cross images are large), registry upload bandwidth (each per-arch stage ends in a slow push), and single-threaded serial work (GCC mega-TUs, final links).

Do not “free up” the torch estimate. If torch speed matters, add RAM.