# Build cache tiers (Linux cross lane) > **STATUS 2026-08-23: § 4 (S3 / T4) is DESIGN ONLY — that code change was > written, reviewed and REVERTED.** The first implementation reintroduced the > literal `${tag}-buildcache` cache ref, which `verify-critical-fixes.sh` has forbidden > since fix7 (2026-07) precisely because that ref was self-defeating; the > preflight gate failed on it immediately. It was also inert by default and had > no regression coverage, so a future "just turn it on" edit would have flipped > ~12-13 GB of extra registry traffic per run with nothing asserting the > behaviour. § 4 is therefore the design and the constraints any implementation > must satisfy — read the flake-recovery interaction below before writing code. > § 5 is not covered by this banner: the ccache→sccache switch it describes is > shipped and running. **Audience:** anyone about to "add a registry cache", "turn on `mode=max`", "wire sccache", or free disk by deleting a cache directory. This is the *why* companion to the one-screen map in [`linux-build-basics.md` § Caching Layers](linux-build-basics.md#caching-layers-what-is-cached-where). That table says what is cached where; this document says **what each tier does NOT cover, what destroys it, and what it costs to replace it** — the three questions that produced an 18-hour cold rebuild (backlog S3, BIT 2026-08-21). Scope: the Linux cross lane (`build-cross-chain.sh` → `01-core/cross-stage-build.sh`). The Windows lane keeps its own cache story in its own docs; nothing here proposes work there. --- ## 1. The tiers | # | Tier | Wired at | Covers | Stored in | Destroyed by | |---|---|---|---|---|---| | T0 | BuildKit layer/history cache | implicit (the worker) | **every** vertex, including intermediate `FROM … AS x` stages | the buildkitd store | `nerdctl system prune`; buildkitd GC (`gckeepstorage` in `~/.config/buildkit/buildkitd.toml`) | | T1 | exec cache mounts (ccache, sccache, apt, cargo, uv, pip, source trees) | `--mount=type=cache,…` in the Dockerfiles | compiler/package output at translation-unit / file granularity | the buildkitd store (`exec.cachemount` records) | `nerdctl system prune` (**35 → 1 records observed 2026-08-21**). Survives `linux/host-config/prune-safe.sh` and a daemon restart — both proven repeatedly | | T2 | local stage cache export | `cross-stage-build.sh:193-197` (`--cache-to type=local,…,mode=max`) | **every** vertex of that stage's Dockerfile | `~/.cache/kata-buildcache//` — *outside* the buildkit store | our own disk guard (LRU below `CROSS_DISK_GUARD_GB`=40 G free, plus the `CROSS_CACHE_MAX_GB`=250 G total cap, `build-cross-chain.sh:523-614`); the preflight, watchdog and lane-entry trims (§ 3.1, § 3.2); the disk-preflight's `rm -rf` hint (`build-cross-chain.sh:467`); `CROSS_NO_LOCAL_CACHE_EXPORT=1` stops new writes | | T3 | inline registry cache | `cross-stage-build.sh:200-205` (`--cache-to type=inline` + `--cache-from type=registry,ref=`) | **only the final image's own layers** (`mode=min`) | inside the pushed image config — no separate blob, so no ghcr 400 | a re-push of the tag; `ghcr-cleanup.yml` then deletes the untagged old version (every tagged version is kept; 14-day floor) | | T4 | per-stage registry cache, `mode=max` | **not enabled, and DESIGN ONLY** — the pilot knob `CROSS_REGISTRY_CACHE=max` was written and reverted with the rest of S3; no code reads it (`cross-stage-build.sh:178-207` adds the local `mode=max` export and, when pushing, `type=registry` cache-from + `type=inline` cache-to, and nothing else) | every vertex, off-host | a `-buildcache` ref on ghcr | see § 4 — the cost, not the mechanism, is the blocker | Two properties are worth internalising because they are counter-intuitive: - **T2 is not in the buildkit store.** `nerdctl system prune` does not touch `~/.cache/kata-buildcache`. The things that empty it are *ours*: the disk guard, its trims and the preflight hint. - **T0/T1 are in the same store**, so the one command that wipes layers also wipes the compiler caches. That is why Standing rule 3 exists. ### 1.1 The cerbero state cachemount (CERB-CACHE) `03-media/build/gstreamer/android/build-android-from-source.sh` keeps its cerbero state in a T1 exec cachemount (`linux/Dockerfile.android`, the `android-gstreamer` RUN, `target=/var/cache/cerbero`) with an unusual job: it makes a *failed* build resumable. bootstrap + package is ONE `RUN`, so before this mount existed any failure inside it discarded the whole ~40-60 min bootstrap — wave5k/5l re-paid it on every freedesktop flap, while wave6b restarted WARM (`HIT: resuming … 13G / 20G`) and finished with zero fetch failures. What lives there (cerbero's `home_dir`): `sources/local` (downloaded tarballs and the git repos it checks out), `sources/` (extracted + compiled build trees), `build-tools/` (the host bootstrap prefix), `rust/`, `dist/android_/` (the target prefix), and the `cache-file.cache` / `build-tools.cache` pickles recording which recipes are already built (`build/cookbook.py: _cache_file()`). Four properties worth internalising: - **A resume is not a re-verification.** Read against the pinned cerbero 1.29.2: `Oven._cook_recipe_step` returns before it ever calls the step function when that step is already in the pickle (`build/oven.py:511`), and `_cook_recipe` skips the whole recipe once `needs_build` is false (`:554`). The only caller of `Source.verify()` — the sha256 against the recipe's `tarball_checksum` — is the fetch step itself (`build/source.py:366/378/458`). Bytes reused out of `sources/local` are therefore never re-hashed: they buy speed, not trust. - **The pickle only invalidates on recipe content.** A recipe's status is thrown away when its `built_version` changes, or when the content hash of the recipe file plus its patch files changes (`build/cookbook.py:426-440`; `Recipe.get_checksum` = `files_checksum` over the recipe's own FILES, not over the tarball) — so the sed-patched recipes in that script (PKGCFG-MIRROR, soundtouch, glib libiconv) do invalidate themselves. Nothing in the pickle knows about the toolchain, which is why `ANDROID_NDK_VERSION` and `ANDROID_API_LEVEL` are part of the cachemount id: a pin bump must start a NEW store, because cerbero would happily resume a tree built against the old NDK. - **The state dir is deliberately NOT `/opt/cerbero`.** That path stays the git CHECKOUT: a mount target already exists when the RUN body starts, so the `[ ! -d cerbero ]` guard would skip the clone and `git clone` refuses a non-empty destination anyway. Separating them also un-collides cerbero's read-only seed dir `cached_sources` (= `/sources`) from the live tree. - **The disk trade-off, with the number, because the host runs near-full.** The prune at the end of the script is on the SUCCESS path ONLY: the package exists, so the extracted/compiled trees are dead weight and everything kept lives outside `sources/` (the dist prefix, the build-tools prefix at `config.py:1065`, the two pickles) plus `sources/local` — that is `local_sources` (`config.py:1146-1151`, because `home_dir` is non-default), the tarballs and git repos that make the next run cheap and network-independent (FD-OUTAGE). A FAILED attempt keeps its whole tree — that is the entire point of the mount, and it costs up to ~10-15 GB per lane, i.e. ~30-45 GB with all three arch lanes sitting on failures. Retries reuse the same cachemount id and the same paths, so a failing lane overwrites in place rather than accumulating a tree per attempt. Pruning on ENTRY instead was considered and REJECTED as unsafe: cerbero records progress per STEP, so a recipe can be mid-flight with `extract` recorded and `compile` not (`oven.py:511` then skips straight to a configure/compile on a directory the prune would have deleted) — that wedges the lane on every retry instead of bounding disk. `CERBERO_CACHE_RESET=1` empties the CURRENT store on entry when a lane must be forced cold. GENERATIONS are the unbounded axis: the id carries the NDK/API pins, so a pin bump starts a fresh store and ORPHANS the previous one, and nothing inside the build can reach it (a different cachemount id is simply not mounted into the RUN). `linux/host-config/prune-safe.sh` will not clean it up either — it prunes `type==regular` only and treats every cachemount as sacred by design (CACHE1: it aborts if the cachemount count drops). Reclaiming an orphan is a deliberate host-side act, e.g. a duration-bounded `buildctl prune --filter type==exec.cachemount --keep-duration ` chosen so the ccache/sccache mounts a live build keeps warm stay above the cutoff — check what it would remove before running it. The id keys the lane on `${TARGET_ARCH}`, never `${TARGETARCH}`: the stage's `ARG TARGETARCH` rebinds to BuildKit's default, the `--platform linux/amd64` that `cross-stage-build.sh` passes on every lane, so all lanes would share one store. The NDK/API pins expand from the `android-sdk` parent's ENV, where an ARG-only name would expand empty and merge every generation into one store. The tarballs stay per lane rather than in a shared id, because cerbero downloads them non-atomically next to its live per-lane git repos, and the `sharing=locked` that would need holds its lock for the whole multi-hour RUN. `${ANDROID_LIB}` is in the id so the RUN stays byte-identical across the five library stages while the four non-cerbero lanes get their own empty store. ### 1.2 The web-lane tool binaries (riscv-tools, 2026-09-23) Three T1 ids hold the from-source `wasm-pack` and `flutter_rust_bridge_codegen` ([contract](consumer-image-contract.md#building-the-web-lane-tools-from-source)): | id | Stage | Holds | | --- | --- | --- | | `web-lane-tools-cross-` | android, `web-lane-tools` | cross-built binaries, one entry per key | | `cargo-registry-web-lane-tools-` | android, `web-lane-tools` | crate sources for that producer only, `sharing=locked`: cargo's `.package-cache` lock sits outside the mount, as for the media lanes above | | `web-lane-tools-bin-` | package, the setup RUN | native builds, so a pin reaches QEMU once per key | An entry is `-_rust-__/` holding the binary and a manifest whose first eight lines are the key text. A read accepts it only when that text equals this build's exactly, the sha256 matches, and the binary passes the same gate a fresh build must pass; anything else is deleted with a `WARN` and rebuilt. Writes go to `.tmp..` and are renamed into place, and each tool keeps its newest three entries, so a pin bump replaces an entry rather than orphaning one. The ids themselves are fixed. The trust boundary is the one every cachemount here has: a binary and manifest forged consistently by someone who can write the buildkitd store pass. Since `RUNTIME_NO_CACHE=1` does not touch a cachemount, `WEB_LANE_TOOLS_CACHE=refresh` is the way to rebuild past a good entry; `off` neither reads nor writes. ### 1.3 Cache-mount ids and source checkouts `Dockerfile.media` keys its cache ids three ways on purpose. `${TARGET_ARCH}` (one store per lane) serves the `sharing=locked` apt mounts and the arch-specific ffmpeg SDK store; `${TARGETARCH}` serves ccache, sccache, uv and pip, which are unlocked and content-addressed, so a lane collision is free while a split would triple the cold compile; and `onnxruntime-web` carries no arch, because its wasm output is arch-independent and the lanes serializing on that one locked RUN is the price of one emscripten compile instead of three. `locked` serializes every consumer of an id, concurrent lanes included, and `private` hands each concurrent consumer a new empty directory, so it never buys sharing. The cargo registry is the odd one: its sources are arch-independent, but the gstreamer RUN patches crates in place and cargo's `.package-cache` lock sits at the `CARGO_HOME` root, outside the mounted subtrees, so concurrent containers on one id are unguarded. Source checkouts are not cache-mounted, and a mount could not be populated: `build-opencv.sh` and `build-ffmpeg.sh` delete their source dir on cleanup, `setup-gstreamer.sh` builds in a PID-suffixed dir, and ONNX Runtime clones into the image path `/opt/onnxruntime`. A future source cache must key its id on the pinned version through an `ARG` declared in the same stage above the RUN (an out-of-scope variable expands to nothing and collapses every version onto one store), and still serves a frozen tip whenever the ref is a branch. --- ## 2. The blind spot: what `type=inline` cannot carry `--cache-to type=inline` is `mode=min`. It records cache keys for the layers that end up **in the exported image** and nothing else. The media stage's expensive work is not in those layers. `Dockerfile.media` builds onnxruntime / litert / tvm / opencv / app-wheelhouse / armnn / ffmpeg / pyav / gstreamer / opencv-gst as **named intermediate stages** (`FROM base AS …`, `:309-1080`) and lifts the results out with `COPY --link --from=…` (`:911-918`, `:1168-1211`). The layers those COPYs produce are in the image; the RUN vertices that *built* them are not, and a `mode=min` export carries no cache record for them. So: > A host that still has the pushed image, and reads the inline cache back from > it, will still recompile every framework. This is not a theory, it has been measured twice: - **2026-08-09** — after a prune emptied the local media slugs, the next amd64 media run recompiled every framework (~1.5–2 h) *despite* the previous day's push (original S3 write-up, `refactoring-backlog-archive-2026-08-10.md:2010`). - **2026-08-21** — a `nerdctl system prune` took T0+T1, the media slugs were not there to save the run either, and the relaunch cold-rebuilt all media intermediates: ~18 h across torch/IREE/litert × 3 arches (backlog S3, BIT). So S3's premise is correct: **only a `mode=max` tier carries intermediates**, and today the only `mode=max` tier is the local one (T2). --- ## 3. Failure modes and the recovery path that already exists | Symptom | Cause | What the code does by itself | Operator knob | |---|---|---|---| | `DeadlineExceeded: failed to compute cache key: … httpReadSeeker … no active session` | ghcr cache **import** flake (killed 6 attempts across 2 lanes in one afternoon, 2026-08-18) | retry; after **2** such failures drop the registry cache-from + inline cache-to and finish on the local tier (`_cross_build_drop_registry_cache_after_flake`, cross-stage-build.sh) | `NO_CACHE_EXPORT=1` does the same thing up front | | `error writing layer blob: 400 Bad Request` on cache push | ghcr rejecting an oversized `mode=max` cache blob (why T4 was removed in `4f27634`; `195285f` replaced it with T2+T3) | **nothing** — 400 is not in the transient class (`_cross_stage_push_error_is_transient`), so the stage fails | `NO_CACHE_EXPORT=1` | | A stage fails after hours of completed work → T2 exports **nothing** | `--cache-to type=local` only materialises on a successful solve (~8 h of arm64 media work exported ZERO once) | S1 salvage: re-drive the same build per named `--target`, each a pure cache hit, so the export lands anyway (`cross-stage-build.sh:209-247`) | `SALVAGE_CACHE_EXPORT=0` | | Disk below 40 G between stages | T2 growth (209 G measured today) | LRU-prune unprotected slugs; stages still to run in **this** chain are protected; if still short, stop writing new exports | `CROSS_DISK_GUARD_GB`, `CROSS_CACHE_MAX_GB`, `CROSS_NO_LOCAL_CACHE_EXPORT=1` | | `could not read …/kata-buildcache/…` | empty slug dir, i.e. a clean miss | suppressed: cache-from is only added when `index.json` is non-empty (`cross-stage-build.sh:186-190`) | — | | Chain refuses to start: `Insufficient disk: 19G free, ~180G recommended` | T2 grew 62 G → 110 G in ONE session (D4) and there is no room for the run | **preflight trim**: reclaim oldest-first from `~/.cache/kata-buildcache` up to the deficit, then re-measure and continue if it now fits (§ 3.1) | `CROSS_PREFLIGHT_TRIM=0`, `FORCE_LOW_DISK=1` | | A stage fails while disk is nearly full, and the salvage then writes GBs more | S1 salvage re-drives up to 15 named media targets, exporting cache for stages that get rebuilt anyway (D5) | skip the salvage, with a warning naming the free space and the threshold (§ 3.1) | `SALVAGE_MIN_FREE_GB` (`0` = always salvage), `SALVAGE_CACHE_EXPORT=0` | | Sessions die after 1–2 h of parallel load | BKD1 (buildkitd session rot) | — | restart buildkitd between rounds; cachemounts provably survive | **The rule any new cache tier has to obey:** it must be *droppable on the flake path*. The auto-recovery works by removing argument pairs from `build_cmd` before the retry; a tier that is not in that case statement keeps hammering a registry that is already timing out. ### 3.1 Preflight trim (D4) and the salvage disk gate (D5) Both landed 2026-08-31, after the r3 disk emergency forced a controlled chain stop at 19 G free. **Preflight trim.** `_chain_disk_preflight` used to *print* `rm -rf ${bc_dir}/*` and abort, leaving the reclaim to a human. It now calls `_disk_guard_trim_cache_export` (`01-core/disk-guard.sh`) first: * target = the same `need_gb` the preflight already computes; * budget = `need_gb - free_gb`, i.e. **the deficit only** — the loop stops the moment either the target or the budget is reached, so it trims instead of emptying the directory. Without that bound a second process eating disk would turn the trim into `rm -rf` of every slug; * victims come from `_disk_guard_pick_victim`, so removal is strictly oldest-mtime-first and honours a protected list (the preflight passes none — nothing has run yet, and protecting every upcoming stage would protect everything); * every removal is logged with its size, plus a `removed N slug(s), freed X GiB` summary and the resulting free space. A silent reclaim is how people stop trusting a tool. Then free space is re-measured. If it now clears `need_gb` the chain proceeds; otherwise the old hint-and-abort path runs unchanged. `CROSS_PREFLIGHT_TRIM=0` skips the trim entirely and restores the pre-2026-08-31 behaviour. **Why this is safe.** T2 is a *cache export*, not the buildkit store. Losing a slug costs export reuse for that stage and nothing else — no ccache, no sccache, no cachemount. The trim only ever `rm -rf`s directories directly under `BUILDKIT_CACHE_DIR`; it never invokes any prune. The buildkit store is reached only by the **filtered** `buildctl prune` of § 3.2.1, and by `linux/host-config/prune-safe.sh` by hand. `nerdctl builder prune` and `nerdctl system prune` are never run by anything in the chain. **Salvage disk gate.** `_cross_salvage_disk_ok` gates the S1 salvage on free space: below `SALVAGE_MIN_FREE_GB` (default `CROSS_DISK_GUARD_GB`, i.e. 40 G) the salvage is skipped and says so, because those stages are rebuilt anyway and the export is the last thing that should run when disk is scarce. Unknown free space, a non-numeric threshold, or `SALVAGE_MIN_FREE_GB=0` all keep the old always-salvage behaviour; `SALVAGE_CACHE_EXPORT=0` still disables it outright. Unit coverage: `linux/scripts/tests/test-disk-guard.sh` (oldest-first order, budget respected, no-op when ample, protected slugs, never aborts under `set -euo pipefail`) and `linux/scripts/tests/test-cache-salvage-gate.sh`. --- ### 3.2 In-stage disk watchdog and the runtime-lane gate (B2) Landed 2026-09-01, after the r2 rebuild died with `no image was built` on the amd64 wrapper export. **What actually happened.** `_chain_stage_disk_guard` runs *between* stages only. The runtime lane is ONE stage that contains three per-arch wrapper builds, so nothing sampled disk inside it. The evidence from `out/rebuild-20260901-r2/chain.log` (483 MB) and its resource CSV: | time | free | what | |---|---|---| | 02:55:18 | **88 G** | android done → between-stage guard runs, 88 G > 40 G, silent no-op | | 02:55–03:23 | 88 → **4 G** | amd64 wrapper build + export, no sampling at all | | 03:23:12 | 4 G | `level=fatal msg="no image was built"` → push fails 4× → `Arch amd64 failed during build` | `grep -c disk-guard` over the whole log returns **0**. The last *measured* headroom was comfortable; the drain happened entirely inside the stage that followed. **The watchdog.** `_chain_disk_watch_start` backgrounds `_disk_guard_watch_loop` for the duration of the runtime helper and `_chain_disk_watch_stop` reaps it on **both** the success and the failure path. Each tick: * logs `[disk-watch] G free on ` — an unconditional sample, so the chain log alone reconstructs the disk trajectory of a finished run. This is also what makes an *external* reclaim visible: the two large mid-run jumps in the r2 CSV (+138 G at 02:03, +271 G at 03:25) were operator prunes in another terminal and left no trace anywhere in the chain's own log; * below `CROSS_DISK_GUARD_GB` it calls the same `_disk_guard_trim_cache_export` as § 3.1 — keep-floor and all — and then `_disk_guard_reclaim_record`, which emits exactly one `[disk-reclaim]` line per reclaim **including the reclaim that freed nothing**. That last case is a `WARN`: it is the one an operator must act on, and it is what silence used to look like. It never calls `nerdctl system prune` / `builder prune`; those wipe the `exec.cachemount` records and cost hours (§ 7). When the trim leaves the disk still short, the watchdog falls back to a *filtered* buildkit prune instead of giving up — § 3.2.1. **The lane-entry gate.** `_chain_runtime_lane_disk_gate` runs immediately before `build-runtime-manifest.sh`. It requires `CROSS_RUNTIME_LANE_GB` (default **120**) × concurrency free, trims first, and then refuses. `FORCE_LOW_DISK=1` or `CROSS_RUNTIME_LANE_GB=0` override it. *Why concurrency and not arch count.* 120 G is the transient cost of **one** wrapper build, measured off the r2 CSV. The chain runs the lane's arches sequentially — it does not forward `--parallel-archs` to `build-runtime-manifest.sh` — so the requirement is 120 G once, not 360 G. Sizing it per-arch would have refused essentially every run on this 915 G host (218 G free, 26 G of trimmable cache export) — a gate that refuses everything gets switched off, and then it guards nothing. The start-of-run preflight gained the same term as an **advisory** warning on top of its existing `need_gb`, for the same reason: at chain start the earlier stages' consumption is still unknown, so the honest enforcement point is lane entry, where free space is actually measured against what is about to be spent. The r2 run would have been refused at 02:55 with 88 G — hours before it died, and after a trim that might have rescued it. ### 3.2.1 The buildkit-store fallback (DISK1) Landed 2026-09-04, after the 2026-09-03 runtime run died at **4 G** free while **415 G** sat in `~/.local/share/buildkit`. The watchdog fired, trimmed `~/.cache/kata-buildcache`, freed nothing — the dir held exactly three slugs and `CROSS_TRIM_KEEP_SLUGS` is 3 — and logged `NOTHING was reclaimable`. The manual rescue, `PRUNE_KEEP_GB=120 linux/host-config/prune-safe.sh`, freed **223 G in 36 s** with all **97** cache-mount records surviving. `_disk_guard_buildkit_fallback` is that rescue, wired **after** the cache-export trim at all three places the chain reclaims — `_disk_guard_watch_once` (the in-stage sampler, where 2026-09-03 actually bit), `_chain_stage_disk_guard` (between stages, just before it gives up and sets `CROSS_NO_LOCAL_CACHE_EXPORT=1`) and `_chain_runtime_lane_disk_gate` (lane entry, between the trim and the `[disk-reclaim]` record). The two chain gates are the ones that REFUSE a run, so a fallback unreachable from them could not save the run it was written for. It re-measures free space and returns immediately unless the trim left the path still under the threshold. Only then does it run ``` buildctl prune --filter type==regular --keep-storage ``` the same filtered form `prune-safe.sh` uses. `nerdctl builder prune -f` is not an alternative and never becomes one: it has no `--filter`, so it deletes the `exec.cachemount` records too — ccache, sccache, uv, cargo, apt — and cost 1.5–2 h of cold LLVM on 2026-08-17 (§ 7). `prune-safe.sh` remains the operator-facing tool with the full before/after inventory; the guard is the in-chain, non-interactive subset. **Why 120 G of keep-storage, when the guard fires at 40 G free.** The two numbers measure different things and do not compete. `CROSS_DISK_GUARD_GB` (40) is *free space on the filesystem* — the point at which a stage is about to ENOSPC. `CROSS_BUILDKIT_KEEP_GB` (120) is *how much layer cache stays in the store* — the reuse the remaining stages still need. A prune that clears the 40 G emergency while keeping 120 G of layers satisfies both, which is exactly what the 2026-09-03 rescue did on a 415 G store. Below ~100 G of retained layers the next stages start recompiling what they should have pulled from cache (the `rebuild-disk-management` note), so a non-zero `CROSS_BUILDKIT_KEEP_GB` under 100 is raised to 100 and logged. `0` is the explicit "reclaim everything regenerable" escape hatch and is left alone. When the store is smaller than the keep value the prune is a no-op and the log says it freed 0 G — honest, and the operator still gets the `NOTHING was reclaimable` warning from the reclaim record. **The keep value is layer cache, so the cache mounts are added to it.** BuildKit's `--keep-storage` bounds the *whole* store: `cache/manager.go` (v0.33.0) sums every unshared record, of every type, before the type filter is applied, and deletes filtered records until that sum fits. The `type==regular` filter never frees cache mounts or local sources, so a bare `120000` against 196 GB of cache mounts (2026-10-01) deletes every idle layer record. That is how `PRUNE_KEEP_GB=100 prune-safe.sh` once left 0.17 GB (CON53); the same call "kept" 91 GB later that day only because a running chain held those records. `_disk_guard_regular_keep_mb` now adds the unprunable records' size, read from `buildctl du --format '{{json .}}'`, so 120 means 120 GB of layers. A cache mount in use by a running build reports 0 B there, so mid-run the result is a floor. Proof: `test-disk-guard.sh` and `test-prune-safe.sh`. **Once per reclaim episode.** The watchdog samples every `CROSS_DISK_WATCH_SECS` (120 s). After the first prune the store is already at `--keep-storage`, so a repeat would walk the whole store — 36 s of I/O during a running build — and free nothing. The second and later samples log `already pruned once here` and stop. An episode is one whole stage for the sampler, which is a backgrounded subshell and so latches on its own; for the two chain gates, which run in the orchestrator process and are hours apart, it is one invocation. Those call `_disk_guard_reclaim_begin` first — without it the store the base stage pruned would still be latched when the runtime lane refuses, which is the same "reclaim unreachable from the gate that refuses" defect one level down. **What it logs.** Every arm is greppable as `[disk-buildkit]`: | line | when | |---|---| | `G free < G after the cache-export trim — pruning type==regular layer cache with --keep-storage G` | the prune is about to run | | `reclaimed G of layer cache in s (G -> G free; keep-storage G)` | after it, always — including `reclaimed 0G` | | `all cache-mount record(s) survived` | the filter held | | `cache-mount records dropped -> ` (WARN) | it did not; compile caches are gone | | `SKIP: no buildctl on PATH` (WARN) | tool missing | | `SKIP: buildkit store unreachable` (WARN) | `buildctl du` will not answer | | `already pruned once here` | the latch | | `disabled (CROSS_BUILDKIT_PRUNE=0)` | the knob | The reclaimed GB also reaches `_disk_guard_reclaim_record`, so the single `[disk-reclaim]` line per reclaim now reads `… freed X GiB + 223G of buildkit layer cache`, and `NOTHING was reclaimable` is emitted only when *both* tiers came up empty. Pinning that give-up line as correct behaviour is what the old suite did. **Degradation.** A missing `buildctl`, an unreachable socket, a non-numeric target or an unknown free-space reading all return 0 after one WARN, leaving exactly the pre-2026-09-04 behaviour. The fallback never aborts the stage it samples — it is a watchdog, and a watchdog that kills the build it watches is worse than the drain. Unit coverage: the DISK1 section of `linux/scripts/tests/test-disk-guard.sh` (trim-first ordering, the filter and keep-storage argument, the keep floor, the once-latch, the survival count, both SKIP arms, and a structural assertion that `disk-guard.sh` does not name `nerdctl` or `builder prune` anywhere), the DISK2 cases in `test-chain-lifecycle.sh` (both gates reach the store before they refuse, and each opens its own episode), plus the `disk-guard.*` mutations. **What DISK2 left open, measured on the live store 2026-09-05** (buildkit v0.31.2, 219.13 GB total / 187.99 GB reclaimable, no build running): * `--keep-storage` is documented by `buildctl prune --help` as **MB**, so the guard's `keep_gb * 1000` really is ~120 GB. The convention was inherited from what worked in anger; it is now checked against the installed flag. * the readiness probe is not the expensive half: bare `buildctl du` and `du --filter type==exec.cachemount` both return in **under a second** on that store, and the second reports the same **97** records the 2026-09-03 rescue saw. The 36 s belonged to the prune. * `BUILDKIT_HOST` defaulting to `unix:///run/user/$(id -u)/buildkit/buildkitd.sock` resolves for the chain's own uid on this rootless host — `buildctl du` answers rc 0 — so `_disk_guard_buildkit_ready` passes rather than logging its SKIP. Still unanswered, and still needing a real run: whether a prune issued *while a build holds records open* reclaims anything like the idle number, and whether 120 G of retained layers avoids recompile churn afterwards. The 2026-09-04 `--only runtime` run entered with 275 G free and never crossed 40 G, so it sampled (`[disk-watch]` down to 106 G) and reclaimed nothing. ### 3.2.2 The image-store lever (DISK3) Landed 2026-09-07, after the 2026-09-05 rebuild reported this at 28 G free: ``` [INFO] [disk-trim] removed 0 slug(s), freed 0.0 GiB; 28G free now [INFO] [disk-buildkit] already pruned once here -- the store is at keep-storage [WARN] [disk-reclaim] in-stage: NOTHING was reclaimable (28G -> 28G free) ``` It was right that it could not free anything and wrong that nothing was reclaimable. At that moment `~/.local/share/containerd` held **295 GB**, including three `cross-android-*` stage images from a PREVIOUS run at 41.5 / 41.8 / 38.0 GB. `disk-guard.sh` contained no image listing at all: it knew its own cache-export slugs and BuildKit, and BuildKit was already at `--keep-storage`, so its two levers were spent while its third, larger one was invisible to it. That run needed four manual rescues. **The ordering constraint is the whole design, and it is not a preference.** `buildctl prune` never touches the containerd image store and is safe while a stage runs — that is why the in-stage sampler may pull it. Removing IMAGES is safe only when nothing is in flight: `nerdctl image prune -f` killed the arm64 runtime lane on 2026-09-06 while the chain was `unpacking overlayfs@sha256:…`, the OCI stage handoff, and the prune took a blob out from under it (`content digest sha256:d694…: not found`). The same command had worked several times earlier that day, which is luck, not safety. So `_disk_guard_image_store_fallback` takes a `stage_in_flight` argument and refuses BY NAME when it is 1. `_disk_guard_watch_once` — the sampler, which runs DURING a stage by definition — passes 1 and therefore never removes anything; the two gates that run between runs (`_chain_stage_disk_guard` after the arch loop has joined, and `_chain_runtime_lane_disk_gate` before any wrapper build starts) pass 0. `CROSS_IMAGE_PRUNE=0` disables the lever entirely. **Two steps, in risk order.** First `nerdctl image prune` with no `-a`: dangling images only, zero risk, and 20 GB on its own in the run that found this. Then this chain's own `cross--` tags that the rest of the run does not need. Protected are the stages still to build AND the one just completed, whose image is the next stage's parent under the local OCI handoff; everything else in that shape is re-pullable from ghcr by the digest each stage pins. Nothing else is a candidate — not `ubuntu:24.04`, not an untagged image (that is the dangling prune's business), and never `-a`, which would take the parents this run pinned. **Size is not the metric — unique layers are.** Deleting the three `cross-sdk-*` images (80 GB by `nerdctl images`) freed *zero* bytes in that run, because every layer they hold is also held by the `cross-android-*` images built on top of them; the previous release's `latest-cross*` freed 113 GB from a similar nominal size, because its layers were nobody else's. A guard that picks the biggest tags does nothing, so this one measures free space after EACH removal and stops the moment the target is reached. Each removal logs what it actually freed. A tag still listed after its own `rmi` joins the protected set rather than staying the head of the list forever — the same spin the cache-export victim loop guards against. **The give-up warning names the store it did not look in.** When nothing was reclaimable, `[disk-reclaim]` now adds how many tagged images the store holds and says *stop the lane, then reclaim* — because "NOTHING was reclaimable" read like an environment limit when it was a coverage gap. `linux/host-config/prune-safe.sh` remains the operator-facing tool for the BuildKit half. ### 3.3 Runtime-failure summary (B3) When one arch fails, `set -e` aborts `build-runtime-manifest.sh` inside `run_parallel_arch_loop`, and **every** gate downstream of that loop is skipped: the wrapper content gate, `verify-shipped-wrapper.sh`, the runtime-image smoke (and with it `assert_pinned_versions`), and the manifest coherence / completeness / freshness gates. In r2 the run ended with a bare `[ERROR] runtime stage failed`; `grep -c` returns 0 for all of them. `_chain_runtime_failure_report` now runs on that failure path. It reads each wrapper tag's own run-id stamp (`ancestry_recorded_run_id`, the same provenance the coherence gate uses) and classifies the arch as `built-this-run`, `stale` or `missing`, then: * names the arches with no image from this run; * names the gates that were therefore skipped — but **only** in that case. If every arch carries this run's id the failure is at or after the gates, and claiming a skip would be a lie; * writes both into `chain-status.json` as `arch_outcomes` and `gates_not_run`, so a later `--manifest-only` repair cannot assemble an index believing everything was checked. A green run's file is unchanged — the keys appear only when a failure recorded them. Unit coverage for § 3.2–3.3: `test-disk-guard.sh` (sampling on every tick, the reclaim record, the no-op-reclaim warning, the loop actually looping, the concurrency sizing) and `test-chain-lifecycle.sh` (the lane gate going red at 88 G, its three overrides, per-arch classification, the "do not claim a skip" branch, the JSON fields, and call-site wiring assertions for all of it). --- --- ## 4. S3 — per-stage registry cache refs (`mode=max`) ### 4.1 What it would buy Exactly the gap in § 2: intermediate stage layers, off-host, surviving a store wipe. Had it been in place on 2026-08-21 the relaunch would have been a fast-forward instead of ~18 h. ### 4.2 What it costs — measured, not estimated The local `mode=max` exports are *the same content* a registry `mode=max` ref would hold, in the same compressed form, so the disk sizes are a direct proxy for the bytes on the wire (± the export compression setting). Measured on this host, 2026-08-23 (`du -sh ~/.cache/kata-buildcache/*`): | slug | size | |---|---| | `cross-media-amd64` | 52 G | | `cross-media-arm64` | 49 G | | `cross-media-riscv64` | 34 G | | `cross-android-amd64` | 27 G | | `cross-android-arm64` | 26 G | | `cross-android-riscv64` | 24 G | | **total** | **209 G** | The measured uplink is **~4–5 MB/s** (the PUSH1 note at `cross-stage-build.sh:161-162`; PUSH1 also measured a media *image* push at ~10 min with zstd, i.e. ~2.7 GB of new layers). At 4.5 MB/s: | what | bytes | added push time | |---|---|---| | one media arch | ~50 G | **~3.1 h** | | all three media arches | 135 G | ~8.5 h | | all three android arches | 77 G | ~4.9 h | | everything | 212 G | **~12–13.5 h per chain run** | **Verdict: a blanket per-stage `mode=max` export to ghcr is net negative here by roughly an order of magnitude.** It would add ~12 h to *every* chain run to insure against an ~18 h loss that has happened twice in a fortnight. On a 100 Mbit uplink the same design would be obviously correct — the numbers, not the idea, are what disqualify it. Re-run the table above before re-opening S3 if the uplink ever changes. Secondary costs, for completeness: - **Registry storage.** When this was costed, `ghcr-cleanup.yml` kept 3 versions *per tag*, so 11 stage tags × 3 generations of cache was up to ~600 G of cache blobs retained. Since 2026-09-08 it keeps the tagged version and deletes untagged ones after 14 days (`ghcr-prune-package.sh`). Its referential-integrity pass (added 2026-08-11) resolves kept tags' manifests, so it will not orphan a live cache ref — but the package listing doubles in size. - **Failure surface.** A cache export that 400s fails the stage outright (§ 3), and it does so *after* the image already pushed. ### 4.3 Where it *does* pay: the cheap shared stages The value ratio is `recompute_time / push_time`, and it inverts by stage: | stage | cache size | push at 4.5 MB/s | cold recompute | verdict | |---|---|---|---|---| | `compiler` (GCC + LLVM) | not currently on disk; the archive recorded 12 G-class `sdk` slugs | ~45–60 min for a 12–16 G cache | LLVM alone was ~50 min *warm* and projected 11 h cold (PAR4 note, CHANGELOG 2026-08-21) | **plausible** — and it is built once, not per arch | | `sdk` | ~12 G (archive) | ~45 min | ~1 h class | marginal | | `media`, `android` | 24–52 G per arch | 1.5–3.3 h per arch | 1.5–2 h per framework | **no** | Read that table with the right arithmetic: the push cost is paid on **every run**, the recompute saving only on the **rare** run that starts with the local tiers gone. For media the premium is ~3 h per arch per run against a payout of roughly one cold media rebuild (~6 h per arch, from the 18 h / 3 arches figure) — it only breaks even if the local tier is lost more often than every other run. It is not; it has been lost twice in a fortnight. So if S3 is implemented at all, it is implemented **for `compiler` (and maybe `sdk`) only**, never chain-wide. ### 4.4 The shape that dodges the cost entirely A re-drive of an identical build is a pure cache-hit solve — S1 salvage already depends on that property and it is proven in production. Therefore the cache **export does not have to sit on the build's critical path**: > After a green chain, re-drive the stage you want insured with the registry > cache export configured. Every vertex hits, nothing recompiles, and > the multi-hour upload happens in a window you chose, can interrupt, and whose > failure cannot kill a build that already succeeded. That is the recommended S3 shape: an out-of-band, opt-in "cache snapshot" pass, not a flag on every stage build. It also composes with the flake recovery for free, because it is not on the retry path at all. Cheaper still, and worth doing first because it costs zero bytes: 1. **Fix the preflight hint.** `build-cross-chain.sh:467` tells the operator to `rm -rf ${bc_dir}/*` to reclaim space, describing T2 as "regenerable" — it is, at ~18 h. The hint should point at the LRU path (the disk guard's own ordering, or `prune-safe.sh`), not at the wholesale delete of the only tier that holds intermediates. *(Filed, not fixed here: that file is outside this change's scope.)* 2. **Never `nerdctl system prune`** — Standing rule 3, already sharpened. ### 4.5 The pilot knob, and the rules it must obey `CROSS_REGISTRY_CACHE=max` is **DESIGN ONLY** — read the STATUS banner at the top. It was written so the experiment above could be run **without editing code mid-wave**, and it was reverted along with the rest of S3; `grep -rn CROSS_REGISTRY_CACHE` over the tree matches these docs and nothing else, so setting it today is a silent no-op. What follows is the shape a re-implementation has to take. In the reverted version it was off by default and the default path was byte-identical to what shipped before it existed (verified by dry-running the old and new function side by side and diffing the assembled command) — keep that property. When set to `max` the stage would additionally get: ``` --cache-from type=registry,ref=-buildcache --cache-to type=registry,ref=-buildcache,mode=max,image-manifest=true,oci-mediatypes=true ``` `image-manifest=true,oci-mediatypes=true` is there because the cache is then exported as a plain OCI image manifest rather than the index shape that motivated the original ghcr 400. **That is a hypothesis this repo has not yet tested** — proving or disproving it is the first job of the pilot. Rules the reverted implementation satisfied, and that any future one must keep: - **It composes with the flake auto-recovery.** The DeadlineExceeded/ `httpReadSeeker` handler must strip `--cache-to type=registry*` alongside the registry cache-from and the inline cache-to, so two import flakes still degrade the build to local-only instead of leaving a multi-GB export pointed at a registry that is timing out. The reverted implementation was verified by driving the real `_cross_stage_build_impl` through four failing attempts: attempts 1–2 carried all four registry/inline arguments, attempts 3–4 only the local `--cache-to`. **The handler that ships today strips only `--cache-from type=registry*` and `--cache-to type=inline*` (`cross-stage-build.sh:255-281`)** — because there is no registry export left to strip. Re-adding one means re-adding its case arm. - **`NO_CACHE_EXPORT=1` still turns everything registry-facing off**, because the knob lived inside that guard. - **A missing `-buildcache` ref on the first run is a miss, not a fatal.** In-repo evidence: `195285f`'s post-mortem of the never-written `-buildcache` ref — the importer failed and the stages rebuilt, the solve continued. **Pilot protocol** (do not skip a step; this is the item that has burned this repo before): 1. Pick `compiler`, and use `build-cross-stage.sh` — **not** a full chain. A cache-export 400 is non-transient and will fail the stage. 2. Run 1 (cold ref): re-implement the knob, then set `CROSS_REGISTRY_CACHE=max`. Confirm the build *accepts* the flags (nerdctl/buildkit here must support `image-manifest`), that the `-buildcache` ref appears in the ghcr package list, and record the wall time the export added. 3. Run 2 (warm): same command. Confirm the cache-from actually resolves and the solve reports cache hits for intermediate vertices, not just final layers. 4. Run 3 (the real test): move the local slug aside (`mv`, do not delete) and re-run. If the intermediates still fast-forward, T4 works and S3 is proven for that stage. If they do not, the tier bought nothing and the item closes. 5. Only after 3 green runs, and only for the stages in the § 4.3 "plausible" row, consider making it a default. In the reverted implementation, revert was `CROSS_REGISTRY_CACHE=inline` — or unsetting it, or `NO_CACHE_EXPORT=1`; any value other than the literal `max` gave exactly the shipped behaviour, and nothing else in the chain read it. That is the property to preserve in a re-implementation, not a knob to reach for today. --- ## 5. SCC1 — the ccache/sccache hybrid ### 5.1 The division of labour — REVERSED 2026-08-26 > **This section previously read "ccache owns C/C++, sccache owns rustc + the > GPU compilers … a full switch to sccache was rejected by the owner on > 2026-08-17". The owner reversed that on 2026-08-26 and the C/C++ switch is > implemented and built (5d94a37, c42091e, d5bafe8).** The old text is kept > nowhere but in git history, because a design doc that states a superseded > decision reads authoritative and gets followed. **sccache is the C/C++ compiler cache. ccache is the automatic fallback.** There is no per-language split any more, and every site resolves through ONE resolver: `compiler_cache_launcher()` (`01-core/common.sh:459-483`) direct from the 02-toolchain GCC/LLVM builds and the CMake/onnxruntime/wheelhouse call sites, and via `_resolve_compiler_cache_launcher()` on the media lane (`01-core/compiler-cache.sh:85-106` — the single seam that forwards to the common.sh resolver when 01-core is loaded, 2026-08-30 backlog F2). Either way: the guarded launcher if sccache's server answers, else ccache, else build uncached. It is never *bare* sccache in a stage that mounts `01-core`: each site starts at `sccache` and upgrades to `sccache-launcher.sh` as soon as one is executable, keeping the bare name only where the helper is absent (`common.sh:467-474`, `compiler-cache.sh:97-99`). The gate at `verify-critical-fixes.sh:98-119` is what stops the *hardcoded* bare form coming back — that is how the first cut shipped inert. All three load-bearing points of the old rationale still hold — the last one only after a detour through the config file: - **Still true:** ccache cannot wrap `rustc`, `nvcc` or `hipcc`; sccache can. - **Still true, and now the operating constraint:** sccache HARD-FAILS where ccache simply execs the compiler. That is not limited to unidentifiable compilers, and it bit hard during the switch: > **The TryCompile trap (root cause, 2026-08-26).** CMake creates a > `CMakeScratch/TryCompile-XXXX` directory, compiles a probe in it, and > DELETES it. sccache then spawns the compiler with that directory as the > working directory — and spawning a process whose cwd no longer exists fails > with `ENOENT`. Measured directly: spawning `/bin/true` with `cwd` set to a > removed directory returns errno 2, exactly what sccache reports. The same > trap in preprocessor-cache mode surfaces earlier, as > `while hashing the input file`, because that mode re-reads the source after > the compile. > > It killed the media stage three times, MOVING between OpenCV, onnxruntime > and litert depending on which probe lost the race — and it does not > reproduce against directories that persist. Nine isolated attempts came > back clean before the pattern was read correctly. The answer is `01-core/sccache-launcher.sh`: sccache runs for every compile, and only its OWN fatal errors ("sccache: encountered fatal error") fall through to running the compiler directly. A real compile error is passed through untouched — retrying blindly would hide genuine failures. In the run that proved it, the guard fired 1451 times with ZERO build aborts. ccache stays installed and mounted as the deeper fallback, and `probe-sccache.sh` exists — it asserts, per compiler SHAPE this chain feeds a launcher, that the compile survives AND that cache activity was recorded. - **True again, and on purpose:** "sccache's C/C++ path always preprocesses". The base image *does* bake `use_preprocessor_cache_mode = true` plus `file_stat_matches` / `use_ctime_for_stat` into `/etc/sccache/config.toml` (`Dockerfile.base:115-142`, reached via `SCCACHE_CONF`, `:102`) — those knobs have NO environment-variable path in sccache, which is why the config file exists at all. (It is not read while `SCCACHE_DIR` is set, which is always: [§ The shipped image's cache dirs](#the-shipped-images-cache-dirs).) And the runtime turns the mode off: both entry points export `SCCACHE_DIRECT=false` (`01-core/common.sh:419-433`, `01-core/compiler-cache.sh:132`), and the environment variable wins over the config file. That is the TryCompile trap seen from the other end — preprocessor-cache mode re-reads the INPUT FILE *after* the compile in order to store the entry, so a deleted scratch dir surfaces as `while hashing the input file` and is fatal. It killed OpenCV's compiler test, then onnxruntime's, three times running. **Effective state today: preprocessor-cache mode is OFF and every TU is preprocessed.** The cost is hit rate, not correctness; set `SCCACHE_DIRECT=true` to re-test it once the ephemeral-dir interaction is understood. `ignore_time_macros` stays **false** on purpose: ccache's sloppiness list included it, but that trades correctness for hit rate and this chain ships compilers. Two knobs have no counterpart and must not be looked for: there is no `sccache -M` (the cap is `SCCACHE_CACHE_SIZE`), and `CCACHE_COMPILERCHECK=content` needs none — sccache does not key on mtime+size the way ccache does by default, so the invalidation that setting prevents does not arise. **Rust rejoined this on 2026-08-27; nvcc did not.** The hard clear that used to sit in `build-gstreamer-monorepo.sh` is gone. `RUSTC_WRAPPER` resolves to the same guarded launcher from two places: `setup_sccache` exports it (`01-core/compiler-cache.sh:172-210`), and `build_gstreamer_monorepo` installs it for any process where the variable was never set at all (`build-gstreamer-monorepo.sh:649-659`). What made that safe is the UDS fix in § 5.4 (`compiler-cache.sh:62-76`, `common.sh:409-418`), not optimism — the deaths at 99 % were the *wrong server* answering, and the launcher is the second belt that turns a remaining sccache hiccup into lost hits instead of a lost build. To go back to uncached Rust, export `RUSTC_WRAPPER=""`; that is what `Dockerfile.toolchain:66` and `Dockerfile.package:262` do, and it holds for every stage that never calls `setup_sccache`. Where `setup_sccache` *does* run it overwrites the empty value (`compiler-cache.sh:199`), so the off switch on that lane is `USE_SCCACHE=0` — which also drops C/C++ back to ccache (`compiler-cache.sh:136-148`). nvcc and hipcc stay untouched (§ 5.4), and the Windows lane records that released sccache breaks the build around them. ### 5.2 Does the current setup double-cache or fight ccache? No — verified The precedence is explicit and correct: - `setup_ccache` (`01-core/compiler-cache.sh:108-170`) sets `CMAKE_C/CXX_COMPILER_LAUNCHER` to the **guarded launcher** (`01-core/sccache-launcher.sh`, resolved via `_resolve_compiler_cache_launcher`, `:85-106`) when `USE_SCCACHE` is not disabled, sccache is on `PATH`, and its server answers `--show-stats`; otherwise it falls back to **ccache** and says why. (Before 2026-08-26 it set ccache unconditionally. Its first cut at the switch hardcoded *bare* `sccache` here, which shipped inert; `verify-critical-fixes.sh:98-119` now fails any launcher in this file pointed at bare sccache.) - `setup_sccache` (`:172-210`) sets those two launchers **only if they are empty** (`:201`), and otherwise touches only `RUSTC_WRAPPER` (`:199`) — resolved the same way (`:193-198`): the guarded launcher where it is mounted, bare `sccache` only where it is not. - `media_common_init` (`03-media/core/common.sh:125-143`) always runs `setup_ccache` **first**, then `setup_sccache` only under `ENABLE_SCCACHE_RUST=1`. So on the media lane there is exactly one C/C++ launcher, and it is the guarded sccache launcher; ccache is used only when sccache's server does not answer. The LLVM cross build picks the same way round (`02-toolchain/llvm-cross.sh:394-399`, commented "preference INVERTED": it asks `compiler_cache_launcher` (`01-core/common.sh:459-483`), i.e. sccache first, ccache only as the fallback). The cache **mounts** do not collide either: `/var/cache/ccache` and `/var/cache/sccache` are separate ids. Their `${TARGETARCH}` (not `${TARGET_ARCH}`) suffix is deliberate and documented at `Dockerfile.media:26-31` — they are unlocked and content-addressed, so a lane collision is free while splitting them would triple the cold compile. **Do not "align" those ids with the apt ones.** ### 5.3 Three things that are not what the comments claim Found while reading for this document; all filed, none fixed here (the files are outside this change's scope): 1. **The `ENABLE_SCCACHE_RUST` gate is not the single control point it says it is — and since 2026-08-27 it barely controls anything.** `setup-gstreamer.sh:50` calls `setup_sccache` **unconditionally**, and `USE_SCCACHE` defaults to `true` (`compiler-cache.sh:19`, `Dockerfile.media` `ARG USE_SCCACHE=true`). So in that process `RUSTC_WRAPPER` is exported even with `ENABLE_SCCACHE_RUST=0`, and an sccache server is started. The counterweight used to be an explicit `export RUSTC_WRAPPER=""` inside `build_gstreamer_monorepo`; it is gone. That block is now the opposite — it *installs* the guarded launcher when `RUSTC_WRAPPER` is unset (`build-gstreamer-monorepo.sh:647-659`, sourced into the same process at `setup-gstreamer.sh:557-562` and only *called* at `:645`). The monorepo's Rust is therefore cached deliberately, and the gate does not gate it. What the gate still decides is one thing: `media_common_init` runs `setup_sccache` only under `ENABLE_SCCACHE_RUST=1` (`03-media/core/common.sh:136-143`), and that is the path `install-rice-proto.sh`'s `cargo cinstall` takes — a *different* process (`build-gstreamer-stage.sh:112`, before `setup-gstreamer.sh` is invoked at `:147`), which therefore still inherits the image's `RUSTC_WRAPPER=""` (`Dockerfile.toolchain:66`) and builds uncached. Between `:50` and `:645` the only cargo touch is `cargo --version` (`:447`). So the knob's name promises the monorepo and delivers rice-proto; read the two together before trusting either. 2. **A dead parameter.** `source_build_acceleration_helpers` takes `include_sccache` (`03-media/build/onnxruntime/build/lib/common.sh:193-213`), but all three call sites (`30-build-native.sh:7`, `-amd.sh:11`, `-nvidia.sh:26`) pass nothing, so the `true` branch has never run. 3. **`USE_CCACHE=false` does not disable compile caching — it migrates C/C++ to sccache.** With ccache off (or simply absent from `PATH`), `setup_ccache` returns early leaving the launchers unset, and any later `setup_sccache` fills them (`compiler-cache.sh:172-210`). There is no log line that distinguishes "ccache disabled" from "sccache silently took over C/C++". ### 5.4 Where sccache is actually worth it | target | gate | state | recommendation | |---|---|---|---| | **rustc** (gst-plugins-rs, the monorepo's Rust) | none any more — `ENABLE_SCCACHE_RUST=1` only reaches `media_common_init` (§ 5.3 item 1) | **ON by default since 2026-08-27**, through the guarded launcher (`compiler-cache.sh:172-210`, `build-gstreamer-monorepo.sh:649-659`) | [details](#rustc-gst-plugins-rs-the-monorepos-rust) | | **nvcc / hipcc** | `ENABLE_SCCACHE_CUDA=1` (one gate, three sites: `build-opencv.sh:647`, `30-build-native-nvidia.sh:244`, `30-build-native-amd.sh:65`) | wiring exists, default OFF | [details](#nvcc--hipcc) | | **C/C++** | — | sccache via the guarded launcher, always on; ccache is the automatic fallback | leave it — this is the owner-directed default since 2026-08-26 (§ 5.1), and both launcher resolvers already pick sccache first (§ 5.2). | | **cross-machine tier** (`SCCACHE_MULTILEVEL_CHAIN`, webdav L2) | — | Windows lane only | [details](#cross-machine-tier-sccache_multilevel_chain-webdav-l2) | ### Per-target detail The targets whose recommendation needs more than a table cell. #### **rustc** (gst-plugins-rs, the monorepo's Rust) **The one genuine win — and it was taken on 2026-08-27.** It is also the one that broke: sccache's server died mid-compile in three separate rounds ("Failed to send/receive data from server", "No such file or directory" on trivial crates), each time killing an otherwise-green gstreamer build at 99 %. That signature was root-caused on 2026-08-26 and it was never about Rust: the server is located by a fixed TCP port, which is not container-local, so concurrent BuildKit steps reached each *other's* server — one that cannot see their files. `SCCACHE_SERVER_UDS` took the media stage from 2359 sccache faults to zero (it then regressed on 2026-08-30 and was re-fixed on 2026-09-05 — [why](#the-server-address-must-be-exported-where-the-compiles-run)), so Rust caching came back, pointed at the guarded launcher rather than bare sccache (`build-gstreamer-monorepo.sh:647-659`); a server hiccup now costs hits, not a build at 99 %. The preconditions this section used to prescribe are already unconditional in code: `SCCACHE_IDLE_TIMEOUT=0` (`compiler-cache.sh:129`, `common.sh:390` — the Windows-lane forensics traced all-zero end-of-vertex stats to the server idle-exiting at 600 s), `SCCACHE_ERROR_LOG` (`compiler-cache.sh:135`, `common.sh:434`), and `sccache --show-stats` printed **to stderr**, the stream buildkit's 2 MiB step-log clip never cuts. **What is still open is the measurement:** two consecutive green cross-arch media runs with a non-zero *Rust* hit rate. Until those are on the board the re-enable is shipped but unproven — judge it by the stats line, not by the flag (§ 7). #### **nvcc / hipcc** **Do not flip on the strength of the hit-rate argument alone.** The theoretical win is large (`CUDA_ARCHITECTURES` compiles each kernel 4×) but a sibling lane produced an evidence-backed verdict that the pinned sccache's **nvcc decomposition silently drops device-conditional code** — a byte-identical link failure across two runs, falsified as cache poisoning, i.e. *silent wrong code*, disqualifying regardless of hit rate (CHANGELOG 2026-08-10/11). Any re-enable needs the three-canary bar recorded there: verify probe + a real kernel compile + a full link canary, on the sccache version actually installed. #### **cross-machine tier** (`SCCACHE_MULTILEVEL_CHAIN`, webdav L2) Out of scope for this document. Note only that the mechanism is real and version-gated: on an older sccache the chain variable is **ignored silently**, which is exactly the failure shape this repo keeps eliminating — so it would need a pinned sccache and a hit-rate assertion before it could be considered on the Linux lane. ### 5.5 The one-line answer to "does sccache fight ccache?" No — but the *gates* still lie about who is on. The honest summary is: C/C++ runs through the guarded sccache launcher everywhere, with ccache as the automatic fallback for a server that will not answer; Rust runs through that same guarded launcher since 2026-08-27, which `ENABLE_SCCACHE_RUST=0` does not stop (§ 5.3 item 1); only the GPU compilers are uncached, and that is a correctness verdict, not an oversight. Any change to that should be a measurement, not a default flip. --- ## The rules as AGENTS.md carried them, with their reasons Moved out of `AGENTS.md` on 2026-09-15. Apart from this heading, only its code pointers and counts have changed since. The RULES stay there, in four lines each; what is below is the evidence for them -- which file lists, which commits, which measured failure -- and it belongs with the tiers rather than in a guardrails file. Full map: `docs/linux-build-basics.md` § Caching Layers. Toggles: `USE_CCACHE`, `USE_SCCACHE`, `USE_LLD` accept `0/false/no/off` to disable (since 2026-08-08 — previously ONLY the literal `false` worked and `USE_CCACHE=0` was silently ignored); `ENABLE_SCCACHE_RUST`/`ENABLE_SCCACHE_CUDA` are strict `0/1`. The rules an agent must never violate: 1. **Closure freeze between runs that should cache-hit.** Editing ANY file in the base/toolchain closure changes the compiler image digest and forces sdk/media/android to rebuild from scratch on the next run. Since 2026-08-08 (A1 applied, commit 5d7a318) **`Dockerfile.base` mounts a traced per-file closure**, not whole directories — 15 bind mounts per block (12× `01-core` `.sh` + `versions.env` + `cmake.sh` + `packaging-deps.sh`; the lists live in Dockerfile.base itself), one block adding `cpython-dev-packages.sh` and one the `linux/vulkan` directory. The base/toolchain closure adds `python/build_python.sh`, the three bundled `06-packaging/smoke-*` scripts, `Dockerfile.base` and `Dockerfile.toolchain`. Editing 01-core files OUTSIDE those lists no longer busts base — but `Dockerfile.toolchain`'s verify layer (`# 3c. VERIFY TOOLCHAIN CONTRACT`, :216) still binds `01-core` and `02-toolchain` **whole**, so an edit anywhere in either directory re-runs it — and everything after it, Rust and the source-built CPython included. It sits after the GCC/LLVM compiles, so those still cache-hit: minutes, not hours. A file NEWLY needed by a base RUN must still be ADDED to the per-file mount lists (closure = source edges + **exec/`bash` edges**; the A1 validation build caught exactly such a miss). Batch closure edits; apply them in ONE commit at a planned rebuild boundary. Files in a not-yet-started stage's closure are free to fix until that stage begins (each `nerdctl build` snapshots its context at stage start). 2. **`~/.config/buildkit/buildkitd.toml` pins the GC budget** (`gckeepstorage`) so the multi-hour layers survive between runs. Restart buildkitd only BETWEEN runs (`systemctl --user restart buildkit`), never while a build solves. Do not delete this file. 3. **sccache is the C/C++ compiler cache; ccache is its FALLBACK. Keep both wired.** Owner directive 2026-08-26 (c42091e), REVERSING the earlier "full switch rejected (2026-08-17)". This is no longer a per-language split: every C/C++ launcher resolves at RUNTIME through `compiler_cache_launcher()` (`01-core/common.sh`), which returns sccache when its server answers, else ccache, else fails so the caller builds uncached. Call sites: build-gcc.sh (CC/CXX prefix), build-clang.sh and llvm-cross.sh (`CMAKE_*_COMPILER_LAUNCHER`), cmake-cache-linker.sh, the onnxruntime build lib, and build-app-wheelhouse.sh (IREE). Since 2026-08-30 (backlog F2) `compiler-cache.sh` routes through the SAME resolver: `_resolve_compiler_cache_launcher()` calls `compiler_cache_launcher()` when 01-core is loaded (every media/ORT caller) and inlines the identical decision only for the android preamble, which sources compiler-cache.sh standalone. Both `setup_ccache` and `setup_sccache` consume it; the agreement is pinned by `tests/test-compiler-cache.sh`. New cache logic belongs in the resolver, not in another duplicate. - Do NOT hardcode `ccache` as a launcher anywhere. `cmake-cache-linker.sh` is SHARED; a literal there silently overrides the decision for every consumer. - `--ccache` on build-gcc.sh/build-clang.sh is a historical FLAG NAME. It means "use the compiler cache", not "use ccache". llvm.sh passes it. - Both mounts stay on every heavy RUN (ccache AND sccache), because the fallback needs somewhere to persist. - **Rust IS cached again (2026-08-27, 4200f7b + 54fc1df) — through the guarded launcher.** The 2026-08-20 "Rust stays UNCACHED" rule was earned by the sccache SERVER dying mid-compile in three media rounds, killing green builds at 99% — but that signature was the wrong-server-by-fixed-TCP-port bug, cured by `SCCACHE_SERVER_UDS` (2359 media-stage sccache faults → 0). Two places set the wrapper, in this order: `setup_sccache` (compiler-cache.sh:172-210), which setup-gstreamer.sh:50 runs unconditionally for the Rust-heavy gstreamer lane, and build-gstreamer-monorepo.sh's launcher-preferring block, which only fires when `RUSTC_WRAPPER` is still UNSET. Both PREFER `01-core/sccache-launcher.sh`, so an sccache hiccup costs cache hits, not a build at 99%. Since 26a30740 (2026-08-27, owner decision "immer sccache") their FALLBACKS AGREE: with no executable launcher on disk both ship BARE sccache — build-gstreamer-monorepo.sh's `for _rw in …sccache-launcher.sh` loop (`export RUSTC_WRAPPER="${RUSTC_WRAPPER:-sccache}"`) and `setup_sccache`'s `_sc_launcher="sccache"` default (compiler-cache.sh:193). Never uncached; the launcher is an upgrade, not a precondition. The launcher is only reachable because 01-core is bind-mounted at `/opt/scripts/core` on every heavy media RUN; keep it on those mount lists. Exporting `RUSTC_WRAPPER=""` is the opt-out — `Dockerfile.toolchain:66` and `Dockerfile.package:262` do exactly that. nvcc stays untouched — the Windows lane records that released sccache breaks around it. - sccache-specific knobs live in `/etc/sccache/config.toml` (baked in `Dockerfile.base`, reached via `SCCACHE_CONF`), because `CCACHE_SLOPPINESS` and preprocessor/direct mode have NO env-var path in sccache. They take no effect while `SCCACHE_DIR` is set, which the chain always does ([§ The shipped image's cache dirs](#the-shipped-images-cache-dirs)). The size cap is `SCCACHE_CACHE_SIZE`; there is no `sccache -M` to call. - **PREFER `01-core/sccache-launcher.sh`; fall back to bare `sccache` rather than to nothing.** Superseded 2026-08-27 (`26a30740`): this rule used to read "NEVER point a launcher at bare `sccache`", and taken literally it tells you to delete the default at `compiler-cache.sh:193` (`_sc_launcher="sccache"`, upgraded to the launcher when one is on disk) — which would turn `verify-critical-fixes.sh` RED, because that gate checks the DECISION (never UNCACHED), not the spelling. Always cache; use the launcher when it is available. The launcher still matters: sccache ABORTS the compile on its own internal errors where ccache would just exec the compiler, and that is not theoretical: it killed the media stage three times. The root cause — CMake creates a TryCompile scratch dir, compiles in it, DELETES it, and sccache then spawns the compiler with that dir as cwd (ENOENT) — is written up ONCE, in [build-cache-tiers.md](build-cache-tiers.md) § 5.1, with the measurements. Read it there. The launcher runs sccache for every compile and only bypasses on "sccache: encountered fatal error"; a REAL compile error is passed through untouched, because blindly retrying would hide genuine failures. - **Preprocessor cache mode stays OFF** (`SCCACHE_DIRECT=false`, set in ensure_sccache_env and compiler-cache.sh; it overrides the `use_preprocessor_cache_mode = true` that Dockerfile.base's config.toml still bakes, § 5.1). We turned it on to recover ccache's direct-mode hit rate; it re-reads the input file AFTER the compile to store the entry and therefore dies on the same deleted scratch dirs. It is off by default upstream. - **Resolve the launcher through `compiler_cache_launcher()`.** If you add a new cache call site, route it through the helper. The duplicated resolution in compiler-cache.sh is why this class shipped INERT twice: `setup_ccache` hardcoded the string `sccache`, so the guard had no effect on the media lane for three runs (fixed c5b17ce); then `setup_sccache` exported `RUSTC_WRAPPER="sccache"` unconditionally, and because setup-gstreamer.sh:50 runs it BEFORE build-gstreamer-monorepo.sh's `[ -z "${RUSTC_WRAPPER+x}" ]` test, every gst-plugins-rs crate went through bare sccache (measured on the live media lane 2026-08-27; fixed 54fc1df). Because the class shipped inert twice, `verify-critical-fixes.sh` now GATES compiler-cache.sh against `export RUSTC_WRAPPER|CMAKE_C{,XX}_COMPILER_LAUNCHER="sccache"`. The failure mode this replaced (mount without wiring, wiring without mount) was invisible — builds stayed green, just slow. Before committing a multi-hour run to a change here, run `bash linux/scripts/02-toolchain/probe-sccache.sh` INSIDE the compiler image: it costs seconds and asserts, per compiler shape this chain actually feeds a launcher, both that the compile survives AND that sccache recorded cache activity. sccache HARD-FAILS on a compiler it cannot identify where ccache would simply exec it, so "it compiles" is not the whole question. 4. **Never edit a running orchestrator's main script** (`build-cross-chain.sh` while a chain runs): bash reads it incrementally by byte offset; an edit can corrupt the in-flight process. Sourced library files are safe to edit for FUTURE runs (the running process holds them in memory) but see rule 1. ## 6. Knob reference | Knob | Default | Effect | |---|---|---| | `NO_CACHE=1` | unset | `--no-cache`; also disables T2 and T3 entirely | | `RUNTIME_NO_CACHE=1` | unset | `--no-cache` on just the runtime package + wrapper builds (and the `export` build below) | | `RUNTIME_WHEELS_SOURCE` | `auto` (= `image`) | `export`: the wrapper mounts a small directory staged from the android image before the package build, so it no longer needs the ~12.7 GB android snapshot in the BuildKit store; a prune after the package build then costs nothing. Only the torch RUN's key changes; the lib is outside every closure ([both paths](linux-cross-builds.md#the-wrappers-wheelhouse-two-deliveries)) | | `NO_CACHE_EXPORT=1` | unset | drops everything registry-facing (T3, and T4 when enabled); keeps T2 | | `CROSS_NO_LOCAL_CACHE_EXPORT=1` | unset (set by the disk guard) | stop *writing* T2; still read it | | `SALVAGE_CACHE_EXPORT=0` | `1` | disable the per-`--target` salvage re-drive after a failed stage | | `SALVAGE_TARGET_TIMEOUT` | `600` | per-target timeout for that salvage pass | | `CROSS_DISK_GUARD_GB` | `40` | free-space floor that triggers T2 LRU pruning (`0` disables) | | `CROSS_CACHE_MAX_GB` | `250` | total T2 size cap (`0` disables) | | `CROSS_PREFLIGHT_TRIM=0` | unset (trim on) | skip the disk-preflight trim of T2 (§ 3.1) | | `CROSS_RUNTIME_LANE_GB` | `120` | free GB the runtime lane needs per *concurrent* wrapper build; `0` disables the lane-entry gate (§ 3.2) | | `CROSS_DISK_WATCH=0` | unset (watch on) | disable the in-stage disk watchdog (§ 3.2) | | `CROSS_DISK_WATCH_SECS` | `120` | in-stage sampling interval | | `CROSS_TRIM_KEEP_SLUGS` | `3` | newest T2 slugs the trim will never remove | | `CROSS_BUILDKIT_PRUNE=0` | unset (fallback on) | disable the guard's filtered buildkit-store prune (§ 3.2.1) | | `CROSS_BUILDKIT_KEEP_GB` | `120` | GB of `type==regular` layer cache that prune keeps; a non-zero value under 100 is raised to 100, `0` reclaims all of it (§ 3.2.1) | | `CROSS_IMAGE_PRUNE=0` | unset (lever on) | disable the between-stage image-store lever (§ 3.2.2) | | `SALVAGE_MIN_FREE_GB` | `CROSS_DISK_GUARD_GB` (40) | free-space floor below which the S1 salvage is skipped (`0` = always salvage) | | `BUILDKIT_CACHE_DIR` | `~/.cache/kata-buildcache` | where T2 lives | | `PUSH_MAX_ATTEMPTS` / `PUSH_RETRY_BASE_SECS` | `4` / `15` | transient-push retry budget | | `ENABLE_SCCACHE_RUST` | `0` | **not** the monorepo's Rust switch any more — it only adds `setup_sccache` to `media_common_init`, i.e. it caches `install-rice-proto.sh`'s `cargo cinstall` (§ 5.3 item 1) | | `ENABLE_SCCACHE_CUDA` | `0` | sccache as the CUDA/HIP compiler launcher (§ 5.4) | | `USE_CCACHE` / `USE_SCCACHE` / `USE_LLD` | `true` | per-tool switches in `compiler-cache.sh`; note § 5.3 item 3 | | `KATAGLYPHIS_LINKER` | unset (`default`) | `lld` or `mold` for a consumer build: `LDFLAGS` plus the host Rust target, fails rather than falls back ([`linker-select.sh`](shared-script-libraries.md#linker-selectsh--an-opt-in-linker)) | | `WEB_LANE_TOOLS_CACHE` | `on` | `refresh` rebuilds past the web-lane tool cache and re-stores; `off` bypasses it (§ 1.2) | | `HAILO_NESTED_CACHE` | `carry` | `off` builds the protobuf that HailoRT's configure compiles under `env -i` without the cache, as before 2026-09-24 ([two switches](hailo-support.md#the-nested-build-cache-and-pyhailort-two-switches)) | `CROSS_REGISTRY_CACHE` is deliberately **not** in this table. The T4 pilot knob was written and reverted, nothing in `cross-stage-build.sh` reads it, and listing it beside live knobs is how an operator ends up setting a no-op. § 4.5 keeps the design. --- ## 7. Do-not-repeat list - Do not add a `--cache-from` whose matching `--cache-to` is gated off. That self-defeating pair made every stage rebuild from scratch until `195285f`. - Do not add a cache tier that the DeadlineExceeded auto-recovery cannot strip. - Do not "align" the ccache/sccache cache-mount ids with the apt ones (`Dockerfile.media:26-31`). - Do not reclaim buildkit-store disk with `nerdctl builder prune -f` or `nerdctl system prune`, from a script or by hand. Neither has a `--filter`, so both delete the `exec.cachemount` records with the layer cache: 4.9 GB of compile cache thrown away to reclaim 207 GB of regenerable layers on 2026-08-17, and both target-LLVM builds then ran cold. The filtered form is `buildctl prune --filter type==regular` (§ 3.2.1, `prune-safe.sh`). - Do not size the runtime-lane disk gate per arch. It is per *concurrent* build (§ 3.2); the per-arch number refuses every run on this host. - Do not judge a cache change by whether the flags appear in the command. Judge it by bytes that did not move: cache hits on **intermediate** vertices, or a measured wall-clock delta. Two changes were reverted on 2026-08-23 for looking right and doing nothing. ## sccache failure classes the launcher must bypass `01-core/sccache-launcher.sh` decides whether an sccache failure costs cache hits or the whole build. Two classes must both bypass to a direct compiler run: - `sccache: encountered fatal error` — the CMake `TryCompile` ENOENT class. - `sccache: error: failed to execute compile` with `caused by: Failed to send data to or receive data from server` — the sccache **server** died mid-build. Observed live on the first F2 media validation build (2026-08-30); the launcher did not classify it as sccache-internal, handed it to ninja as a real compiler failure, and killed the TVM step. A real compiler error must pass through untouched (the compiler is never re-run), and a clean compile passes through. `test-sccache-launcher.sh` pins every case. **A consumer's own CMake gets it too (2026-10-07).** `cmake/Cache.cmake`'s `myproject_enable_cache` used to set both compiler launchers to the bare `sccache` it found. BeschleunigerBallett run 37509585546 then failed every compile of one job with `Connection to server timed out ... Connection refused`. Off Windows, with `COMPILER_CACHE=sccache`, it now sets the guarded launcher beside it in the hub (`linux/scripts/01-core/sccache-launcher.sh`), which runs sccache from `PATH`. ccache and Windows keep their bare binary, and `test-cmake-cache-launcher.sh` pins all three. ### What was measured about the ENOENT class From the 2026-09-01 run, 3062 bypasses: * **2952x** `sccache: encountered fatal error` + `failed to spawn … No such file or directory (os error 2)`, on sccache's OWN `-E` preprocessor pass. It is **intermittent** (~10–40% of compiles) and appears **in the heavily parallel steps only**. The compiler path is absolute, the cwd is a **live build dir**, and the direct fallback of that same argv **succeeds right after** — so neither a missing compiler nor sccache's scrubbed env explains it. * **110x** `sccache: error: failed to execute compile` + `Failed to send data to or receive data from server` — the sccache server died mid-build. Root cause **FOUND 2026-09-05**, and it is not inside sccache's spawn at all: the client was addressing another container's server. See [the server address section](#the-server-address-must-be-exported-where-the-compiles-run). ### The single retry (2026-09-03) Because the direct fallback of the same argv succeeds immediately afterwards, the launcher **retries once** before giving up the cache entry. A bypass was never a correctness problem — it compiles fine, it just discards the cache, and that is the entire cost of this bug. The retry is bounded: a second sccache-internal failure falls through to the direct compile exactly as before, and a real compiler error surfacing on the retry is handed back untouched rather than re-run. #### The retry's own measurement — ANSWERED, and the answer is negative The retry doubles as the experiment: `retry succeeded (cache kept)` versus `failed twice` in a build log says directly whether the class is transient. That count was taken on the 2026-09-03 media-arm64 build and the question is **answered — do not re-run it**: | outcome | count | |---|---| | `retry succeeded (cache kept)` | **27** | | `failed twice` | **514** | **The retry stays** — it costs nothing and 27 recovered entries are 27 recovered entries — but it is a mitigation, not the fix. **Read that table by STEP, not as one rate (2026-09-05).** Splitting the same log by BuildKit vertex dissolves the "5%": every one of the 514 sits in `#24` (litert, 364), `#34` (ORT genai, 100) and `#45` (pyav, 50), and all 27 recoveries sit in `#29` (TVM) — which has no failures of the other kind at all. Four containers, each all-or-nothing, not one population recovering 5% of the time. A per-request property inside one server cannot produce that segregation; a per-CONTAINER one does, which is what the next section shows it was. ### The server address must be exported where the compiles run **This is the root cause of both classes above, and it is a regression of a bug this document already carried as fixed (§ 5.4).** `SCCACHE_SERVER_UDS` makes the server address container-local. `4aa92fb6` (2026-08-27) exported it from the body of `setup_ccache`, i.e. in the shell that then ran cmake and ninja, and the media stage went from 2359 faults to zero. The F2 one-resolver refactor (`8c97cdd8`, 2026-08-30) moved that block, unchanged, into `_resolve_compiler_cache_launcher` — a function every caller invokes as `$(_resolve_compiler_cache_launcher)`. **A subshell cannot export to its parent.** From that commit on, the launcher name reached the build and the server address did not, so every sccache client fell back to the default TCP port 4226, which under rootless buildkitd is reachable from every concurrently running step. The same shape predates F2 on the other resolver: `compiler_cache_launcher()` (`common.sh`) is called as `CC="$(compiler_cache_launcher) gcc"` at every one of its ~15 sites, so `ensure_sccache_env`'s exports have never reached a compile there either. Reproduced statically, no build needed: source `compiler-cache.sh`, run `setup_ccache` with a fake sccache on `PATH`, and read the caller's environment — `CMAKE_C_COMPILER_LAUNCHER` is set, `SCCACHE_SERVER_UDS` is not. **The evidence in the run logs is arithmetic, not inference.** `sccache --show-stats` reports the counters of whichever server answered: | log | step | at | Compile requests reported | |---|---|---|---| | `sccache-retry-20260903-150440` | `#71` libcamera | **0.357 s** into the step | **1264** (1230 hits) | | `sccache-retry-20260903-150440` | `#34` ORT genai | 0.382 s / 0.634 s | 90 → 131 … then **97** at 199.5 s | | `rvv-media-20260902-025642` | `#73` | 0.226 s / 0.372 s | 65 → **544** (+479 in 0.15 s) | | `qnn-media-arm64-20260903-125457` | `#34` | 0.305 s | **423** | A step cannot have made 1264 compile requests in the third of a second it has been alive, and a single server's request counter cannot go 131 → 97. Both are only possible if the client is talking to a server in another container — and in `#34`'s case to two different ones inside one step. Three independent runs, one signature. The cost is larger than the bypass counts suggest, because a compile the foreign server *does* serve is stored in the foreign container's cache mount: in that same log `#24` (litert, `id=sccache-arm64`) and `#29` (TVM, `id=sccache-amd64`) were running concurrently, mounting different cache directories at `/var/cache/sccache` while addressing one port. That also explains the retry's per-step segregation: the foreign server spawns the compiler by the path the request carries, that path exists only in the caller's container, and it stays missing for as long as that server owns the port — so an immediate retry changes nothing, while the direct fallback runs in the caller's own container and always works. It explains the second class too — "Failed to send data to or receive data from server" is what a client sees when the container that owned the server finishes its step. **The fix** puts the address back where it has to be: `sccache_export_server_address` in `01-core/compiler-cache.sh` is the one owner of the address, and `setup_ccache` and `setup_sccache` call it **before** the substitution, so cmake, ninja, every launcher process and `--show-stats` all address the same container-local socket. Pinned by `test-compiler-cache.sh`; the resolver's bootstrap arm calls the same function instead of repeating it. **The other resolver has its own owner, for the same reason.** `compiler_cache_launcher_env` in `common.sh` runs `ensure_sccache_env >&2 || true` and is called by all 12 `$(compiler_cache_launcher)` sites — `02-toolchain`'s GCC/clang/llvm-cross, opencv, `media_compiler_launcher` (ffmpeg and pyav), onnxruntime (lib + AMD + NVIDIA), tvm, `cmake-cache-linker` and the wheelhouse's two. It is a one-line wrapper on purpose: it is where the RULE is named, it is what a mutation can pin, and it keeps the `>&2` discipline that the 2026-08-26 `CC="[INFO] …sccache gcc"` disaster taught in one place rather than twelve. It is TOTAL (`|| true`) because it runs in the caller's shell ahead of the substitution, where a host with no usable sccache must not kill a build that would compile uncached perfectly well, and it honours `USE_SCCACHE=0` before anything starts a server. `test-compiler-cache-launcher.sh` pins the parent-shell propagation, the totality, the switch, the empty stdout, AND that every resolution site in the closure is preceded by the call — that last one is a drift pin, because one owner is worth nothing if a site forgets it. **`Dockerfile.toolchain` now mounts `sccache-launcher.sh` beside `common.sh`** in both per-file `01-core` blocks. Before that the GCC/LLVM stages resolved to **bare** sccache — no guarded launcher — so an sccache fault there was a build abort rather than a lost cache entry. The suite asserts the two mount counts are equal, so a future block that mounts one without the other fails. **One duplication is left and it is blocked on a Dockerfile, not on judgement.** `ensure_sccache_env` still carries its own copy of the address block. It is not dead — it is what makes the server exist for the subshell's own `--start-server` — but it is a second owner, and collapsing it means `common.sh` sourcing `compiler-cache.sh`, which `Dockerfile.base`'s six per-file blocks do not mount. Measured cost of leaving it: 28 shingles, inside its recorded budget. **How the next chain proves it.** The launcher now prints the address it used on both of its lines (`[server=/tmp/sccache-0.sock]` versus `[server=tcp:4226]`). `tcp:4226` in any bypass line is this bug, still live, in whichever step printed it; a socket path in every line is the fix holding. Count the two message classes per step, not over the whole log. Reading these counts out of a log needs one caution: a **cached** BuildKit step replays its old output verbatim, so a pre-retry launcher's messages can reappear in a post-retry build (496 of them from one cached step, `#30`, on the first read of that same build) and look like the new launcher failing to take effect. Check which step the lines came from before believing them. A second sample needs a compile-heavy lane — `--only runtime` produces no `sccache-launcher` lines at all. --- ## The shipped image's cache dirs `:latest-cross` shipped `CCACHE_DIR=/workspace/.ccache` and `SCCACHE_DIR=/workspace/.sccache` until 2026-09-04. `/workspace` is the **consumer's** bind-mounted checkout (`Dockerfile.torch` makes it the WORKDIR and a VOLUME), so every compile inside the image wrote a cache into someone else's repository: it pollutes their working tree, can be swept into CI artifacts, and on a non-ext4 host mount `flatpak-builder` aborts outright with `Can't initialize ccache use: Failed to set permissions of /workspace/.ccache/disabled/ccache.conf: Operation not permitted`. Reported by the OmniAccelerANT lane and reproduced as uid 1001. **Where the value came from.** `eff8deac` (2026-06-01) added `CCACHE_DIR` next to `CCACHE_SECONDARY_STORAGE=true`, i.e. as one half of a "cache travels with the checkout" idea. The other half was wrong and was deleted in `025bf68c` — `CCACHE_SECONDARY_STORAGE` is a URL and set to `true` it failed *every* ccache compile — but the path it was invented for stayed. `d5bafe86` (2026-08-26) then added `SCCACHE_DIR` deliberately alongside it so a consumer would not have a ccache dir and no sccache dir; the `/workspace` prefix was inherited from the line above, not chosen. **Does any stage need a per-workspace cache? No.** Nothing else in the Linux tree names `/workspace/.ccache`; the build stages get their cache dirs from `Dockerfile.base`'s ENV, from `llvm-cross.sh`'s explicit exports, and from `--mount=type=cache,target=/var/cache/ccache` — none of which reads the package stage's runtime ENV. The only instructions after that ENV block are symlink wiring, the apt mirror rewrite, `setup-package-image.sh` and `validate-compilers.sh`, and none of them runs a compiler through a cache launcher. So the value was runtime-only, and repointing it cannot change what a build does. A consumer who *wants* the cache next to their checkout still sets `-e CCACHE_DIR=...` themselves; a consumer who wants it to persist across runs mounts a volume at `/var/cache/ccache`, which is what that directory is for. **Why the ENV cannot simply be deleted.** `Dockerfile.base` already declares the same defaults, but its image config does not reach the shipped image: the runtime package stage is built on a rootfs/OCI-layout export of its parent (`context-management.sh`), which carries the filesystem and drops the parent's `Config.Env`. Inspecting the shipped `latest-cross-amd64` shows exactly `Dockerfile.package`'s own ENV blocks plus `Dockerfile.torch`'s — base's `CCACHE_MAXSIZE`, `SCCACHE_CACHE_SIZE`, `SCCACHE_CONF`, `SCCACHE_IDLE_TIMEOUT` and `SCCACHE_ERROR_LOG` are all absent from it, while base's *filesystem* effects (the 1777 cache dirs) are present. The runtime image must therefore declare the paths itself. **One owner.** `01-core/compiler-cache.sh` owns the two paths as its `:=` defaults; `Dockerfile.package`'s ENV repeats them because a Dockerfile cannot source shell, and `test-compiler-cache.sh` sources the library with both vars unset and asserts the ENV equals what it produces — plus a second assertion that neither value sits under `/workspace`. Drift on either side fails the suite. **The caps and server knobs (2026-09-28).** `Dockerfile.package` now also repeats base's `CCACHE_MAXSIZE=30G`, `SCCACHE_CACHE_SIZE=30G`, `SCCACHE_IDLE_TIMEOUT=0` and `SCCACHE_ERROR_LOG=/tmp/sccache.log`. Until then the published `:latest` (v0.0.28) carried only the two dirs, so every consumer's sccache ran with its 10 GiB default. `test-compiler-cache.sh` asserts the four equal base's. `SCCACHE_CONF` stays out on purpose: it changes nothing while `SCCACHE_DIR` is set (see below). **What the shipped image already provides.** `/var/cache/ccache` and `/var/cache/sccache` are `drwxrwxrwt` (`Dockerfile.base`, 1777 since `35f83d62`) and uid 1001 writes to both — verified by running the shipped image. A missing cache directory is not an error either: `ccache` creates `CCACHE_DIR` on first use, which is how a consumer's `-e CCACHE_DIR=/some/mount` keeps working. Layer cost: none. This is an ENV change, so nothing is copied up and no file metadata is rewritten. **One build in this image's lineage does compile through the launcher** (since the Hailo fold-in, 2026-09-20): the Hailo `RUN` in `Dockerfile.torch` builds HailoRT, pyhailort, libzmq and TAPPAS on the `ccache-` and `sccache-` cache mounts. The paragraph above says nothing after the package ENV does; that stopped being true there. Base's caps do not reach that `RUN` either, for the same rootfs-handoff reason, so `compiler-cache.sh`'s 10G defaults applied. An sccache server trims its directory to its own cap on its first compile (the design probe of 2026-09-23 saw an 8 MiB cap cut a 41 MiB directory to 8 MiB), and `ccache -M 10G` rewrote the shared `ccache.conf`. The `RUN` mounts `sccache-${TARGETARCH}` and `ccache-${TARGETARCH}`, the ids every cross stage mounts for the arch `CROSS_BUILD_PLATFORM` names. On that arch the Hailo `RUN` shares the chain's own cache and could shrink it: amd64 on the cross host, arm64 on the Jetson (`CROSS_BUILD_PLATFORM=linux/arm64`). Only an arch the runtime lane builds under QEMU, such as arm64 on the cross host, has mounts no cross stage uses. riscv64 skips the Hailo build and never writes its mounts. `SCCACHE_CONF` does not set the cap, nor any other knob in the file: as soon as `SCCACHE_DIR`, `SCCACHE_CACHE_SIZE`, `SCCACHE_DIRECT` or `SCCACHE_LOCAL_RW_MODE` is set, sccache builds its disk config from the environment, and that replaces the file's whole `[cache.disk]` section (sccache `src/config.rs`, `config_from_env` and `CacheConfigs::merge`). Only `SCCACHE_CACHE_SIZE` sets the cap. Measured on the published `:latest` on 2026-09-28 with `sccache --show-stats`: | Environment | Max cache size | | --- | --- | | `SCCACHE_DIR` | 10 GiB | | `SCCACHE_DIR` + `SCCACHE_CONF` | 10 GiB | | `SCCACHE_CONF` alone | 30 GiB (the file) | | `SCCACHE_DIR` + `SCCACHE_CACHE_SIZE=30G` | 30 GiB | Base's ENV and `compiler-cache.sh` always set `SCCACHE_DIR`, so the baked `file_stat_matches = true` and `hash_working_directory = false` have never applied. What applies is sccache's env default: preprocessor-cache mode **on**, with `file_stat_matches = false` and `hash_working_directory = true`. The chain's entry points turn the mode off with `SCCACHE_DIRECT=false`; the shipped image sets no `SCCACHE_DIRECT`, so a consumer runs with the mode on. Since 2026-09-24 the Hailo `RUN` passes `SCCACHE_CACHE_SIZE=30G` and `CCACHE_MAXSIZE=30G`, and `tests/test-hailo-build.sh` pins them to `Dockerfile.base`; its `[CACHE] hailo/` lines print `cap=30 GiB`. `compiler-cache.sh` keeps its 10G defaults: changing them is an `01-core` edit, which re-keys the chain from the compiler stage. ## Keeping the compiler cache across CI runs A consumer lane starts every step in a fresh `--rm` container, so the image's `/var/cache/{sccache,ccache}` die with it and every CI run compiles cold. The [`compiler-cache-restore` / `compiler-cache-save`](../.github/actions/README.md#compiler-cache-restore--compiler-cache-save) actions mount a runner directory at those paths and carry it between runs through `actions/cache`. **Measured** in AccelerANTgine's linux-x64 lane on 2026-09-29, each warm run beside a control run of the same commit at the same time (3 control, 2 warm, 1 cold): | clang job | no cache | warm cache | | --- | --- | --- | | whole job | 25.1 min (20.3-29.4) | 14.4 min (14.1-14.7) | | the four `cmake --build` | 14.7 min | 3.9 min | The cache was 405 MB and restored in about 7 s. A cold run, which is what the first run after an eviction gets, took 32.6 min. What a warm run cannot shorten is the host prologue: the image pull stayed at about 8 min. **What caches.** clang's C++20 modules cache with the stock sccache on Linux: the interface units store their BMI as an output and the importers hash every BMI they import, so an interface edit misses exactly its importers (verified on BeschleunigerBallett and AccelerANTgine). GCC's `-fmodules-ts` translation units do not cache with sccache 0.18.0 or ccache 4.12.3, so a GCC lane gains only its non-module TUs. **The quota is 10 GB per repository**, shared with every other cache (CodeQL, setup-uv). An `actions/cache` entry cannot be overwritten, so every save gets a per-run key and restore takes the newest. Since 2026-09-29, `compiler-cache-save` also deletes the key's entries from older runs on the same ref before it saves, so each key holds one entry. Before that, the superseded entries stayed in the quota until GitHub evicted them. Its eviction removes the least recently used entry of the whole repository, not of the key. BeschleunigerBallett reached 11.2 GB and OxidANT 12.4 GB. Ballett's TSan job then restored nothing: its only entry had been evicted, while two stale x64 duplicates, read more recently, survived. A scheduled clean-up job would come too late for this, because GitHub evicts at the moment a save passes the quota. The deletion needs `permissions: actions: write` in the saving job. Without it the action warns and saves anyway. Budget a repository so one generation of all its lanes, plus one entry being saved, fits inside the quota: - one key per preset that is worth it; - each capped with the `env` input (sccache keeps its most recent entries under the cap). `enabled: auto` saves on pushes to the default branch only (since 2026-09-29). The deletion works per branch, so a release push to `main` would otherwise have stored a second generation of every key: about 7.3 GB more in BeschleunigerBallett. Every other branch, and every pull request and dispatch, restores the default branch's entry, which GitHub lets any branch read. **Windows container lanes** take `container-ci-windows.yml`'s `compiler-cache-key` input (2026-09-29). An `SCCACHE_DIR` on a mounted volume stored nothing on the dev host ([§ sccache's cache directory on a Windows container volume](windows-container-build-performance.md)), so the cache is not mounted for sccache to write into. `actions/cache` restores it into the workspace as `.ci-cache`, the container gets `CI_COMPILER_CACHE=C:\ws\.ci-cache`, and the build script moves it into container-local directories before the build and back after it (OxidANT's `Invoke-WindowsLane.ps1`). Pushes to the default branch save it. That reusable workflow saves with `actions/cache/save` and deletes nothing. Its job grants only `contents: read`, and a reusable workflow cannot ask for more than every caller grants. A caller that sets `compiler-cache-key` therefore runs `compiler-cache-prune` from a later job that grants `actions: write` (OxidANT's `windows-x64.yml`). A clang-cl C++20 module build must not use it: the released sccache serves stale objects for module importers until mozilla/sccache#2876 ships. **A cap below the working set is worse than no cache.** BeschleunigerBallett's release and profile caches filled their first caps (489/500 MB, 981/1000 MB): sccache evicted entries during the cold build, and the warm release build took 782 s against 533 s without a cache, paying the hashing for no hits. With room (1.1 GB each) the same builds took 257 s and 349 s. Size a cap from a cold run's saved size, with headroom. ## The shipped image carries no build-host setting **A published image's environment names nothing outside the container** (both lanes, 2026-09-23). The Windows image shipped the owner's LAN WebDAV as `SCCACHE_WEBDAV_ENDPOINT`, and every consumer's sccache 0.18 server died on its storage check — the full account, the Windows split of ARG versus ENV and the consumer-side probe are in [`windows-build-resources.md` § What the published image carries](windows-build-resources.md#what-the-published-image-carries). The Linux lane never had the leak: its cache is BuildKit cache mounts, with no remote tier (§ 5.4, the cross-machine tier), and the mirror URLs (`FAST_UBUNTU_MIRROR_URL`, …) are ARGs. The runtime defaults above (`CCACHE_DIR`, `CCACHE_MAXSIZE`, `SCCACHE_DIR`, `SCCACHE_CACHE_SIZE`, `SCCACHE_IDLE_TIMEOUT`, `SCCACHE_ERROR_LOG`) are container-local and stay. What keeps a leak from ever reaching `:latest`: - **The static pass**, `linux/scripts/verify_image_env.py --dockerfile`, runs in `lint-dockerfiles.sh` (preflight slug `dockerfile-lint`) over both lanes' Dockerfiles. It refuses an ENV of an sccache remote-backend or layout name (`SCCACHE_WEBDAV_*`, `SCCACHE_REDIS*`, `SCCACHE_BUCKET`, `SCCACHE_MULTILEVEL_CHAIN`, `SCCACHE_FORCE_LOCAL`, …), an ENV that expands one under another name, and an RFC1918 or link-local literal in an ENV or an ARG default. On Windows it also refuses a compiling RUN whose stage never declares `ARG SCCACHE_WEBDAV_ENDPOINT`. - **`_manifest_image_env_gate`**, the first step of `create_manifest` in `build-runtime-manifest.sh`, reads each wrapper's config ENV with `nerdctl image inspect --platform linux/ --format '{{range .Config.Env}}…'` and runs the same script with `--env-file`. An unreadable config fails it too, and a tag missing locally is pulled first (only then, like the content gate). It has no switch: it runs on every path that creates an index, `--manifest-only`/`--repair` included, and neither `RUNTIME_IMAGE_SMOKE=0`, `WRAPPER_CONTENT_GATE=0` nor `--force` waives it. Until 2026-09-24 it was check 6 of `verify-shipped-wrapper.sh`, which sits behind `RUNTIME_IMAGE_SMOKE` and never runs on `--manifest-only`. It does not cover the per-arch wrapper tags: those are pushed by the build loop before any runtime gate, as they always were. The address rule, the version exemption and what neither pass covers (hostnames, loopback, files, image history) are the same on both lanes, because one case file, `linux/scripts/tests/image-env-cases.json`, grades this script and the Windows PowerShell twin alike. Tests: `linux/scripts/tests/test-image-env.sh`; mutations `dockerfile-lint.image-env-*` and `image-env.*`.