# Common failure modes — symptom lookup **Start from the error message.** Every entry below is keyed by what you actually see, then names the cause and the fix. All of them were hit live in this repo; the dates are when. Two neighbours, so you land on the right page: - **A rule you must not break** while editing the Windows chain → [`windows-build-invariants.md`](windows-build-invariants.md). - **A host that was never set up correctly** (rather than one that broke) → [`windows-host-setup.md`](windows-host-setup.md) for Windows, [`linux-host-setup.md`](linux-host-setup.md) for Linux. > Several rows here point at *deeper* write-ups in > [`windows-builds.md`](windows-builds.md) and its sibling pages. This page is > the triage index: enough to recognise the failure and act, with a link when > the full history matters. ## Contents **Linux and cross-lane** - [`exec format error` on a foreign-arch build](#exec-format-error-on-a-foreign-arch-build) - [`no space left on device`](#no-space-left-on-device) - [Stale downstream images](#stale-downstream-images) - [`no active session` / `grpc: the client connection is closing` mid-chain](#no-active-session--grpc-the-client-connection-is-closing-mid-chain) - [`registry_pin_ref` fails on a fresh push](#registry_pin_ref-fails-on-a-fresh-push) - [Terminal freeze during a long build](#terminal-freeze-during-a-long-build) - [LiteRT configure: `fatal: expected flush after ref listing`](#litert-configure-fatal-expected-flush-after-ref-listing) - [`logging.sh: line NNN: action: unbound variable` instead of the real error](#loggingsh-line-nnn-action-unbound-variable-instead-of-the-real-error) - [A fault-injection test passes and proves nothing (`grep` is ugrep)](#a-fault-injection-test-passes-and-proves-nothing-grep-is-ugrep) - [`GENAI-BIND SKIP` reported green while the native binding is broken](#genai-bind-skip-reported-green-while-the-native-binding-is-broken) - [The documented `GENAI_ALLOW_RISCV64` back-out does not reach the smoke](#the-documented-genai_allow_riscv64-back-out-does-not-reach-the-smoke) - [An unresolved `NEEDED` in a library that nothing scans](#an-unresolved-needed-in-a-library-that-nothing-scans) - [A prune step deletes the wheel a later step requires](#a-prune-step-deletes-the-wheel-a-later-step-requires) - [The disk guard aims at the wrong number](#the-disk-guard-aims-at-the-wrong-number) - [A renamed or dropped distro package kills a stage hours in](#a-renamed-or-dropped-distro-package-kills-a-stage-hours-in) - [apt: `libc6:i386` install is unsatisfiable after an archive/ports drift](#apt-libc6i386-install-is-unsatisfiable-after-an-archiveports-drift) - [A from-source CPython silently drops an extension module](#a-from-source-cpython-silently-drops-an-extension-module) - [The delete guard denies its own legitimate work](#the-delete-guard-denies-its-own-legitimate-work) - [OpenCV: `std::complex` breaks on a shadowed `complex.h`](#opencv-stdcomplex-breaks-on-a-shadowed-complexh) - [A smoke that never passed and always excused itself](#a-smoke-that-never-passed-and-always-excused-itself) - [A trailing conditional fails the whole script](#a-trailing-conditional-fails-the-whole-script) - [The copied Rust toolchain is the builder's arch](#the-copied-rust-toolchain-is-the-builders-arch) - [A packaging script dies with no message](#a-packaging-script-dies-with-no-message) - [A callee invoked in an `if !` condition runs with errexit off](#a-callee-invoked-in-an-if--condition-runs-with-errexit-off) - [A checksum probe that cannot reach the server reads as "nothing to verify"](#a-checksum-probe-that-cannot-reach-the-server-reads-as-nothing-to-verify) - [A soname with no map entry is resolved by an `apt-cache` prefix guess](#a-soname-with-no-map-entry-is-resolved-by-an-apt-cache-prefix-guess) - [The NVIDIA apt keyring 404s on `ubunturesolute`](#the-nvidia-apt-keyring-404s-on-ubunturesolute) - [apt `gpgv` exits 111 after a `COPY --link` into `/tmp`](#apt-gpgv-exits-111-after-a-copy---link-into-tmp) - [nvcc rejects the image's GCC 16](#nvcc-rejects-the-images-gcc-16) - [A CUDA compile is `Killed` though average memory looked fine](#a-cuda-compile-is-killed-though-average-memory-looked-fine) - [An exception through std::call_once segfaults in libunwind](#an-exception-through-stdcall_once-segfaults-in-libunwind) - [lavapipe segfaults building an acceleration structure on arm64](#lavapipe-segfaults-building-an-acceleration-structure-on-arm64) - [Headless Chrome prints nothing under qemu-user](#headless-chrome-prints-nothing-under-qemu-user) - [Chrome stops with `No usable sandbox!`](#chrome-stops-with-no-usable-sandbox) - [An arm64 app dies in `ndk_translation` on an API 30 emulator](#an-arm64-app-dies-in-ndk_translation-on-an-api-30-emulator) - [The Vulkan variables name an arch-specific SDK dir](#vulkan-env-names-an-arch-specific-sdk-dir) - [The core registry blacklists `libgstvalidatessim.so`](#the-core-registry-blacklists-libgstvalidatessimso) - [DeepStream: `nvstreammux` stops the pipeline with `reason error (-5)` and no message](#deepstream-nvstreammux-stops-the-pipeline-with-reason-error--5-and-no-message) - [DeepStream: `Unable to load library: libnvinfer_builder_resource_sm75.so.10.16.1`](#deepstream-unable-to-load-library-libnvinfer_builder_resource_sm75so10161) - [DeepStream: `nvv4l2decoder`: `/dev/nvidia0` "isn't a v4l2 driver"](#deepstream-nvv4l2decoder-devnvidia0-isnt-a-v4l2-driver) - [A no-push wrapper build cannot find its own android image](#a-no-push-wrapper-build-cannot-find-its-own-android-image) - [The torch RUN idles for minutes before `uv venv`](#the-torch-run-idles-for-minutes-before-uv-venv) - [`RUNTIME_WHEELS_SOURCE=export` stops a runtime lane](#runtime_wheels_sourceexport-stops-a-runtime-lane) - [A GPU venv ships two onnxruntime distributions](#a-gpu-venv-ships-two-onnxruntime-distributions) - [The torch stage fails with `ORT-CENSUS FAIL`](#the-torch-stage-fails-with-ort-census-fail) - [The torch stage fails with `CHAIN-WHEEL FAIL`](#the-torch-stage-fails-with-chain-wheel-fail) - [A build or smoke stops on an ONNX Runtime that is not the chain's](#a-build-or-smoke-stops-on-an-onnx-runtime-that-is-not-the-chains) - [The wrapper smoke fails `clang --version` after a partial rebuild](#the-wrapper-smoke-fails-clang---version-after-a-partial-rebuild) - [`fatal error: sanitizer/common_interface_defs.h: No such file or directory` on arm64/riscv64](#fatal-error-sanitizercommon_interface_defsh-no-such-file-or-directory-on-arm64riscv64) - [A Jetson GPU container sees no GPU](#a-jetson-gpu-container-sees-no-gpu) - [A USB camera delivers half its frame rate](#a-usb-camera-delivers-half-its-frame-rate) - [A push leaves the repo bare: `core.bare and core.worktree do not make sense`](#a-push-leaves-the-repo-bare-corebare-and-coreworktree-do-not-make-sense) - [A cross-built Rust tool links the build host's libbz2](#a-cross-built-rust-tool-links-the-build-hosts-libbz2) - [HailoRT's configure is slow while the cache looks healthy](#hailorts-configure-is-slow-while-the-cache-looks-healthy) - [`import hailo_platform` fails: `_pyhailort` is a 4 KB module](#import-hailo_platform-fails-_pyhailort-is-a-4-kb-module) **Windows: the layer store (hcsshim)** - [`hcsshim::ActivateLayer 0x20` on an AMD Radeon host](#hcsshimactivatelayer-0x20-on-an-amd-radeon-host) - [`ExportLayer 0x3`, spawn flakes, `ExportLayer 0x70` — disk exhaustion in costume](#exportlayer-0x3-spawn-flakes-exportlayer-0x70--disk-exhaustion-in-costume) - [`ExportLayer 0x3` at finalize of a heavy media layer, disk is fine](#exportlayer-0x3-at-finalize-of-a-heavy-media-layer-disk-is-fine) - [Every RUN step reports `DONE 2841.2s` — the same number, whatever it runs](#every-run-step-reports-done-28412s--the-same-number-whatever-it-runs) - [`hcsshim::ActivateLayer failed (0x20)` during build](#hcsshimactivatelayer-failed-0x20-during-build) - [`ActivateLayer 0x20 "file used by another process"` on commit](#activatelayer-0x20-file-used-by-another-process-on-commit) - [`ImportLayer ... (0xb7) "already exists"` — deterministic, burns the retry budget](#importlayer--0xb7-already-exists--deterministic-burns-the-retry-budget) - [`ImportLayer ... (0xb7)` on the SAME chain-IDs across retries](#importlayer--0xb7-on-the-same-chain-ids-across-retries) - [`failed to reimport snapshot` / `failed to write compressed diff` — the hcs-temp flake family](#failed-to-reimport-snapshot--failed-to-write-compressed-diff--the-hcs-temp-flake-family) - [Finalize dies `unknown stream ID 9` on a Windows Update `.msu`](#finalize-dies-unknown-stream-id-9-on-a-windows-update-msu) - [`exporting layers` prints nothing for 20+ minutes](#exporting-layers-prints-nothing-for-20-minutes) - [A stage fails instantly with `exit code: 1` and zero container output](#a-stage-fails-instantly-with-exit-code-1-and-zero-container-output) - [A Python module imports nowhere though its distribution is installed](#a-python-module-imports-nowhere-though-its-distribution-is-installed) **Windows: container networking** - [`Could not resolve host: github.com` seconds into the first downloading RUN](#could-not-resolve-host-githubcom-seconds-into-the-first-downloading-run) - [`The remote name could not be resolved` — CNI nat subnet drift](#the-remote-name-could-not-be-resolved--cni-nat-subnet-drift) - [nerdctl DNS failure in build](#nerdctl-dns-failure-in-build) **Windows: buildkitd and the store** - [A BK compile step freezes silently after `[output clipped, log limit 2MiB reached]`](#a-bk-compile-step-freezes-silently-after-output-clipped-log-limit-2mib-reached) - [`buildctl prune` returns `Total: 0B` no matter what you pass](#buildctl-prune-returns-total-0b-no-matter-what-you-pass) - [`buildctl` local export dies `file already closed` on a Windows rootfs](#buildctl-local-export-dies-file-already-closed-on-a-windows-rootfs) **Windows: Stevedore and the docker service** - [`cannot access containerd socket ... Zugriff verweigert` (non-admin)](#cannot-access-containerd-socket--zugriff-verweigert-non-admin) - [`runtime "com.docker.hcsshim.v1" binary not installed`](#runtime-comdockerhcsshimv1-binary-not-installed) - [`failed to create TTRPC connection`](#failed-to-create-ttrpc-connection) - [Stevedore service won't start (1053 timeout)](#stevedore-service-wont-start-1053-timeout) - [`error getting credentials - err: exit status 1`](#error-getting-credentials---err-exit-status-1) - [`failed to extract layer ... failed to find link target` pulling servercore](#failed-to-extract-layer--failed-to-find-link-target-pulling-servercore) **Windows: build content and toolchain** - [Windows media build crawls; `Building with ninja -j2`](#windows-media-build-crawls-building-with-ninja--j2) - [Rust smoke test: "rustup could not choose a version of cargo/rustc"](#rust-smoke-test-rustup-could-not-choose-a-version-of-cargorustc) - [A GitLab download "succeeds" with HTTP 200 but is a few KB](#a-gitlab-download-succeeds-with-http-200-but-is-a-few-kb) - [`TVM: llvm-config.exe not found on PATH`](#tvm-llvm-configexe-not-found-on-path) - [`TVM ROCm spike: C:\llvm-patched\bin\llvm-config.exe has no AMDGPU target`](#tvm-rocm-spike-cllvm-patchedbinllvm-configexe-has-no-amdgpu-target) - [`TVM: no member named 'matchIntrinsicSignature' in namespace 'llvm::Intrinsic'`](#tvm-no-member-named-matchintrinsicsignature-in-namespace-llvmintrinsic) - [`lld-link: error: undefined symbol` for template instantiations after a green compile](#lld-link-error-undefined-symbol-for-template-instantiations-after-a-green-compile) - [meson cross: `Summary section 'Build environment' already have key 'host cpu'`, then `Subproject "subprojects/glib" required but not found`](#meson-cross-summary-section-build-environment-already-have-key-host-cpu-then-subproject-subprojectsglib-required-but-not-found) - [Windows base: scoop cannot install a pinned tool, 404 on the installer](#windows-base-scoop-cannot-install-a-pinned-tool-404-on-the-installer) - [AArch64 cross compile aborts with `error: fixup value out of range`](#aarch64-cross-compile-aborts-with-error-fixup-value-out-of-range) - [A source build produces UNPATCHED sources and says `SKIP: ... (already applied)`](#a-source-build-produces-unpatched-sources-and-says-skip--already-applied) - [`atlbase.h` not found when building LLVM in the container](#atlbaseh-not-found-when-building-llvm-in-the-container) - [A build script dies with `The term ... is not recognized`, in the container only](#a-build-script-dies-with-the-term--is-not-recognized-in-the-container-only) - [A consumer's CMake says clang-cl "is not able to compile a simple test program"](#a-consumers-cmake-says-clang-cl-is-not-able-to-compile-a-simple-test-program) - [CodeQL `database create` hangs for hours at near-zero CPU](#codeql-database-create-hangs-for-hours-at-near-zero-cpu) - [A Windows chain stops in seconds at `[bk:publish-gate:bk-windows-…]`](#a-windows-chain-stops-in-seconds-at-bkpublish-gatebk-windows-) - [A declaration that masks its command's exit status](#a-declaration-that-masks-its-commands-exit-status) - [RV1-FREETYPE: riscv64 OpenCV freetype/harfbuzz](#rv1-freetype-riscv64-opencv-freetypeharfbuzz) - [A half-fetched dependency that every retry inherits](#a-half-fetched-dependency-that-every-retry-inherits) - [RVV changed what the optimizer can prove](#rvv-changed-what-the-optimizer-can-prove) - [`webrtcbin`'s action signals never reply; `ffi_prep_cif` returns `FFI_BAD_TYPEDEF`](#webrtcbins-action-signals-never-reply-ffi_prep_cif-returns-ffi_bad_typedef) - [`dtlsconnection ... unexpected eof while reading` at the first DTLS step](#dtlsconnection--unexpected-eof-while-reading-at-the-first-dtls-step) --- ## The shortlist AGENTS.md carried Moved out of `AGENTS.md` on 2026-09-15 (owner decision D10), unedited except for this heading, the relative links and a failure count that had gone stale. The RULES stayed there; this is the reference behind them. Symptom → cause → fix for every failure seen live on both lanes, keyed by the error message you actually get: [`docs/failure-modes.md`](failure-modes.md). Grouped as Linux/cross-lane · the Windows layer store (hcsshim) · container networking (CNI) · buildkitd and the store · Stevedore and the docker service · build content and toolchain. **Four reflexes that page encodes — worth holding before you need them:** 1. **Check free disk FIRST** on any weird hcsshim failure. Disk exhaustion wears three different costumes and only one of them names the disease. 2. **Prefer letting a doomed solve fail cleanly over killing it.** A clean finalize failure leaves no debris; a `buildctl` kill mid-finalize manufactures the deterministic `0xb7` that then costs a `-NoCache` re-run. 3. **After ANY red finalize, REBOOT before further A/B tests.** A wedged hcs state falsifies every experiment run after it. 4. **Identical step timings mean a TIMEOUT, not slow work.** When every RUN lands on the same number, decode that number against the shim teardown knob before debugging the workload (the 2026-08-31/09-01 incident: 2841.2 s byte-identical = lost exit notification × 45 min constant; a 240 s probe kill then misread it as a hard wedge for a day). Timing table + re-mitigate command: `docs/failure-modes.md` § "Every RUN step reports DONE 2841.2s". --- ## Linux and cross-lane ### `exec format error` on a foreign-arch build **Symptom.** `exec format error` **Cause.** QEMU/binfmt not registered after host reboot **Fix.** `linux/scripts/setup-rootless-binfmt.sh` — NOT the tonistiigi container: that installs into the wrong namespace under rootless nerdctl (see *Host prerequisite: QEMU/binfmt* in `docs/linux-cross-builds.md`; this row prescribed exactly that dead-end until 2026-08-24) ### `no space left on device` **Symptom.** `no space left on device` **Cause.** Disk full from cached images/artifacts **Fix.** In order: (1) `linux/host-config/prune-safe.sh` (spares compile caches), (2) `nerdctl rmi` of specific already-pushed tags, (3) `nerdctl system prune -a -f` ONLY with no chain running — it deletes ALL non-container-referenced images INCLUDING tagged cross-stage locals (bit us mid-run 2026-08-18; registry-pinned handoffs survived via re-pull) ### Stale downstream images **Symptom.** Stale downstream images **Cause.** Base image rebuilt but downstream not refreshed **Fix.** Use `--verify-chain` or rebuild from replaced stage ### `no active session` / `grpc: the client connection is closing` mid-chain **Symptom.** `no active session` / `grpc: the client connection is closing` / `DeadlineExceeded` mid-chain (cache reads, layer export, pushes) **Cause.** buildkitd session rot after ~1-2 h of parallel load (BKD1, bit 6× on 2026-08-19/20) **Fix.** Let the worker retries absorb one-offs; on a repeat: stop chain → `systemctl --user restart buildkit.service` → relaunch (cache mounts provably survive; builds fast-forward) ### `registry_pin_ref` fails on a fresh push **Symptom.** `registry_pin_ref` fails on fresh push **Cause.** Registry hasn't propagated the new manifest **Fix.** Now uses `retry()` with 5 attempts; wait a few seconds and retry ### Terminal freeze during a long build **Symptom.** Terminal freeze during long build **Cause.** Build output overwhelms terminal **Fix.** Use `setsid` / `disown` for very long builds ### LiteRT configure: `fatal: expected flush after ref listing` **Symptom.** Every LiteRT/TFLite cmake configure fails cloning eigen; all three android lanes down inside one window (2026-08-21) **Cause.** `tools/cmake/modules/eigen.cmake` declares a single `GIT_REPOSITORY` (`gitlab.com/libeigen/eigen`), so a momentary gitlab outage is a total outage **Fix.** Already handled by `linux/scripts/03-media/build/litert/android/litert-eigen-fetch.sh`, sourced by both LiteRT build scripts. It sets two knobs of upstream's `OverridableFetchContent` and pins no commit of its own (the tag stays upstream's): - `GIT_REPOSITORY_AND_TAG_TO_URL_eigen=ON` turns the clone into an archive download of the *same* pinned commit — how the C-API/wheel configure paths already fetch it; - `_MATCH`/`_REPLACE` rewrite that archive URL. A `;` in the replacement makes it a cmake **list**, which `ExternalProject` downloads "in turn until one succeeds". The second URL is TensorFlow's own mirror of the same gitlab path (the `tf_mirror_urls()` rule in `third_party/eigen3/workspace.bzl`), verified byte-identical on 2026-08-23: both URLs for commit `ea13a98d…` returned sha256 `35c6126e…`, 2870994 bytes. It degrades safely — if the regex ever stops matching, the URL is left untouched and the fetch behaves as it does today. ### `logging.sh: line NNN: action: unbound variable` instead of the real error **Symptom.** A build stage dies and the only diagnostic in the log is `.../01-core/logging.sh: line 119: action: unbound variable`. The command that actually failed is nowhere. Hit on 2026-08-30, where it stood in for a parallel-GCC apt-lock failure. **Cause.** `_install_trap` defined `on_err` as a NESTED function that read its reporting action (`err` or `warn`) out of the installer's own `local action` — i.e. through **dynamic scope**, which only resolves while `_install_trap` is still on the call stack. An ERR trap fires long after the installer has returned, so under `set -u` the handler aborted on its first line. Two things were lost, not one: that abort **replaced** the real error text, and the action it existed to run never ran — `install_err_trap` never exited 1 and `install_warn_trap` never printed. Every script that sources `logging.sh` inherited it. **Fix.** Fixed 2026-08-31. `on_err` is a top-level function, and `_install_trap` bakes the resolved action into the trap string with `printf -v '%q'` so it is expanded AT INSTALL TIME, while `LINENO` and `BASH_COMMAND` stay escaped and expand AT FIRE TIME (reverse that escaping and every failure is reported at the install site's line number, forever). `_LOG_TRAP_ACTION` keeps the two-argument call working for `build-gcc.sh`, which tears the trap down and re-arms the bare string by hand around its configure step. Regression cover: `linux/scripts/tests/test-logging-err-trap.sh`. **Behaviour note:** inside a `set +e` window `install_err_trap` now exits 1 where it used to print noise and carry on. No caller has such a window today; a new one would feel it. ### A fault-injection test passes and proves nothing (`grep` is ugrep) **Symptom.** A regression suite is green — and stays green when you break the very thing it guards. Its fault-injection arm (a stub that fails when its argv matches a pattern) never fires, and nothing reports that: a pattern matching nothing looks exactly like a run with no fault injected. **Cause.** The pattern began with `--` and was handed to `grep` positionally. `grep` on this host is **ugrep**, which parses a leading-`--` pattern as a command-line option instead of as the pattern. Found 2026-08-31 in `linux/scripts/tests/test-iree-wheelhouse-stages.sh`, whose header claimed five `|| return 1` call sites were covered when two were. **Fix.** Pass the pattern behind `-e`: `grep -qE -e "$pat"`. The general rule — a regression test has to be SHOWN to go red before it counts, and its header must say what it does not cover — is in [`cross-build-verification.md`](cross-build-verification.md#the-linuxscriptstests-suites), along with the second half of this same incident: with injection finally working, the obvious `rc == 1` assertion still passed on every mutation. ### `GENAI-BIND SKIP` reported green while the native binding is broken **Symptom.** The runtime-image smoke prints `GENAI-BIND SKIP: onnxruntime_genai is not installed` and the run stays green — on an image where that wheel IS installed and only its native library refuses to load. **Cause.** The smoke collapsed two unrelated situations into one exit code (3 = skip): "this arch ships no genai wheel", which is benign and is the ARCH-PARITY table's business, and "the distribution is installed but importing it raises", which is a defect. Nothing else in the chain can see the second one: `smoke-torch-venv.sh`'s `installed_version()` falls back to `importlib.metadata` when the import raises, so it cheerfully reports the pinned version of a package that cannot be imported, and the ARCH-PARITY table reads only dist-info directory names. A riscv64 binding with an unresolved `__atomic_*` or a missing `NEEDED` would have shipped behind three green gates. The lane this was found on is written up in [`gen1-riscv64-genai.md`](gen1-riscv64-genai.md). **Fix.** Fixed 2026-08-31 in `smoke_genai_py` (`06-packaging/smoke-common.sh`): on an import failure the program asks `importlib.metadata` whether the DISTRIBUTION is present. Present → exit 1 with `GENAI-BIND FAIL: … is INSTALLED but not importable`, pointing at the `.so`'s NEEDED/undefined symbols; absent → the SKIP. Transferable form: an exit code meaning "there was nothing to test" must not be reachable from a state meaning "the thing under test is broken". ### The documented `GENAI_ALLOW_RISCV64` back-out does not reach the smoke **Symptom.** A lane toggle is set in `versions.env`, the build honours it, and the runtime-image smoke still reddens by asserting the component the toggle switched OFF. Exporting the variable in the host shell before running the smoke changes nothing. **Cause.** Two boundaries, and the value crossed neither of them. As far as the host shell is concerned a `nerdctl run` begins with an empty environment, so exporting the toggle before the smoke is invisible inside the container. And the ARG/ENV that carries it is declared on a BUILDER stage, while the final stage of `Dockerfile.media` descends from a different parent — so the shipped image holds no copy of it either. The baked `smoke-torch-venv.sh` therefore read the toggle as unset every time and took its default arm: the escape hatch the docs promised had never once worked. Depth on this particular toggle: [`gen1-riscv64-genai.md`](gen1-riscv64-genai.md). **Fix.** Fixed 2026-08-31: `smoke-runtime-image.sh` reads the value host-side out of `versions.env` (`_rt_versions_env_pin`, empty on a miss = "not asserted") and forwards it with an explicit `-e` on the `nerdctl run`, so the smoke asserts the policy the build was configured with. **A new toggle has to be checked on both hops** — build ARG into the SHIPPED stage, host env into the container — and a verifier should read a marker the producer actually wrote rather than re-derive the decision: a producer defaulting `:-false` against a verifier defaulting `:-true` disagrees in the failing direction. ### An unresolved `NEEDED` in a library that nothing scans **Symptom.** Nothing at all, which is the point. A shipped `.so` carries a dependency that cannot be resolved, every gate is green, and the failure surfaces as an import or `dlopen` error on the target. **Cause.** `03-media/runtime/validate-media-runtime.sh` hunts unresolved `NEEDED` entries over `ARTIFACTS` (the gst / libcamera / ffmpeg binaries) plus the GStreamer plugin directory. Its `LIB_DIRS` sweep is a DIFFERENT check — ELF **machine** only, and advisory. An install prefix in neither list is scanned by nobody; `/usr/local/lib/onnxruntime-genai/lib` was in neither until 2026-08-31. The concrete thing that hid behind it: GenAI's CMake, unlike upstream onnxruntime's, carries no libatomic probe, so a riscv64 build can need `-latomic` and only say so at load time. That lane: [`gen1-riscv64-genai.md`](gen1-riscv64-genai.md). **Fix.** Fixed 2026-08-31 — the prefix joined `LIB_DIRS` (resolution root plus the advisory arch sweep) AND its `lib/` is now walked by `scan_plugin_directory` (misnamed but generic, so it was reused rather than copied), feeding the existing missing-soname machinery. A `[ -d ]` guard makes it a no-op when the lane is off. **The check to carry away:** when an image gains an install prefix, ask which scanner covers it — and remember that an ELF-machine sweep is not a dependency scan. ### A prune step deletes the wheel a later step requires **Symptom.** Again nothing yet — the step that would have broken was disarmed by an unrelated accident. `install_project_environment` prunes conflicting wheels before `build_uv_sync_args` goes looking for the local ones, and one prune glob matched a wheel the later step requires. **Cause.** `03-media/runtime/assemble-torch-app.sh`'s `prune_conflicting_onnx_wheels` carried a `*genai*.whl` glob on its DEFAULT `ONNX_PACKAGE=onnxruntime` arm. Far too broad: it also names the plain CPU genai wheel this media lane now produces for every architecture, which the uv-sync step and the ARCH-PARITY table both rely on being present. Had the deletion succeeded, the earlier GENAI-DRIFT bug would have returned — absent a local wheel, the resolver quietly prefers the app lock's PyPI build. It survived only because `/opt/wheels` happens to be mounted read-only, so the removal errored and `|| true` discarded that error; flipping the mount to writable would have broken all three architectures at once. Full narrative: [`gen1-riscv64-genai.md`](gen1-riscv64-genai.md). **Fix.** Fixed 2026-08-31 by narrowing the glob to the GPU-flavoured variants the arm actually means (`*genai_cuda-*`, `*genai_rocm-*`, `*genai_directml-*`), beside its `*_gpu-*` / `*_migraphx-*` neighbours. **The lesson is about `|| true` on a destructive step:** it hides the failure AND the fact that the step was wrong, so the bug sits latent until something unrelated arms it — here, a read-only mount becoming writable. ### The disk guard aims at the wrong number **Symptom.** The chain prunes at 40G free between stages, reports success, and the NEXT stage refuses anyway: `[ERROR] runtime lane refused: 56G free, ~120G needed` (2026-09-02: six manual prunes; 2026-09-03: the Flutter ship build, attempt 1). **Cause.** Two different numbers. `CROSS_DISK_GUARD_GB` (40) is a floor for *this* stage; the runtime lane needs `CROSS_RUNTIME_LANE_GB` (120) per wrapper build, and `--only runtime` additionally pulls the three `cross-android-` images (~40G uncompressed each) as its artifact source. A guard that reclaims to a fixed floor is therefore satisfied exactly when the lane is not. **Fix.** `_chain_stage_disk_guard` in `build-cross-chain.sh` reclaims to what comes *next* (`_chain_runtime_lane_need_gb`), and the launch-time preflight warns when the run will enter the lane with less than stage cost + lane need. **When the guard reclaims nothing.** `buildctl du` showing every regular record `Reclaimable: false` is not a full cache — it is killed chains' leaked *leases* (2026-09-03: 351 records / 251 GB pinned, `prune-safe.sh` freed 0). Leases die with the daemon: `systemctl --user restart buildkit.service` (no build running), then `PRUNE_KEEP_GB= linux/host-config/prune-safe.sh`. `--keep-storage` bounds the WHOLE store, and the non-candidates (cache mounts ~166G + `source.local` ~10G) count toward it; since CON53 (2026-10-01) the script and the chain's fallback add them, so `` is GB of layer cache, not of the store. Local `:latest-` images are re-pullable and not build inputs; `nerdctl rmi` them last. Restarting the daemon is the BKD1 remedy above wearing a disk costume. ### A renamed or dropped distro package kills a stage hours in **Symptom.** A media or toolchain stage dies four hours into a rebuild with `E: Unable to locate package ` — or, worse, does not die: an `install_target_packages ... || true` swallows it and the feature silently vanishes from the shipped image. **Cause.** Ubuntu renames and drops binary packages between releases, and this tree pins a rolling one (`UBUNTU_CODENAME` in `versions.env`). 26.04 renamed `libfreetype6-dev` to `libfreetype-dev`, replaced `libopenexr-3-dev` with `libopenexr-dev`, and `libvvdec-dev` never existed on ports at all. Nothing noticed for months because warm apt caches still answered for the old names — the tree was un-buildable from scratch and no one knew, because nothing ever built from scratch. **Fix.** `linux/scripts/verify_package_names.py`, wired into `preflight.sh` as the `pkg-names` check. It extracts every distro package name **`linux/scripts/**`** asks for and resolves each against the live Ubuntu indices for the pinned codename — `archive.ubuntu.com` for amd64, `ports.ubuntu.com` for arm64/riscv64 — before a build starts rather than four hours in. What it reads, and how a verdict is reached: - Sources: `install_target_packages` / `install_optional_target_packages` / `install_host_packages` / `install_deps_preamble` / `apt_install` call sites, bare `apt-get install` lines, `*_packages` and `*_pkgs` array literals, `append_unique_packages` / `append_available_packages` in `01-core/package-lists.sh`, the `_CPYTHON_EXT_DEV_PKG_TABLE` rows in `01-core/cpython-dev-packages.sh`, and the soname map at `03-media/runtime/so-package-map.txt`. Names occur one-per-line AND several-per-line; a per-line assumption is how an earlier audit reported the wrong count, so `--list` prints exactly what was extracted. - **UNGUARDED** requests FAIL the gate: a missing name aborts the whole apt transaction and kills the stage. **GUARDED** ones (`|| true`, a `||` fallback chain, a self-filtering helper, an enclosing apt probe) only WARN — the cost there is one wasted apt round-trip per run, not a dead build. - A name apt can still install through `Provides:` counts as present but is reported: a virtual name disappears the next time the provider is renamed. - Names from a non-Ubuntu repo (NVIDIA/ROCm/TensorRT) are UNVERIFIABLE, never dead; that file list lives in the script. - An array no installer ever consumes is reported as unchecked, not silently dropped — that is what keeps sdkmanager component lists out of the verdict. - **Not covered:** package names written directly in a `Dockerfile` RUN (`Dockerfile.toolchain`'s `binutils-dev` is the only one today), and anything a script assembles at runtime rather than spelling out. **Offline is a SKIP, never a pass.** Indices are cached per codename with a 6h TTL (`PKG_NAMES_TTL`, `PKG_NAMES_CACHE_DIR`): ~13s cold, ~1.5s warm. With no network and no cache the check says so loudly and exits 0; with a stale cache it verifies anyway and says the cache was stale. **The transferable lesson.** The gate carries its own extraction self-check — known package names that MUST be found (a several-per-line array row, a backslash-continued call, a `package-lists.sh` helper) and decoys that must NOT be (an sdkmanager component, a table column, a word from an `echo` string). Break the extractor and the check exits 2 before it ever reports green. A gate whose input extraction can silently degrade to nothing is the failure this repo keeps re-learning; a scanner has to prove it is still scanning. ### apt: `libc6:i386` install is unsatisfiable after an archive/ports drift **Symptom.** The android stage dies a few seconds into the SDK install, on every arch, with: ``` E: Unable to satisfy dependencies. Reached two conflicting assignments: 1. libgcc-s1:arm64 is selected for install 2. libgcc-s1:arm64 Depends libc6:arm64 (>= 2.35) but none of the choices are installable: - libc6:arm64 is selected for removal because: 1. libc6:i386=2.43-2ubuntu2.4 is selected for install 2. libc6:arm64 Breaks libc6:i386 (!= 2.43-2ubuntu2.3) ``` The arch named varies by lane: `arm64` on the riscv64 image, `riscv64` on the arm64 one, both on amd64. **Cause.** Two facts combine. The compiler base installs `libc6` for BOTH foreign arches, but `Dockerfile.media`'s apt reset wipes all sources and writes only the build host's and the current target's. When archive.ubuntu.com publishes a libc6 update before ports.ubuntu.com (measured 2026-09-11: i386 2.4 on archive, arm64/riscv64 still 2.3 on ports), the freshly installed `libc6:i386` Breaks the frozen foreign copies — and apt cannot upgrade an arch it has no source for. The mirrors were back in sync minutes later; the failure mode is the drift, not the package. **Fix.** `cross_ensure_installed_foreign_arch_sources` (`01-core/cross-apt.sh`), called from `android-sdk.sh` before `apt-get update`: it writes a per-arch ports source for every installed foreign arch that has none, so apt moves all libc6 arches to the new version together. Guarded by `test-cross-apt.sh`, including a whole-line wiring check — the first mutation run showed a substring check over the file still passes with the call deleted, because the helper's name appears in a comment. **Retry after a drift.** Nothing in the image needs rebuilding. Wait for the mirror to catch up and re-run the stage (`--from-stage android`, ancestry checked) — the failed run left no partial tags (0/7 cache stages exported). ### A from-source CPython silently drops an extension module **Symptom.** `import ssl` / `import sqlite3` / `import lzma` raises `ModuleNotFoundError` inside a shipped image, or an interactive `python3` has no line editing. The toolchain stage was green: `make -k || true` skips an extension whose dev header was missing at configure time and says so only in passing. **The 2026-08-09 incident** was `libsqlite3-dev`: the host closure got sqlite transitively through GUI dev packages, the cross-target install list forgot it, and half the Python ecosystem imports `sqlite3` on the way to something else. **One table, two consumers.** `01-core/cpython-dev-packages.sh` holds every row as ` ...`, and nothing else may keep a second list: | consumer | reads | verdict | | --- | --- | --- | | `package-lists.sh base_image_os_packages` | `cpython_ext_dev_packages` | the HOST closure installs the same set | | `build_python.sh _python_cross_stage_target_dev_pkgs` | `cpython_ext_dev_packages` + `cpython_ext_dev_packages_required` | one atomic apt install, then a per-package `dpkg-query`; a missing REQUIRED package is **fatal** | | `build_python.sh _python_dynload_audit` | `cpython_ext_modules` | audits each staged `lib-dynload`; a missing `.so` **warns**, on every row | The class column therefore governs the *package*, not the `.so`. The audit is warn-only on purpose: promoting the required rows to fatal there flips all three arches at once and only a cross rebuild can price that, so the decision stays open rather than being smuggled in with a refactor. The free-threaded trees are the exception: they ship as the image's runtime interpreter, so a missing `_ssl`, `_hashlib`, `_sqlite3`, `zlib`, `_bz2`, `_lzma` or `_ctypes` is fatal there ([`consumer-image-contract.md` § The free-threaded Python](consumer-image-contract.md#the-free-threaded-python)). Until 2026-09-05 the audit carried its own hand-written array instead — seven modules that had never gained `readline` after LOG23 added it to the table, which is the same desync in miniature. **The parsing trap.** `build_python.sh` runs under `IFS=$'\n\t'`, where an unpinned `read` does not split on spaces: a row naming two modules (`libssl-dev required _ssl _hashlib`) would arrive as one word and the audit would look for an extension called `_ssl _hashlib`. Every accessor in the table file pins `IFS=' '` on its `read` for exactly that reason, and `tests/test-cpython-ext-table.sh` runs `cpython_ext_modules` and `cpython_ext_dev_packages` under both values of `IFS`. **What only a real build shows.** Whether the five modules the audit gained on 2026-09-05 (`_zstd`, `readline`, `_curses`, `_uuid`, `_decimal`) actually land in `lib-dynload` on arm64 and riscv64. Four of them are `optional` rows and `readline` is `required`, which makes its package fatal at the install step, not its `.so`. The audit warns on every row, so a warning there is information, not a failure, and the toolchain smoke's own stdlib battery is what turns a genuinely broken interpreter red. ### The delete guard denies its own legitimate work `.claude/hooks/guard-destructive-deletes.py` matched a delete VERB and a PROTECTED PATH independently, anywhere in the command. So any command that both deleted something harmless and merely *mentioned* a system path was denied: ``` rm -rf scratch && cc -o x /opt/gcc-16.2.0/bin/gcc # denied: "a system directory" ``` The `/opt` here is a compiler, not a delete target. This fired five times on 2026-09-02 against a scratch directory under the job tmpdir, each time costing a tool call and a rewrite. Two earlier variants of the same shape are recorded in the file's own header: `--rm` matching `\brm\b`, and `sed 's/^/ /'` reading as the filesystem root. **The fix.** Split the command on `;`, `&&`, `||`, `|` and newlines, never inside a quoted span (`_split_outside_quotes`), and run the protected-path patterns only on the segments that actually carry a delete verb. A preceding `cd` target is carried into later segments, so `cd /usr && rm -rf *` still denies — the relative delete cannot escape the directory it was given. **The transferable lesson.** A guard whose two halves are matched independently across a whole command is not checking a relationship, it is checking co-occurrence. Co-occurrence guards look strict and behave randomly: they deny safe work and, worse, teach the operator to phrase commands to avoid the guard rather than to be safe. Bind the dangerous verb to its own argument. Both directions are pinned in `test-delete-guard.sh` — widening the scope back to the whole command turns the allow-cases red, and dropping the `cd` tracking turns the deny-cases red; the recorded mutation, `delete-guard.quote-aware-split`, pins the quote-aware split. ### OpenCV: `std::complex` breaks on a shadowed `complex.h` The riscv64 `opencv-gst` stage dies compiling `modules/core/src/hal_internal.cpp`: ``` error: expected unqualified-id before '_Complex' [-Wtemplate-body] 540 | int ldsrc1 = (int)(src1_step / sizeof(std::complex)); ``` The caret sits on `complex` inside `std::complex`, because `complex` is a macro by then. `hal_internal.cpp` includes the **C** header at line 50, under `HAVE_LAPACK`, and `precomp.hpp` has already pulled in `` — so the class is declared correctly and only later *uses* of the token expand. In C++, `` is supposed to reach libstdc++'s wrapper, which includes glibc's header and then removes the damage: ```c # include_next # ifdef _GLIBCXX_COMPLEX # undef complex // <- the guarantee # endif ``` glibc's own `/usr/include/complex.h` has `#define complex _Complex` with no `__cplusplus` guard, so whoever wins the lookup decides whether C++ still works. CMake passes OpenCV `-isystem /usr/include`, and `-isystem` is searched before the compiler's C++ directories — so glibc's header wins and the `#undef` never happens. Measured with the riscv64 cross g++: | flags | `` resolves to | `complex` macro | | --- | --- | --- | | none | `…/riscv64-linux-gnu/include/c++/16.2.0/complex.h` | not defined | | `--sysroot=/ -isystem /usr/include` | `/usr/include/complex.h` | `#define complex _Complex` | Reproduced in three lines — `#include `, `#include `, then one `std::complex` — and it fails on every target compiler once the flag is present, so this is not architecture-specific in nature. **The fix.** `build-opencv.sh` writes a two-branch `complex.h` shim next to the source tree and prepends it with `-I`. GCC searches every `-I` before every `-isystem`, so the shim wins regardless of what CMake appends — the same precedence trick the file already uses to pin our FFmpeg prefix. The shim restores libstdc++'s behaviour in C++ and is a transparent pass-through in C, which matters because the same flag string is handed to both `CMAKE_C_FLAGS` and `CMAKE_CXX_FLAGS` (OpenCV compiles C third-party code such as openjp2 that legitimately wants the `complex` macro). **Why it surfaced only now.** It is not a regression. The previous run never compiled this file: libcamera failed earlier on an unrelated link error, and BuildKit builds those stages in parallel, so the whole lane died first. Fixing libcamera let the build get far enough to reach a defect that was always there. **Still open:** *which* CMake package contributes `-isystem /usr/include` on riscv64 and not on the other lanes. `ZLIB_INCLUDE_DIR=/usr/include` is passed on every cross arch and all three resolve zlib identically, so that alone does not explain it. The shim makes the build immune either way; the injector is worth finding so the flag can be removed at the source. ### A smoke that never passed and always excused itself `validate-media-runtime.sh` carried a "QEMU Cross-Arch Binary Smoke" that ran each built binary under qemu-user and, on failure, printed: ``` WARN: /usr/bin/qemu-riscv64 gst-launch-1.0 --help failed (may be expected for cross builds without full sysroot) ``` Three things were wrong with it at once, and each alone would have been enough: - **Both branches were `echo`s.** No counter, no variable, no exit code — there was no path from that loop to a failure. The ELF block directly above it does it properly: it counts `core_mismatches` and `exit 1`s. - **The excuse covered every case it could ever see.** The block only ran under `cross_build_is_active`, so "may be expected for cross builds" applied to 100% of its invocations. That is not a diagnosis, it is a blanket. - **The failure was mechanically guaranteed.** qemu-user needs the target's `ld.so` and libraries via `-L` or `QEMU_LD_PREFIX`; neither was passed, and `QEMU_LD_PREFIX` appears nowhere in the repo. Every binary died in the dynamic loader before `main` — exactly the "full sysroot" the message names, which makes it a fixable bug, not an expected condition. Final score before removal: **7 failures, 0 passes**, across arm64 and riscv64 in one run. **Removed rather than repaired, 2026-09-02.** Repairing it needs a multi-hour media build to prove the sysroot prefix actually works, and shipping an unproven-but-now-fatal gate is how you lose the next run. Functional coverage already exists where it belongs: the runtime-image smoke boots the real per-arch image and gates the manifest on it — that is what caught the riscv64 venv gap the same day. **The transferable lesson.** A check whose failure branch is always excused teaches its readers that WARN lines are noise, which is worse than having no check: it costs attention on every run and buys nothing. When you find one, the question is not "how do I make this pass" but "what would it take to make this able to fail" — and if you cannot answer that today, delete it and say where the real coverage lives. --- ### A trailing conditional fails the whole script The 2026-09-03 runtime build died on amd64 in `setup-torch-venv.sh` with `exit code: 1` and no error text — the last lines were a healthy `uv pip install --no-deps ml_dtypes` / `Checked 1 package in 2ms`. The cause was one line at the END of `reconcile_local_wheels` in `assemble-torch-app.sh`: ```bash [ "${have_torch_family}" = "true" ] && _backfill_torch_runtime_deps } ``` A function returns the status of its last statement. With no local torch wheel (every amd64/arm64 build — torch comes from `uv sync` there) the test is false, the `&&` list is `1`, so the function returns `1`; the caller runs under `set -euo pipefail` and exits. Nothing printed anything, because nothing had failed in the ordinary sense. riscv64, the only arch WITH a local torch wheel, was the only arch that would have passed. Two things let it through: - **`set -e` is not tripped by the list itself.** `a && b` is exempt from `-e`, which is why the idiom feels safe; the trap is only that its status becomes the function's status, and the bare call `reconcile_local_wheels` in the caller is NOT exempt. - **The characterisation test ran without `set -e` and never looked at `$?`.** It pinned the sequence of `uv` calls the function makes (its stated purpose) and so proved the refactor behaviour-preserving for every path — including the path that now returned `1`, because a return status is not a `uv` call. Fix: an `if ... fi` (or `|| true` when the right-hand side is genuinely optional). The test harness now runs the function under `set -eu` and asserts `0` for a non-torch wheel set; against the broken revision that case fails (`expected '0', got '1'`). **Where else this hides.** A census of `cond && cmd` as the last statement before a closing brace finds 17 sites in the tree; almost all are deliberate *predicates* (`[ -n "${_t}" ] && [ "${_t}" != "${_b}" ]`) whose callers use them in `if`/`||`. The bug shape is narrower: an *action* on the right-hand side and a *bare* call site under `set -e`. A regex on the function alone cannot tell the two apart, so the gate that landed for it, preflight slug `trailing-conditional` (`linux/scripts/verify_trailing_conditional.py`), flags every function that ends on a conditional and freezes the reviewed predicates in `linux/scripts/trailing-conditional.allow`: [code-quality-tooling.md § Trailing-conditional returns](code-quality-tooling.md#trailing-conditional-returns-trailing-conditional). It cannot see call sites, so a function-level test must still assert the exit status on the "nothing to do" path, not only the calls it makes. ### The copied Rust toolchain is the builder's arch **Symptom.** The arm64 package stage dies with `exit code: 127` and *no* `command not found` line — the last output is `Keeping existing /usr/local/cargo/bin/rustup (rustup toolchain wins over PATH lookup)` (2026-09-03). In the shipped `:latest-cross-arm64` of 2026-09-01: ``` $ rustc --version bash: /usr/local/cargo/bin/rustc: cannot execute: required file not found $ ls /usr/local/rustup/toolchains 1.98.0-x86_64-unknown-linux-gnu nightly-2026-06-28-x86_64-unknown-linux-gnu $ du -sh /usr/local/rustup # 2.0G ``` **Cause.** `Dockerfile.package` COPYs `/usr/local/{rustup,cargo}` from `artifact-source`, and artifact-source is the **amd64 host** image of the cross chain (`--platform=${ARTIFACT_PLATFORM}`). Everything else it copies is a *target* tree (a cross-built `/opt/gcc-`, `/opt/llvm-target`, `/opt/opencv5`), but the Rust toolchain there is the builder's own x86_64 install with `aarch64`/`riscv64gc` *targets* added — that is what a cross-compiler needs and exactly what a native arm64 image cannot run. Every arm64/riscv64 runtime image shipped 2 GB of dead x86_64 binaries at the front of `PATH` (`/usr/local/cargo/bin` precedes `/usr/bin`), so `cargo` was broken rather than falling back to Ubuntu's. **Why nothing caught it.** Three layers, each blind in its own way: - `wire_cargo_symlinks` checked `-x` (true for a foreign ELF) and kept the shims. - `report_rust_provenance`'s hard gate reads `RUST_VERSION`, which did not reach the script until 7eca29c6 (2026-09-03) — the day it started running was the day it failed. Before that: `NOTE: RUST_VERSION unset; cannot verify`. - The runtime smoke's ADV/HAVE table (`_advert_verdicts`) turned an *unreadable* actual value into `SKIP`, so `rustc --version` failing under `2>/dev/null` was reported as "could not read", not as a mismatch. That arm is fatal since 2026-09-04 (`UNREAD`); see docs/cross-build-verification.md, gate A. **Why 127 with no message.** `_got="$("${CARGO_HOME}/bin/rustc" --version 2>/dev/null | awk …)"`: the loader's `cannot execute` goes to `/dev/null`, `pipefail` propagates rustc's 127 through `awk`, and `set -e` exits on the assignment. The pattern is fine for a probe whose result is *tested*; it is not fine when the probe's failure is itself the fault. **Fix.** `setup-package-image.sh` `ensure_native_rust_toolchain` (before `wire_cargo_symlinks`): if `${RUSTUP_HOME}/toolchains/` holds no `*-` dir, wipe `${RUSTUP_HOME}` and everything in `${CARGO_HOME}` but its `registry` cache mount, and run the same `install-rust.sh` the toolchain stage uses, with `BUILD_MODE=native` and `RUST_INSTALL_CARGO_C=0` (compiling cargo-c under QEMU would cost an hour; the `cargo-c` apt package is what `wire_cargo_symlinks` links in that case — one minor version behind the pin, an accepted skew). amd64 is a no-op: the copied toolchain *is* native there. The native install downloads the pinned stable + clippy + rustfmt + `wasm32-unknown-unknown` and the pinned nightly, same set as amd64, so the three arches ship the same toolchain surface. Since 2026-10-13 `prune-host-payload.sh` empties the foreign toolchain in `artifact-source`, before the COPY, so the 745 MB x86_64 layer no longer ships only to be deleted; this function then installs over an empty tree ([why](artifact-copy-completeness.md#the-host-payload-stays-with-the-host-that-runs-it)). **Guards.** `check_rust_toolchain` in `smoke-runtime-image.sh` executes `rustc --version` in the shipped image, requires the `RUST_VERSION` pin, reads the active toolchain's host triple (`rustup show active-toolchain`) and requires `cargo-cbuild` — it fails on the 2026-09-01 image (`tests/test-runtime-image-gates.sh`). The same class applies to any *host* tree copied from artifact-source: `/opt/flutter` ships bare from the amd64 sdk stage, and `bootstrap_flutter_sdk` (`setup-package-image.sh`) caches the target-arch Dart SDK, which `check_flutter` verifies the same way ([Bootstrapping Flutter in the package stage](artifact-copy-completeness.md#bootstrapping-flutter-in-the-package-stage)). The general audit for that class is `check_manifest_tree_arch`: it reads the ELF machine of everything under every tree in `runtime-artifacts.manifest` and fails when a foreign image carries builder-arch objects — docs/artifact-copy-completeness.md#the-shipped-trees-must-carry-the-images-own-arch --- ### A packaging script dies with no message **Symptom.** A `Dockerfile.package` / `Dockerfile.torch` RUN ends with `did not complete successfully: exit code: N` and the last line of output is an ordinary progress line. Nothing names a command, a line, or a reason. Seen twice in the 2026-09-03 18:13 run: the arm64 package stage (`exit code: 127`, last line `Keeping existing /usr/local/cargo/bin/rustup`) and the amd64 torch stage (`exit code: 1`, last line `Checked 1 package in 2ms`). **Cause.** `set -e` exits silently by design. The three image-side scripts — `setup-package-image.sh`, `setup-torch-venv.sh`, `assemble-torch-app.sh` — are several hundred lines of small functions, so the failing command can be any of them, and the usual suspects say nothing on the way out: * a command substitution under `pipefail` whose stderr is discarded (`_got="$(rustc --version 2>/dev/null | awk …)"` — an unrunnable foreign-arch `rustc` produces exit 127 and not one byte of output); * a helper whose last command is a failed test, which becomes the helper's own return value (see *A trailing conditional fails the whole script*); * any `uv`/`git` call that fails with its diagnostics already consumed. **Fix.** The three scripts run `set -Eeuo pipefail` and `install_err_trap` (`01-core/logging.sh`), so a `set -e` death now prints `[ERROR] Command failed (line N): ` before it exits. `-E` is the load-bearing flag: without `errtrace` the trap is not inherited by shell functions, which is where all the work happens. `Dockerfile.torch` COPYs `logging.sh` into `/opt/scripts/core/` for this. The probe that produced the arm64 127 also names its own failure now: an unrunnable `${CARGO_HOME}/bin/rustc` reports `does not execute` and points at `ensure_native_rust_toolchain`. **Reading it.** The trap reports the line of the *failing command*, not of the caller. Match it against the script in the image (`nerdctl run --rm sed -n 'p' /opt/scripts/packaging/