Refactoring backlog — CLOSED 2026-08-30

Split out of refactoring-backlog.md so that file shows only OPEN work. Nothing here needs action; it is kept because the entries record WHY a thing is the way it is, and because one was closed by being REFUTED rather than fixed — the kind of finding that gets re-discovered if the reasoning disappears. Earlier rounds: refactoring-backlog-archive-2026-08-10.md, refactoring-backlog-archive-2026-08-27.md.

Closed on 2026-08-30

• ✅ sccache caches NOTHING in the OpenCV step — CLOSED by REFUTATION. The entry (OPEN 2026-08-26) attributed 2359 sccache-launcher bypass messages to an OpenCV-specific failure (“sccache fails on EVERY compile with the main build dir /tmp/opencv-1/build as cwd”). That attribution is wrong. The 2359 faults were the wrong-server bug: pre-UDS, media stages ran CONCURRENT BuildKit containers that reached each other’s sccache server over the fixed TCP port (4226), the wrong server accepted the job (“Server sent CompileStarted”) and then could not find the caller’s files — caused by: No such file or directory (os error 2). Fixed 2026-08-27: b4078ad1 (container-local server identity) + 4aa92fb6 (SCCACHE_SERVER_UDS — the address is a filesystem path in /tmp, private by construction), the same night the guarded launcher (fece3558) started carrying the build past the failures. DOCUMENTED at the time in docs/build-cache-tiers.md § 5.1: “SCCACHE_SERVER_UDS took the media stage from 2359 sccache faults to zero.” Evidence trail that closes it (log forensics, 2026-08-30):

  • staged-media7.log (2026-08-26 22:02, PRE-UDS): 2359 bypass messages in BOTH the opencv 2/3 and onnxruntime 2/9 steps — not OpenCV-exclusive, as the entry claimed.

  • Every post-UDS run: 0 bypass messages — staged-media9.log, staged-media10.log, and media-arm64.log (2026-08-30, the QNN-LINUX build, where OpenCV compiled all 1660 objects with the launcher in place and the ffmpeg sibling step recorded 3104 compile requests at 99.64 % cache-hit rate). No code change was needed beyond the already-landed fixes; the item is closed as REFUTED, with the wrong-server writeup kept on record so the misdiagnosis is not re-discovered.

• ✅ F2 — Compiler-cache abstraction consolidation — DONE. All launcher resolution now routes through ONE resolver. 01-core/compiler-cache.sh gained _resolve_compiler_cache_launcher(), which uses common.sh’s compiler_cache_launcher() (01-core) when it is loaded — the case for every media/ORT caller — and otherwise falls back to an inline bootstrap (the android preamble sources compiler-cache.sh standalone). Both paths implement the identical decision: guarded launcher when reachable, bare sccache when not, ccache when the server is dead, never empty. setup_ccache and setup_sccache both consume it; the verify-critical-fixes.sh gate (guarded-launcher-plus-sccache decision) still passes without edits. Pinned by the new suite linux/scripts/tests/test-compiler-cache.sh (8 assertions, mutation-covered): framework-path verdict pass-through, bootstrap guarded-launcher preference, dead-server and absent-sccache fallbacks, and setup_sccache keeping RUSTC_WRAPPER sccache-class even on a ccache verdict (Rust has no ccache fallback). Behavior-identical by construction — no zip/closure rebuild needed to validate; the first media run after landing double-checks the stats lines.

• ✅ C — –no-push OCI-layout stage handoff — DONE 2026-08-30. The last open half of the orchestrator-lifecycle item. Hook points:

  • cross-stage-build.sh — cross_local_handoff_enabled(), cross_ensure_local_context_workdir() (mints ${CROSS_CONTEXT_ROOT}/ cross-flow.XXXXXX once per run, age-based orphan sweep mirroring the runtime lane’s), cross_stage_context_dir(); the push=0 parent path in _cross_stage_run_resolve_parent appends --build-context <parent-tag>=oci-layout://<dir> when the parent’s index.json exists (i.e. the parent was BUILT THIS RUN); cross_stage_run exports every local stage to its layout after the build, and android additionally to <workdir>/android-artifacts/<arch> for the runtime lane.

  • build-cross-chain.sh — _chain_no_push_guard now ALLOWS full chains (from base) and refuses only mid-chain resumes; parse-time --no-push message updated; run_runtime_stage passes ARTIFACT_CONTEXT_ROOT + ARTIFACT_CONTEXT_MODE=oci to the runtime helper under --no-push (its existing runtime_use_local_artifact_context path consumes the android layout); _chain_on_exit reclaims the workdir.

  • Proven on the live host with a two-stage test build: stage A built + saved to a layout, stage B’s FROM localtest:stage-a with --build-context localtest:stage-a=oci-layout://… resolved from the LOCAL layout (--pull=false, marker file verified inside B). Unit-pinned by tests/test-cross-oci-handoff.sh (15 assertions, incl. mutation-style guard cases: handoff-off refuses, mid-chain refuses, FORCE escapes).

  • Why the guard still refuses mid-chain: --from-stage after base resuming a no-push chain has no locally-built parent to serve; the registry would be the stale-parent bug again. Documented in the usage text + guard.

  • Residual (documented, not done): a REAL multi-arch no-push chain has not been run end-to-end (validation remains on-demand; mechanism + guard are unit- and live-proven at the buildkit level).

• ✅ TG1/TG3 residuals — CLOSED 2026-08-30 (they were already implemented). The toolchain-closure trim (TG1) and the RUN-3/3d collapse (TG3) both landed 2026-08-24 — f485c132 (“TG1 bounded: lazy cmake/vulkan sourcing + trim their dead toolchain mounts”) and 202634c1 (“closure-window-3 wave”, which deleted RUN 3d and unified the superset build into RUN 3) — but were never CLOSED in the backlog. The validating rebuild started 2026-08-30 (local compiler build with GCC_PARALLEL_TARGETS=1) exercises both: the trimmed per-RUN mounts are what decide whether an unrelated script edit busts the GCC/LLVM layers, and the RUN-3 post-build gates (qemu llvm-config + CMake link smoke, materialize-llvm-target ELF assert) are the 3d-replacement verification.

• ⚠️ GCC_PARALLEL_TARGETS validation — IN PROGRESS 2026-08-30. First real launch of the flag on a compiler build command line (local, no push — so a failure costs only the build, never the public compiler tag; retry without the flag is the byte-identical sequential path). Also the first post-F2 toolchain build, so build-gcc.sh’s compiler_cache_launcher() call site is validated live. Log: out/build-logs/compiler-parallel.log.

• ✅ GCC_PARALLEL_TARGETS validation — DONE + PASS 2026-08-30. Local compiler build with the flag on the command line, both orchestrators. What the run proved, in order:

  1. The plumbing had been broken (the real “missed four times” cause). The flag was never forwarded into the build — no ARG in Dockerfile.toolchain, no –build-arg in the compiler-stage args — so a launch-time GCC_PARALLEL_TARGETS=1 silently did nothing and the sequential path won every validation. Fixed in 92fb9646 (ARG + ENV in Dockerfile.toolchain; append_optional_build_arg forwarding in stage-defs.sh’s compiler case; dry-run now emits --build-arg GCC_PARALLEL_TARGETS=1 when set, absent when not; pinned by test-stage-defs.sh).

  2. First parallel launch hit a real bug, found live: the concurrent per-target build-gcc.sh invocations each ran their own “Installing build dependencies…” apt_install and collided on /var/lib/apt/lists/lock (“E: Could not get lock … held by process N”). Fixed in 5e8b2470: GCC_SKIP_BUILD_DEPS=1 gates build-gcc.sh’s apt step (deps are already installed by build_host_gcc, which runs first) and the parallel driver exports it after the serial pre-pass. Sequential path unchanged. (The failure was also masked by a bash ERR-trap dynamic-scoping bug in logging.sh _install_trap — action: unbound variable replaced the real apt error; logged as a follow-up.)

  3. The parallel path WORKS on the real build: GCC_PARALLEL_TARGETS=1: serial apt pre-pass, then concurrent target builds → amd64 linked serially (symlink-only, as designed) → arm64 + riscv64 built concurrently (JOBS=16 each on this host’s 32 cores): [parallel] arm64: OK at 745.9s, [parallel] riscv64: OK at 823.8s. Both cross-GCC targets done in ~531s wall (launch 292.7 → riscv64 done 823.8) vs ~984s sequential for the two — the documented ~30% GCC-RUN saving.

  4. Full toolchain GREEN: LLVM + Rust + Python + bundle + toolchain smoke all passed — 41 PASS lines, 0 failures, image cross-compiler-amd64 loaded locally. This ALSO validates TG1/TG3 (the trimmed per-RUN mounts) and F2’s toolchain call sites (sccache gcc/ sccache g++ live in the GCC compile lines). The flag is now a real, working knob: GCC_PARALLEL_TARGETS=1 on either orchestrator’s command line. Not pushed (local-only validation); the next full chain that wants the parallel GCC must pass the flag itself.

• ✅ F2 media validation — DONE + PASS 2026-08-30 (sdk→media→android, amd64, pushed). The one-resolver cache consolidation ran in every media RUN (compiler cache enabled: launcher=/opt/scripts/core/sccache-launcher.sh, 100 % C/C++ hit rate), 27 artifact verifies OK, android pushed. modules.sh reorder + the QNN-off fan-out path (litert/tvm/app-wheelhouse/genai with no zip) exercised with no regression.

• ✅ sccache-launcher server-death gap — FOUND LIVE + FIXED (0371d164). The F2 validation caught a second failure class the guarded launcher did not handle: the sccache SERVER died mid-build under full concurrent-media load (sccache: error: failed to execute compile / caused by: Failed to send data to or receive data from server / failed to fill whole buffer). The launcher only bypassed on sccache: encountered fatal error (the TryCompile ENOENT class), so it handed the dead-server error to ninja as a REAL failure and killed the TVM step. Fixed by widening the bypass classification to any sccache-prefixed internal error (sccache: (encountered fatal error|error:| caused by:)) — safe because sccache prefixes only its own failures with sccache:; a real compiler error is echoed un-prefixed and passes through. Pinned by tests/test-sccache-launcher.sh (8 assertions incl. a mutation case proving the old narrow match would NOT have bypassed). Validated live on the follow-up sdk→media→android rebuild: TVM build OK, apache_tvm + apache_tvm_ffi wheels produced, 0 server-death propagations (858 sccache-internal failures caught + bypassed — “a hiccup costs hits, not a build”). Both chains pushed and complete.

• ✅ QNN-LINUX A2 items 1-6 — DONE + PROVEN 2026-08-30 (drop dir + .gitignore, versions.env QNN_SDK_LINUX_ZIP_SHA256 pin, the shared 01-core/qnn-sdk.sh resolve/stage helper, ORT wiring on real SDK v2.49.0.260730, Dockerfile.media mounts, artifact verification) and item 7 framework fan-out WIRED (GenAI/LiteRT/TVM/IREE, every flag mirroring Windows #121). The fan-out VALIDATION remains blocked on the login-gated SDK re-stage — see the open backlog A2.