Build watch list — the 2026-09-05 wave, read against a running chain

Build watch list — the 2026-09-05 wave, read against a running chain#

Follow this while the chain runs. Every row names the exact line that means the change worked and the exact line that means it did not. Grouped by stage, in the order a chain reaches them, so it can be read top-to-bottom live.

The wave that produced this list closed eleven backlog entries on static proof only. A first rebuild attempt already found two build-killing bugs that a full green battery had passed (HEAD e109f5ad), so treat every “expected” below as a hypothesis the chain is testing, not a fact.

Chain in flight when this page was written: chain-status.json run 20260905-120554-7b7a0d4e, sdk..runtime, amd64,arm64,riscv64.

Re-groomed 2026-09-07 for the Vulkan rows only. That chain finished; the wave after it wired all four components this page had listed as known or expected failures. Those rows now say the opposite of what they said, because an unavailable line for any of the four is a regression. VK2 closed on 2026-09-09 with 20/20 components on both foreign arches (archive). The rest of the page is the 2026-09-05 text, with log strings kept matching the code.

Read this first: the three that abort a run#

If the chain dies early, it is almost certainly one of these. Each fails fast, at its own stage, rather than shipping — which is the design, but it is still a dead run.

#

grep for

stage

what it means

1

media_compiler_launcher: command not found

media

the image’s 03-media/core/common.sh is an older layer than build-ffmpeg.sh/build-pyav.sh, which now call the new helper unqualified under set -e. A cache-layering problem, not a wiring one — all three files ship in the same commit

2

any path printed after still holds link(s) that resolve to nothing

sdk (all 3 arches)

_llvm_target_repair_links ends in find <prefix> -xtype l and exits 1 on any survivor. It has never run on any arch. The likely cause is a materialised directory carrying a broken link of its own — the one path the loop cannot reach

3

ERROR from prune-vulkan-host-sdk.sh

package (artifact-source)

a NEW RUN with two bind mounts. Fails before any COPY, so it costs minutes, not hours

SDK stage#

Vulkan: the source tree dies where it is born (HT5)#

verdict

line

PASS

Vulkan cross-targets <arch>: N/N component(s) built immediately followed by Pruning the Vulkan SDK build tree at /opt/vulkan/<ver>/source

FAIL

the prune line appearing before the component verdict — that order deletes the loader/SPIRV-Tools/glslang sources out from under the build

FAIL

no prune line at all on arm64/riscv64 — the foreign lanes then carry 4.09 GB of ./vulkansdk build tree into every downstream image

n/a

amd64 prints neither; it never takes the cross path and has no source/

Vulkan: fifteen components cross-built for the first time (VK1)#

The arm64 lane of this run has already answered this, and the answer is backlog VK2: eleven of fifteen built, including the three that matter for building an application — glslc, vulkaninfo and the validation layers. Use that as the baseline for the riscv64 lane rather than the pre-run guesses.

verdict

line

PASS (measured on arm64)

Cross-building <label> for <arch> with no following unavailable line, for eleven of: vulkan-headers, spirv-headers, vulkan-utility-libraries, volk, vma, spirv-cross, spirv-reflect, shaderc, vulkan-tools, vulkan-extensionlayer, vulkan-validationlayers

FAIL (was KNOWN FAIL before VK2)

vulkan-profiles unavailable — jsoncpp and valijson are rows of their own in _VK_TARGET_COMPONENTS since c59799b4, ordered ahead of it, so find_package(valijson) has somewhere to look

FAIL (was KNOWN FAIL before VK2)

gfxreconstruct unavailable — and the old diagnosis on this row was wrong. The :${arch} packages were already unpacked; find_library could not see them because nothing passed CMAKE_LIBRARY_ARCHITECTURE, which _cross_build_sdk_component now does for every row. The genuinely missing half was GL, added to install_optional_target_packages

FAIL (was EXPECTED FAIL before VK2)

slang unavailable — and NOT for the reason this row gave. It is not host LLVM tblgen; slang cross-compiles its own generators and then cannot run them, so SLANG_GENERATORS_PATH points at the host ./vulkansdk run’s copies. tblgen is the dx* family’s problem; dxc has been a row since VK4 (2026-09-09), fed the host run’s llvm-tblgen the same way (host tools)

FAIL (was EXPECTED FAIL before VK2)

vulkancapsviewer unavailable — qt6-base-dev:${arch} is in the optional set with QT_HOST_PATH=/usr and a sysroot CMAKE_PREFIX_PATH

FAIL

shaderc: source missing at …/shaderc/src; skipping — the checkout IS one level down and _vulkan_target_src is supposed to find it. It built on arm64, so this would be a regression

FAIL

any Cross-building line naming a -B TMP/<label> path for a component whose source is absent — the missing-source guard has stopped skipping

Do not “fix” a Vulkan row while the chain is running. That was the 2026-09-05 instruction and it still holds: editing vulkan.sh mid-run ships two arches from different sources, which is the drift this repo has been bitten by before. The four routes above are already in the tree; what the chain owes is a verdict on them, and <arch>/bin carrying what x86_64/bin does is what closed VK2 (2026-09-09).

LLVM target prefix (HT4)#

verdict

line

PASS

the stage completes with no output at all from _llvm_target_repair_links on arm64/riscv64 (both measured 0 dangling today, so any output there is news)

PASS

amd64 prints amd64 /opt/llvm-target NEEDED walk clean and not is NOT self-contained

FAIL

still holds link(s) that resolve to nothing plus a path list — see #2 above

FAIL

the walk aborting on an unresolved liblldb soname. liblldb* joined the LLVM family this wave, so an apt set that ships the lldb binaries without the lib now stops the stage where it used to fall through

sccache — the YB verdict, and it is finally greppable (YB)#

The launcher used to print its [server=] field only when sccache failed, so a run where the fix worked carried no evidence. Both address setters now print it on the healthy path, identically spelled. One grep covers the whole chain:

grep -o '\[server=[^]]*\]' <log> | sort | uniq -c

verdict

line

PASS

every occurrence reads [server=/tmp/sccache-<uid>.sock]

FAIL

a single [server=tcp:4226] anywhere — the regression, unfixed

context

the two lines are [INFO] Using sccache with SCCACHE_DIR=… (cap …) [server=…] (01-core/common.sh) and [INFO] sccache enabled: SCCACHE_DIR=…, CACHE_SIZE=… [server=…] (01-core/compiler-cache.sh)

Needs a compile-heavy chain. --only runtime emits none of these lines. Also worth reading deliberately: sccache --show-stats at the START of each step should report 0 compile requests, not another container’s hundreds.

Media stage#

what

PASS

FAIL

the ffmpeg/pyav launcher owner (F3)

Using sccache / a --cc=<launcher> in the ffmpeg configure line, and a non-zero sccache hit count on the second arch

media_compiler_launcher: command not found — see #1 above

TFLite for the GStreamer monorepo (CL6)

the monorepo configures with TFLite found on arm64 AND riscv64. riscv64 is the one to watch, as always

TFLite missing on either. First thing to check is whether the parent still calls all three helpers in order — _gst_tflite_probe_flags, _gst_tflite_fix_pc, _gst_tflite_symlink_for_meson

the media stage’s cross assertion (CL7.3)

nothing new printed

media_common_init: critical module(s) did not load — the 03-media fallback clone was deleted, so a caller without cross-env now stops here instead of limping

Package / runtime stage#

The Vulkan host prefix (HT5)#

verdict

line

PASS

the artifact-source prune runs and exits 0 before the /opt/vulkan COPY

PASS, deliberately

on amd64 the prune is a no-op — there x86_64 IS the prefix the image runs

PASS, deliberately

on a lane whose cross Vulkan loader failed to build, the prefix is KEPT and the script says so loudly. The tree-arch gate will then fail that image on 61 X86-64 objects. That is the honest verdict for an image with no Vulkan of its own, and it is a new way for a chain to go red

FAIL

the script deleting anything on amd64 — that takes VULKAN_SDK, the loader and every host tool with it

The host payload (CON96, new 2026-10-13)#

verdict

line

PASS

amd64: host-payload: HOST_PAYLOAD=on for amd64; the Android SDK, its payloads and the Rust toolchain are copied as built

PASS

arm64, riscv64: host-payload: emptying /usr/local/rustup and /usr/local/cargo but its registry (… -- <arch> runs <triple>, …), then host-payload: emptying /opt/android-sdk and /opt/android (HOST_PAYLOAD=off for <arch>); later in the package RUN, Rust toolchain in /usr/local/rustup is not <triple>: -- reinstalling natively

FAIL

any host-payload: emptying on amd64 — the Android lane loses its SDK; the smoke’s check_android_abi fails such an image too

FAIL

the smoke’s BAD android-home … says the image ships no SDK: the ENV and the pruned trees disagree

on the bytes

nerdctl run --platform linux/arm64 <image> sh -c 'du -sh /opt/android-sdk /opt/android; echo "${ANDROID_HOME:-empty}"' prints 4.0K, about 8.0K (the .android-payload-off marker, plus .android-emulator-off where the emulator installer wrote one) and empty; the arm64 child’s compressed size falls by about 3.4 GB against 10.11 GB

absl, the defect the corrected probe found (CL1)#

verdict

line

FAIL

copy-media-payloads: optional payload missing: /artifact-src/usr/local/include/absl on any arch — the artifact image did not carry absl and the fix is inert there

PASS

no such line, on all three

Packaging (new this wave, none of it ever run)#

what

PASS

FAIL

AppImage runtime

Staged AppImage runtime-<arch> (<n> bytes) from …, once per arch

no squashfs superblock in <tool>; runtime-<arch> not staged — a WARN, non-fatal, but the runtime is not staged and consumers go back to fetching it from GitHub

Flatpak runtimes

seven refs installed on amd64/arm64; skipped outright on riscv64 (Flathub builds x86_64 and aarch64 only)

a 404 retry loop on riscv64 means the arch guard stopped working

web-lane toolchain

OK: nightly-<date> installed with rust-src + wasm32-unknown-unknown (the RUST_NIGHTLY_TOOLCHAIN pin) and two OK: <crate> <version> installed

WARN: nightly-<date> is unavailable — non-fatal, but the web lane then auto-installs a nightly per consumer run, which is the cost this exists to remove

web-lane tools from source (riscv64, new 2026-09-23)

android web-lane-tools: OK: web-lane producer: <crate> <version> built for riscv64gc-unknown-linux-gnu in <n> s, or cache HIT; amd64/arm64 android: skipped: not in WEB_LANE_TOOLS_CROSS_ARCHES / native-build-platform. riscv64 package: installed from the cross-built artifact, twice

WARN: web-lane <crate>: … the cross fast path was not taken — non-fatal, but that RUN pays the QEMU compile (~44 min) once per key; the line carries the producer’s reason. Any ERROR: web-lane stops the package stage on purpose (contract)

On the shipped bytes — after the run, not from the log#

This is the half the 2026-09-04 chain never reached, and the half where “verify the shipped BYTES, never the push” applies.

Sizes (CC1 has asked for this baseline twice)#

Today’s reading is 30.37 / 30.84 / 29.69 GB (amd64/arm64/riscv64), verified against nerdctl images. HT5 removes ~5.95 GB from each foreign image; VK1 adds an unknown amount back by cross-building fifteen more components. Read the two together or neither number means anything. Record all three.

| expected | /opt/vulkan ~60 MB on arm64 and ~88 MB on riscv64 instead of 6.0 GB; amd64 byte-identical; the cross-android-* artifact images ~3.9 GB lighter on the two foreign lanes |

Probes to run, read-only#

# HT4 — was 12 / 0 / 0
find /usr/local/llvm-target -xtype l                      # must be 0 on all three
/usr/local/llvm-target/bin/lldb --version                 # works (fails today)
python -c 'import lldb'                                   # imports (fails today)
readlink /usr/local/llvm-target/lib/libclang.so           # = libclang-23.so.23
test -f /usr/local/llvm-target/include/llvm/IR/IRBuilder.h
ls /usr/local/llvm-target/lib/libc++*                     # must be EMPTY
# the unstartable-binary count: 3-of-142 -> 0-of-142

# HT5 — the gate was SHARPENED, not loosened
#   check_manifest_tree_arch now walks the WHOLE /opt/vulkan tree
#   expect: "OK /opt/vulkan: N ELF object(s), all AArch64"  (all X86-64 on amd64)
#   a FAIL naming 1 317 X86-64 objects means the prune did not run
#   check_vulkan_loader now asserts WHERE the loader loaded from:
#     "loads from /opt/vulkan/<ver>/<arch>/lib/..."     = PASS
#     a FAIL naming /usr/lib/<triple>                   = the prune took the prefix
#                                                         the image actually runs
#     a WARN naming no path                             = /proc/self/maps unreadable,
#                                                         not a defect

# CC1 — a REGRESSION watch, not a discovery: all three read 0 today
find /usr/local/rustup /usr/local/cargo ! -user 1001      # must stay 0
find /opt/flutter -user root                              # must stay 0
nerdctl image inspect … | jq .Config.Env                  # the ENV strings must be BAKED

# CL1 / absl — 1 FAIL today, must be 0
nerdctl run --rm --platform linux/<arch> -v <repo>:/repo:ro --entrypoint bash \
  ghcr.io/kataglyphis/kataglyphis_beschleuniger:latest-<arch> \
  /repo/linux/scripts/06-packaging/smoke-critical-fixes.sh
#   expect: PASS absl/types/span.h found in /usr/local/include

# AB1 — 420 objects, every one X86-64, on today's bytes
#   check_android_abi must report the payload matching the advertised ABI,
#   reading .a MEMBERS as well as .so files

What only the runtime smoke can say#

check_consumer_contract asserts 9 rows and has only ever run on amd64 — the 2026-09-04 chain died before arm64 and riscv64. Same for the advertised-key gate’s 16 OKs with zero UNSET/UNREAD. Two green foreign verdicts here are new information.

Known-good noise — do not chase these#

  • VK_LAYER_PATH is empty in every running image and always has been. The entrypoint sources LunarG’s setup-env.sh, which UNSETS it and exports VK_ADD_LAYER_PATH instead. Not touched this wave; backlog R1 (closed 2026-09-07, archive) fixed the value a consumer that does not source it gets (vulkan-foreign-arch-sdk.md).

  • The lib-dynload audit WARNs for five optional modules (_zstd, readline, _curses, _uuid, _decimal). Information on optional rows — but any of them on amd64 is a genuine finding. readline is a required row in 01-core/cpython-dev-packages.sh now, so its WARN is a finding on every arch. _ctypes warns on the GIL cross arches by design (build_python.sh sets ac_cv_header_ffi_h=no there). The free-threaded cross trees build it, and a missing one fails their audit.

  • slang and vulkancapsviewer failing to cross-configure was expected until 2026-09-07 and is not any more — both have a route now, so either one failing is a finding, not noise.