Linux Cross Builds

See also: docs/linux-build-basics.md for build fundamentals, caching, and troubleshooting. AGENTS.md for agent guardrails and the full repo map.

Build-time download speed: the cross-compiler/SDK builds fetch the LLVM source with git inside a RUN step. On this host that is fast because rootless BuildKit runs with --oci-worker-net=host (host networking for RUN steps). Registry mirrors do not help that git fetch; the host-net setting does. The drop-in is linux/host-config/buildkit.service-override.conf, installed by linux/host-config/apply-host-config.sh (AGENTS.md § Validation: Linux host config is code); docs/project-info.md explains it. For repeated LLVM rebuilds, prefer caching the source on the host over re-fetching.

Build logging: build-cross-chain.sh and build-cross-stage.sh accept --log-dir ./out/build-logs to write per-stage build logs. The other orchestrators (build-cross-compiler.sh, build-runtime-manifest.sh, build-runtime-artifacts.sh) do not — capture their output, and any manual nerdctl build, with 2>&1 | tee ./out/build-logs/<name>.log. The standard location for build logs is out/build-logs/.

Cross-Compiler builder (nerdctl, amd64 host; amd64/arm64/riscv64 targets)

The existing multi-platform build stays unchanged. Treat it as the compatibility lane for the current QEMU/binfmt-based end-to-end build.

The cross-compiler path below is additive. It does not replace the existing QEMU workflow. Instead, it prepares a single amd64-hosted builder image that contains cross toolchains for amd64, arm64, and riscv64 for the artifact-based multi-architecture chain (build-cross-chain.sh), which is what publishes :latest.

This lane intentionally builds only a linux/amd64 container image. The three architectures are the compiler targets installed inside that image via CROSS_TARGETS=amd64,arm64,riscv64, not three separate compiler container manifests. This image is a single amd64 builder image: the chain’s compiler stage. It keeps the current native/emulated flow intact while adding source-built GCC 16 target compilers like x86_64-linux-gnu-gcc, aarch64-linux-gnu-gcc, and riscv64-linux-gnu-gcc, plus convenience wrappers such as clang-amd64, clang-arm64, and clang-riscv64 for host-side cross builds.

For the cross-compiler path, the helper can bootstrap the base image locally when needed, so you do not have to rely on a remote base intermediate tag surviving in GHCR.

Fastest entry point:

./linux/scripts/build-cross-compiler.sh --cross-targets amd64,arm64,riscv64 --fast-ubuntu-mirror \
  --fast-ubuntu-mirror-url http://de.archive.ubuntu.com/ubuntu/

Use --fast-ubuntu-mirror-url URL to override the default mirror (https://archive.ubuntu.com/ubuntu/). For example: --fast-ubuntu-mirror-url http://de.archive.ubuntu.com/ubuntu/.

The helper script only uses nerdctl. It builds base and then the compiler image through cross_stage_run(), the function the orchestrator uses; BuildKit’s cache, not a registry pull, is what makes an unchanged base cheap. It only pushes when you pass --push. Internally the script delegates to the shared stage graph (stage-defs.sh) and build helpers — the same infrastructure used by the full orchestrator. The --image-repo flag switches the registry prefix; there are no legacy env var overrides.

If you only need the downstream SDK or media cross stages, build them with build-cross-stage.sh (§ The command reference AGENTS.md used to carry); it resolves the published compiler from the registry itself. A nerdctl pull beforehand does not help: it lands in containerd’s store, which this host’s BuildKit OCI worker does not read (§ --no-push full chains below).

Build the local amd64 base image:

LOG_DIR="logs/$(date -u +'%Y%m%dT%H%M%SZ')-cross-base"
mkdir -p "${LOG_DIR}"

nerdctl build --platform linux/amd64 -t ghcr.io/kataglyphis/kataglyphis_beschleuniger:base \
  --output 'type=image,name=ghcr.io/kataglyphis/kataglyphis_beschleuniger:base,push=true' \
  -f linux/Dockerfile.base \
  --build-arg USE_FAST_UBUNTU_MIRROR=true \
  --build-arg FAST_UBUNTU_MIRROR_URL=http://de.archive.ubuntu.com/ubuntu/ \
  . 2>&1 | tee "${LOG_DIR}/base.log"

Then build the dedicated amd64-hosted compiler image in cross mode for amd64, arm64, and riscv64 targets:

LOG_DIR="logs/$(date -u +'%Y%m%dT%H%M%SZ')-cross-compiler"
mkdir -p "${LOG_DIR}"

nerdctl build --platform linux/amd64 -t ghcr.io/kataglyphis/kataglyphis_beschleuniger:cross-compiler-amd64 \
  --output 'type=image,name=ghcr.io/kataglyphis/kataglyphis_beschleuniger:cross-compiler-amd64,push=true' \
  -f linux/Dockerfile.toolchain \
  --build-arg BASE_IMAGE=ghcr.io/kataglyphis/kataglyphis_beschleuniger:base \
  --build-arg USE_FAST_UBUNTU_MIRROR=true \
  --build-arg FAST_UBUNTU_MIRROR_URL=http://de.archive.ubuntu.com/ubuntu/ \
  --build-arg BUILD_MODE=cross \
  --build-arg CROSS_TARGETS=amd64,arm64,riscv64 \
  . 2>&1 | tee "${LOG_DIR}/cross-compiler-amd64.log"

The explicit nerdctl build --output ... push=true commands above already push the intermediary images to GHCR. Only the helper script keeps the images local by default unless you pass --push.

Expected compiler result inside that image:

• gcc and g++ resolve to /opt/gcc-16.2.0/bin/* and report GCC 16.x on the amd64 host compiler path.

• x86_64-linux-gnu-gcc, aarch64-linux-gnu-gcc, and riscv64-linux-gnu-gcc resolve to /opt/gcc-16.2.0/bin/* and report GCC 16.x.

• clang-amd64, clang-arm64, and clang-riscv64 still exist, but now point Clang at /opt/gcc-16.2.0 as the GCC toolchain root.

Expected result: the build log ends with ghcr.io/kataglyphis/kataglyphis_beschleuniger:cross-compiler-amd64. That is correct for this cross lane because the builder container itself runs on amd64 while shipping source-built GCC 16 host and cross compilers for all three target architectures.

Or let the helper do the push too:

./linux/scripts/build-cross-compiler.sh --cross-targets amd64,arm64,riscv64 --fast-ubuntu-mirror --push

Recommended: digest-pinned orchestrator (build-cross-chain.sh)

For a hands-off, agent-proof end-to-end cross build, prefer the orchestrator (the canonical command is in § The command reference AGENTS.md used to carry). It runs base -> compiler -> sdk -> media -> android -> runtime and, after each cross stage is pushed, captures that stage’s registry-resolvable manifest digest and feeds it to the next stage as --build-arg BASE_IMAGE=<repo>@sha256:<digest>. The final runtime stage delegates to build-runtime-manifest.sh to build the per-arch base -> package -> torch -> wrapper images on the real target platform and publish the multi-arch :latest manifest.

The cross-lane stage chain is defined declaratively in linux/scripts/01-core/stage-defs.sh. Each stage entry (Dockerfile, parent stage, tag function, per-arch flag) is defined in CROSS_STAGE_ORDER so both the build loop and --verify-chain consume the same single source of truth. To add or reorder stages, update CROSS_STAGE_ORDER in that file.

Stage graph management functions (stage-defs.sh)

stage-defs.sh now provides self-contained stage graph management:

• cross_stage_init_pins() — Declares all digest-pin variables (scalar for shared stages, associative arrays for per-arch stages) derived from the stage graph. Call once before entering the build loop. Previously the orchestrator had to manually declare BASE_PIN, COMPILER_PIN, SDK_PIN, MEDIA_PIN, ANDROID_PIN, and ANDROID_BUILT_THIS_RUN — adding a stage required touching both files. Now the graph is the single source of truth for pin variables too.

• cross_stage_validate_graph() — Runs before every build to check internal consistency: parent references resolve to valid stages, tags produce non-empty results, no dependency cycles exist. Hard-fails if the graph is inconsistent, catching configuration errors early.

• cross_stage_ensure_parent_available() — For the runtime handoff: ensures the parent stage images (e.g. cross-android-<arch>) are locally available before delegating to build-runtime-manifest.sh. Images built in the current run are already local; images from prior builds are pulled from the registry. Replaces the old _refresh_android_images() with graph-driven resolution.

Stage build/pin orchestration functions are shared in linux/scripts/01-core/cross-stage-build.sh (sourced via artifact-common.sh). These functions (cross_stage_run(), cross_stage_build_and_push(), cross_stage_build_local(), cross_stage_resolve_parent_pin(), resolve_pin()) are used by both the orchestrator and the standalone build-cross-stage.sh helper. cross_stage_build_local() handles local-only (non-push) builds, while cross_stage_build_and_push() handles registry-pushed builds with cache support.

cross_stage_run() is the shared entry point for all stage builds. It accepts a push flag (3rd argument, default 1):

• push=1: resolves the parent via digest-pinned reference, pushes the stage image to the registry, and captures the digest pin for downstream stages.

• push=0: resolves the parent via mutable tag, builds locally only, does not push or pin.

Both the orchestrator (push=1 for all stages), the single-stage builder (build-cross-stage.sh), and the standalone compiler (build-cross-compiler.sh) use this same function, eliminating duplicated build/pin logic across scripts.

The runtime helpers share initialization logic via linux/scripts/01-core/runtime-flow-common.sh, which provides init_runtime_flow_defaults(). Both build-runtime-artifacts.sh and build-runtime-manifest.sh reach it by sourcing lib-orchestrator.sh, which loads it inside runtime_flow_preamble().

Standalone single-stage rebuild commands: § The command reference AGENTS.md used to carry.

--no-push full chains: FIXED 2026-08-30 via local OCI-layout handoff

History worth keeping: on this host builds run on BuildKit’s OCI worker, which has its own image store. nerdctl build -t loads results into containerd’s store — which the next build’s FROM never consults. The mutable parent tag resolved against the registry, so every downstream stage of a --no-push chain silently built on the last PUSHED parent, not the one just built (two full runs were lost to sdk stages compiled on a months-old compiler before the digest trail exposed it — the freshly built compiler had /opt/gcc-16.2.0, the sdk image it “inherited from” had /opt/gcc-16.1.0).

The fix (cross-stage-build.sh, 2026-08-30), mirroring the runtime lane’s proven mechanism: on --no-push chains every stage built locally is exported to an OCI layout (export_image_to_oci_layout from context-management.sh) at ${CROSS_CONTEXT_ROOT:-~/.cache/opencode/cross-stage-contexts}/cross-flow.*/<stage>-<arch>, and each child’s parent resolution appends --build-context <parent-tag>=oci-layout://<dir> so its FROM resolves to the image THIS RUN built. The android image is additionally exported to <workdir>/android-artifacts-<arch> and handed to the runtime lane as ARTIFACT_CONTEXT_ROOT (mode oci), so the no-push package build copies from the locally-built android, not the registry. The workdir is minted once per run and reclaimed on exit (age-based sweep for killed runs). The multi-stage refusal guard now allows full chains (from base) and only refuses mid-chain resumes (--from-stage after base), which cannot serve the missing prefix. Opt-outs: CROSS_LOCAL_CONTEXT_HANDOFF=0 (reverts to the refusal) and CROSS_NO_PUSH_FORCE=1. Mechanism proven on the live host 2026-08-30 with a two-stage test build: stage B’s FROM resolved from the exported layout (--pull=false, content marker verified), never the registry. CROSS_CONTEXT_ROOT is free space ~= the built stages’ aggregate size (media + android are multi-GB) — transient, deleted at chain exit.

The flow that is correct today: push mode, manifest last

# Build and push every cross stage with digest-pinned handoffs; stop before the
# runtime stage if you do not want to touch the published manifest yet:
bash linux/scripts/build-cross-chain.sh --target-arches amd64 \
  --to-stage android --log-dir ./out/build-logs

# Runtime lane: build+push+smoke the per-arch wrappers WITHOUT recreating the
# multi-arch :latest manifest (so a single-arch run cannot clobber it):
bash linux/scripts/build-runtime-manifest.sh \
  --image ghcr.io/kataglyphis/kataglyphis_beschleuniger:latest \
  --target-arches amd64 \
  --artifact-image-prefix ghcr.io/kataglyphis/kataglyphis_beschleuniger:cross-android \
  --artifact-build-mode cross --push --skip-manifest

# When ALL arches' wrappers exist, publish the manifest in one shot:
bash linux/scripts/build-runtime-manifest.sh --image ...:latest \
  --target-arches amd64,arm64,riscv64 --manifest-only

Push mode is also what arms the ancestry annotations — the machine-checked stale-ancestor guard exists only for pushed stages.

Opt-in: concurrent per-target GCC builds

GCC_PARALLEL_TARGETS=1 builds the arm64/riscv64 cross+Canadian GCCs concurrently inside the compiler stage (~30 % off that RUN at 3 targets). The driver runs a serial apt pre-pass first (dpkg lock), divides JOBS across the concurrent builds, and writes per-target logs (replayed on failure). Default 0 = the sequential flow, byte-identical. The host GCC always builds first (alternatives registration), and the build-arch target is symlink-only. See gcc.sh::_gcc_build_cross_targets_parallel.

Stale-check (--verify-chain and verify-cross-chain.sh)

Before a full build, verify whether downstream registry images are stale without performing any builds. The verification logic is shared via linux/scripts/01-core/chain-verify.sh (sourced by both entry points). The chain verification commands are in § The command reference AGENTS.md used to carry. Both resolve all upstream registry digests and report mismatches so you can decide whether a full rebuild is needed. The standalone script is useful for quick checks without loading the full orchestrator.

Describe chain (--describe-chain)

Print the full stage graph with tag names and parent chains without building:

# Via the orchestrator:
./linux/scripts/build-cross-chain.sh --describe-chain --target-arches amd64,arm64,riscv64 --log-dir ./out/build-logs

# Via the standalone verifier:
bash linux/scripts/verify-cross-chain.sh --describe-chain --target-arches amd64,arm64,riscv64

This shows which Dockerfile produces each stage, the expected tags per architecture, and the parent→child relationships.

Why the handoff must be pinned by digest

Each cross stage is a separate nerdctl build whose next stage does FROM ${BASE_IMAGE}. If BASE_IMAGE is a mutable tag such as :cross-media-arm64, the downstream build can silently consume a stale, locally-cached image instead of the one you just built, because:

• --output type=image,name=...,push=true pushes the new digest to the registry but does not reliably refresh the local containerd tag to that digest, and

• BuildKit’s default FROM resolution prefers an image already present locally (it does not re-pull unless told to).

So a rebuild of media followed by a build of android can quietly use the old media. Two defenses, in order of strength:

1. Digest pinning (used by build-cross-chain.sh): content-addressed repo@sha256:... references can never resolve to a stale image. This is the robust fix and the path you should instruct automated agents to use.

2. --pull=true on every downstream stage (used in the manual loops below): forces BuildKit to re-pull the freshly pushed base tag from the registry. Lighter, but still relies on mutable tags.

Do not capture the pin from the local image store’s RepoDigests. On this host BuildKit pushes a converted docker.v2+json manifest whose digest differs from the local OCI manifest, so RepoDigests is not registry-resolvable. Use nerdctl manifest inspect --verbose <tag> → .Descriptor.digest (this is exactly what the registry_pin_ref helper in linux/scripts/01-core/digest-pinning.sh does — sourced by artifact-common.sh).

Partial runs are supported, e.g. resume from media for one arch:

./linux/scripts/build-cross-chain.sh --target-arches arm64 --from-stage media --to-stage android --log-dir ./out/build-logs

When resuming mid-chain, the required upstream digest is resolved from the parent stage’s current registry tag automatically.

The handoff rules, as AGENTS.md carried them

Moved out of AGENTS.md on 2026-09-15 (owner decision D10), unedited except for this heading and the relative links. The RULES stayed there; this is the reference behind them.

The cross lane is a sequence of separate nerdctl build invocations where each stage does FROM ${BASE_IMAGE}: base → compiler → sdk → media → android → package → torch → wrapper → manifest. The base-image handoff MUST NOT rely on a bare mutable tag, or a stage can silently consume a STALE locally-cached image.

--output type=image,name=... does not reliably refresh the local containerd tag; BuildKit’s default FROM prefers an already-present local image. So rebuilding media then building android can quietly reuse the old media. This is not just a FROM hazard — it broke the runtime wrapper’s OWN tag. On this rootless nerdctl host, nerdctl build --output type=image,name=X creates NO local tag X at all (verify: --output type=image,name=X → X absent from nerdctl images; -t X → present). The runtime lane used the annotated --output type=image,name=<tag>,annotation.* exporter to tag the wrapper, so the freshly built wrapper was invisible and nerdctl push <tag> + nerdctl manifest create <tag> resolved the STALE pre-existing tag — shipping :latest-cross byte-identical five times (2026-08-14/15, amd64 frozen at 35c1f1df). FIXED: runtime-build-fns.sh append_runtime_image_output now uses plain -t on both paths (reliably creates AND overwrites the tag). Do NOT reintroduce the --output type=image,name= exporter for a tag you then push or index — use -t. (The dropped ancestry annotations never reached the registry anyway; they are re-embedded as config labels via ancestry.sh’s --label provenance args.)

Rules:

0. The chain now checks this for you. A run starting after base asserts the recorded ancestry of every stage it inherits (01-core/ancestry.sh) and refuses to build on a parent that was re-pushed after its child. Rules 1-4 below describe what that check enforces and what its failure message asks you to do — they are no longer yours alone to remember. Bypass only deliberately, with --no-verify-ancestry / CROSS_VERIFY_ANCESTRY=0. Images built before this mechanism carry no annotation and only warn, so a chain that predates it still needs rules 1-4 applied by hand until each stage has been rebuilt once.

1. When ANY base image in the registry tag hierarchy is replaced, rebuild every downstream image from the replaced stage, OR verify the downstream images already contain the new content (e.g. check /opt/gcc-16.2.0-native-arm64 exists in the pinned sdk digest).

2. --from-stage only controls where execution starts; it does NOT update the base image of the first stage. If the previous stage’s tag was built from a stale upstream, your rebuild inherits that staleness.

3. After pushing a rebuilt compiler image, run from --from-stage sdk (not media) so the sdk is built from the new compiler.

4. Do NOT use --from-stage android unless you verified the media tag already contains the compiler’s content (e.g. native GCC directories).

5. Prefer linux/scripts/build-cross-chain.sh — it captures each stage’s registry digest after push and feeds it to the next as --build-arg BASE_IMAGE=<repo>@sha256:<digest>, making stale reuse structurally impossible. Supports --target-arches, --from-stage, --to-stage, --only.

6. When driving manual nerdctl loops, pass --pull=true on every stage that consumes a BASE_IMAGE tag (weaker defense; digest pinning preferred).

7. Capture pinnable digests with nerdctl manifest inspect --verbose <tag> → .Descriptor.digest (the registry_pin_ref helper in 01-core/digest-pinning.sh). Do NOT use RepoDigests: BuildKit pushes a converted docker.v2+json manifest whose digest differs from the local OCI manifest and is not registry-resolvable.

Trap: stale-base propagation across orchestrator invocations

Digest pinning only prevents drift within a single orchestration run. Across separate invocations, a registry tag may have been updated — but downstream images still pin the old digest. For example:

1. You build the full chain (compiler → sdk → media → android).

2. You rebuild and re-push the compiler image (e.g. adding /opt/gcc-16.2.0-native-arm64).

3. You run --from-stage media --to-stage android. The orchestrator resolves the sdk pin from the registry tag cross-sdk-arm64 — which was built from the old compiler and is missing the new content.

4. The new media rebuild inherits from the stale sdk, and the new compiler content never reaches the final image.

This is now checked automatically. linux/scripts/01-core/ancestry.sh turns the rule below into a machine-enforced invariant, so a partial run can no longer silently build on a stale ancestor:

• Write: every pushed cross stage records the digest-pinned reference it was actually built FROM as an OCI manifest annotation (org.kataglyphis.parent-digest, plus org.kataglyphis.parent-stage). This is free — it rides along in the manifest the push already writes.

• Read: nerdctl manifest inspect --verbose returns the verbatim registry manifest in its base64 Raw field, so the annotation is readable without pulling the image or fetching config blobs (01-core/manifest-annotation.py).

• Assert: before a run with --from-stage after base, the orchestrator walks the ancestor chain feeding that stage. For each child → parent link the digest the child records must equal the digest the parent tag currently resolves to. A mismatch aborts the run before anything is built, naming the stage to restart from.

Failure semantics are deliberately asymmetric:

situation	verdict	why
annotation present, digests match	pass	ancestry proven current
annotation present, digests differ	hard fail	positive evidence of a stale ancestor
annotation absent	warn	image predates the annotation; provenance unknown, not known-bad
parent tag unresolvable	warn	that is the build’s error to report, not an ancestry violation

Only the digest half of the reference is compared, so --image-repo may move the chain to another registry path without tripping the check.

Escape hatch when you knowingly accept a stale ancestor: --no-verify-ancestry, or CROSS_VERIFY_ANCESTRY=0.

A full --from-stage base run pays nothing for this: there are no prior stages that could be stale, so the check returns immediately.

The manual discipline this replaces (still worth understanding, and still what the check tells you to do):

• After replacing any base image (compiler, sdk, etc.), rebuild from the replaced stage — not a later one. E.g. after a compiler push, start from --from-stage sdk, not --from-stage media.

• Verify downstream images contain the expected new content before relying on them (e.g. check that /opt/gcc-16.2.0-native-arm64 exists in the pinned sdk digest with nerdctl run --rm <repo>@<digest> ls -d /opt/gcc-16.2.0-native-arm64).

• The --from-stage flag only controls where execution starts; it does NOT update the base image of the first stage it runs. If the stage just before your --from-stage inherits from a stale upstream, so will your rebuild.

• For partial runs after a compiler update, always use --from-stage sdk as the minimum starting point.

Note the division of labour: digest pinning keeps a single orchestrator run internally consistent; the ancestry check is the cross-run half. Neither subsumes the other.

The command reference AGENTS.md used to carry

Moved here on 2026-09-15, unedited except for this heading and the relative links. AGENTS.md keeps the four items that are RULES (never pkill the orchestrator, verify the shipped bytes, the closure freeze, --log-dir is not universal); everything below is reference, which is what this page is for.

Build logs. --log-dir ./out/build-logs is accepted by build-cross-chain.sh and build-cross-stage.sh — the two that tee each stage build (the Makefile wraps them). The other three orchestrators (build-cross-compiler.sh, build-runtime-manifest.sh, build-runtime-artifacts.sh) do not take it: pipe them through 2>&1 | tee ./out/build-logs/<name>.log.

Stopping a chain. Use bash linux/scripts/stop-cross-chain.sh — it finds the run via its pidfile (falling back to a bracket-trick pgrep) and reaps the orphaned nerdctl/buildctl subtree. Never pkill the orchestrator; that orphans its children.

Cache knobs — three distinct things, never conflate them:

Knob	What it actually does
NO_CACHE=1	disables ALL --cache-from, local and registry, for the whole chain
RUNTIME_NO_CACHE=1	--no-cache on only the runtime package+wrapper builds (runtime-build-fns.sh) — a targeted guarantee against BuildKit worker-cache reuse of a stale COPY /opt/ffmpeg layer
CROSS_NO_LOCAL_CACHE_EXPORT=1	stops writing the local buildcache, but still reads the registry inline cache

The shipped-bytes rule that used to sit here is a RULE, so it stayed in AGENTS.md § Quick Reference; the saga behind it is cross-build-verification.md.

Three more knobs, all detailed in docs/linux-cross-builds.md § versions.env feature toggles: MEDIA_STRIP=0 turns off the media-prefix symbol-strip pass (default ON; --strip-all keeps .dynsym, so runtime linking is unaffected), while VULKAN_CROSS_STRICT=1 and WHEEL_SOABI_STRICT=1 promote two advisory WARN gates to fatal — the second catches a vendored wheel whose native .cpython-*.so carries a SOABI for the wrong arch, which otherwise fails only at import.

Most common build commands:

# Full cross-build chain (base -> compiler -> sdk -> media -> android -> runtime)
bash linux/scripts/build-cross-chain.sh --target-arches amd64,arm64,riscv64 --log-dir ./out/build-logs

# Compiler image only (amd64-hosted, contains cross toolchains for all arches)
./linux/scripts/build-cross-compiler.sh --cross-targets amd64,arm64,riscv64

# Compiler with custom image repo (matches --image-repo on the orchestrator)
./linux/scripts/build-cross-compiler.sh --image-repo ghcr.io/myorg/kataglyphis_beschleuniger --push

# Single cross stage — the canonical way to rebuild one stage for one arch.
# Handles parent digest pinning, build-arg assembly, log capture, and push.
# See docs/linux-cross-builds.md § "Recommended: digest-pinned orchestrator".
bash linux/scripts/build-cross-stage.sh --stage sdk --arch arm64 --push --log-dir ./out/build-logs
bash linux/scripts/build-cross-stage.sh --stage media --arch amd64 --push --log-dir ./out/build-logs
bash linux/scripts/build-cross-stage.sh --stage media --arch arm64 --push --log-dir ./out/build-logs

# --no-push FULL CHAINS are SAFE since 2026-08-30 (local OCI-layout handoff):
# every stage built locally is exported and handed to the child via
# --build-context <tag>=oci-layout://<dir>, so no FROM resolves against the
# registry (the 2026-08-08 stale-parent bug). Mid-chain resumes (--from-stage
# after base) are still REFUSED — the parent prefix was not built this run.
# CROSS_LOCAL_CONTEXT_HANDOFF=0 reverts to the old refusal; CROSS_NO_PUSH_FORCE=1
# bypasses. Single-stage validation stays supported:
bash linux/scripts/build-cross-chain.sh --only media --target-arches amd64 --no-push --log-dir ./out/build-logs
# Correct full-chain PUSH flow: push mode to android, then the runtime lane with
# --skip-manifest so a partial-arch run cannot clobber the public manifest —
# see docs/linux-cross-builds.md § "The flow that is correct today".

# Opt-in: build the per-target cross GCCs concurrently inside the compiler
# stage (~30% off the GCC RUN at 3 targets; default 0 = sequential).
# Forwarded to the container via the compiler-stage build-arg plumbing
# (stage-defs.sh) — a launch-time value that does not show up as a
# --build-arg in the dry-run command line is DROPPED and the sequential path
# runs instead. Validated on a real compiler build 2026-08-30.
GCC_PARALLEL_TARGETS=1 bash linux/scripts/build-cross-chain.sh --target-arches amd64,arm64,riscv64 --log-dir ./out/build-logs

# Verify chain freshness without building (real FRESH/STALE verdicts for images
# that carry the org.kataglyphis.parent-digest ancestry annotation)
bash linux/scripts/build-cross-chain.sh --verify-chain --target-arches amd64,arm64,riscv64 --log-dir ./out/build-logs

# Partial runs (--from-stage after base) auto-assert the ancestor chain against
# the registry and REFUSE to build on a stale ancestor. Deliberate override:
#   --no-verify-ancestry   (or CROSS_VERIFY_ANCESTRY=0)

# Standalone quick chain verification (lighter, no orchestrator flags)
bash linux/scripts/verify-cross-chain.sh --target-arches amd64,arm64,riscv64

# Print the full stage graph with tag names (no builds)
bash linux/scripts/build-cross-chain.sh --describe-chain --target-arches amd64,arm64,riscv64 --log-dir ./out/build-logs

# Dry-run: print all build commands without executing
bash linux/scripts/build-cross-chain.sh --dry-run --target-arches amd64,arm64,riscv64 --log-dir ./out/build-logs

# Cheap packaging validation before publish (see docs/linux-cross-builds.md)
# Uses the `wrapper-smoke` target in Dockerfile.package

# Reinstall QEMU/binfmt after host reboot OR containerd restart (see § Prerequisites)
linux/scripts/setup-rootless-binfmt.sh --arches arm64,riscv64 --install-service

Fresh Linux host? GPU driver + CUDA install, the NVIDIA default-runtime daemon.json, the rootless container-stack install, CPU/GPU performance mode, and GRUB recovery are docs/linux-host-setup.md — the Linux counterpart to docs/windows-host-setup.md.

See also: docs/linux-cross-builds.md for the full stage graph, digest pinning, and single-stage build details. docs/linux-build-basics.md for build fundamentals, caching, and troubleshooting.

Manual staged build (low-level reference)

For full hands-off builds, prefer the orchestrator build-cross-chain.sh with digest-pinned stage handoff. Each cross stage maps to one Dockerfile and one tag; the orchestrator and build-cross-stage.sh handle the per-arch fan-out, build arg assembly, and pin capture for you. Use the helpers unless you are debugging a specific stage in isolation.

If you must drive individual builds manually, refer to the helper scripts for the canonical argument set (or run them with --dry-run to print the commands they would execute). The essential pattern for each cross-lane stage is:

nerdctl build --platform linux/amd64 --pull=true \
  --output 'type=image,name=<tag>,push=true' \
  -f <dockerfile> \
  --build-arg BASE_IMAGE=<parent_tag_or_pinned_digest> \
  --build-arg BUILD_MODE=cross \
  [--build-arg TARGET_ARCH=<arch> if per-arch] \
  .

For the runtime lane, prefer build-runtime-manifest.sh. The helper handles the base -> package -> torch chain and multi-arch manifest creation. --manifest-only / --repair can recreate the manifest from existing per-arch wrappers without rebuilding any images.

linux/Dockerfile.sdk serves both the sequential SDK build and the amd64-hosted cross SDK artifact lane. The cross path consumes one TARGET_ARCH per nerdctl build, fanned out per architecture.

linux/Dockerfile.package is the handoff point where amd64-hosted cross artifacts are copied into a clean target-native root filesystem. For foreign-architecture images, the package stage must receive:

• A target-native /opt/llvm-target tree, wired to /usr/bin/clang

• A target-native /opt/gcc-16.2.0 (cross-compiled from source via Canadian cross, swapped in by Dockerfile.android), carrying libsanitizer like the amd64 full-make GCC: asan, ubsan, lsan and tsan, plus hwasan on arm64 (cross-build-verification.md#the-native-gcc-ships-libsanitizer)

• A hard-fail CC validation guard (dumpmachine, ELF type, cc1 smoke test)

linux/Dockerfile.torch produces the final :latest-<arch> wrapper images (torch venv, app, runtime scripts, entrypoint). The per-arch wrappers are assembled into the :latest multi-arch manifest.

OpenCV 5.x GStreamer compatibility (applies to all architectures)

OpenCV 5.x reorganized several modules relative to OpenCV 4.x. GStreamer’s bundled gst-plugins-bad “opencv” plugin (1.29.x) still targets the 4.x layout, so it fails to compile against the source-built OpenCV 5 in this image. The build system applies an automatic source patch via patch-gstreamer-sources.sh → patch_gstreamer_sources() that addresses three upstream API changes:

1. contourArea/approxPolyDP/convexHull moved from imgproc to the new geometry module → adds #include <opencv2/geometry.hpp> to gstsegmentation.cpp.

2. findChessboardCorners/findCirclesGrid/drawChessboardCorners + CALIB_CB_* moved from calib3d into objdetect → adds #include <opencv2/objdetect.hpp> to gstcameracalibrate.cpp.

3. cv::CascadeClassifier + CASCADE_* (legacy Haar cascade detection) were removed from OpenCV 5 → the three cascade-dependent GStreamer elements (faceblur, facedetect, handdetect) are dropped from the monolithic libgstopencv.so. The remaining 22 elements (dilate, sobel, smooth, edgedetect, tracker, grabcut, retinex, segmentation, cameracalibrate, etc.) build and function normally.

Additionally, build-opencv.sh creates an opencv4.pc → opencv5.pc compatibility alias because GStreamer’s meson dependency lookup queries dependency('opencv4', '>= 4.0.0').

SDK rootfs artifacts (first host-side build step)

The SDK stage can also be exported as root filesystems. build-sdk-artifacts.sh reuses linux/Dockerfile.sdk in BUILD_MODE=cross, builds target-specific SDK root filesystems for amd64, arm64, and riscv64 on a fast amd64 host, and exports them to disk.

Build the first SDK artifacts for amd64, arm64, and riscv64 while saving this run under one timestamped logs/ directory:

set -o pipefail
LOG_DIR="logs/$(date -u +'%Y%m%dT%H%M%SZ')-sdk-artifacts"
mkdir -p "${LOG_DIR}"

./linux/scripts/build-sdk-artifacts.sh --target-arches amd64,arm64,riscv64 --fast-ubuntu-mirror \
  --fast-ubuntu-mirror-url http://de.archive.ubuntu.com/ubuntu/ \
  2>&1 | tee "${LOG_DIR}/build-sdk-artifacts.log"

For individual SDK artifact builds, use build-cross-stage.sh:

for arch in amd64 arm64 riscv64; do
  bash linux/scripts/build-cross-stage.sh --stage sdk --arch "${arch}" --push --log-dir ./out/build-logs
done

The helper builds the compiler stage first (cross_stage_run compiler), so a warm build cache is what makes that step cheap. Pulling cross-compiler-amd64 beforehand does not: the pull lands in containerd’s store, which BuildKit’s OCI worker does not read.

The helper accepts TARGET_ARCHES=amd64,arm64,riscv64, TARGET_ARCH=amd64,arm64,riscv64, or --target-arches amd64,arm64,riscv64 and then fans that list out into one TARGET_ARCH=<arch> build per target.

Expected output layout:

out/linux-sdk/amd64/rootfs/
out/linux-sdk/amd64/artifact.env
out/linux-sdk/arm64/rootfs/
out/linux-sdk/arm64/artifact.env
out/linux-sdk/riscv64/rootfs/
out/linux-sdk/riscv64/artifact.env

This helper uses linux/Dockerfile.sdk with BUILD_MODE=cross and the amd64-hosted cross compiler image. During successful cross SDK builds, CMake should identify the active C++ compiler as GNU 16.2.0 rather than the Ubuntu 26.04 system GCC toolchain. The exported rootfs trees are for inspection and host-side use; the :latest pipeline (build-cross-chain.sh) does not read them.

linux/Dockerfile.sdk also forwards the checked-in LLVM_RELEASE pin into the target-clang step, so rebuilding an SDK artifact from an older cross-compiler-amd64 base still refreshes /opt/llvm-target to the repository pin instead of inheriting a stale base-image environment value.

Cross packaging to multi-arch manifest

Overview

This is the path that publishes :latest. It keeps the existing QEMU lane for compatibility while adding:

1. Cross-compile target artifacts host-side with the cross builder.

2. Assemble one runtime image per architecture from a clean per-arch linux/Dockerfile.base plus the target-built payload from cross-android-${TARGET_ARCH}.

3. Publish a single multi-architecture manifest.

linux/Dockerfile.package is the shared runtime packaging layer. It starts from a clean per-arch base, copies the selected target payload from the artifact image, replays final runtime dependency setup, and becomes BASE_IMAGE for linux/Dockerfile.torch. In cross mode the artifact image runs on amd64 (cross-android-${TARGET_ARCH}); in native mode it uses the target-platform sequential image directly.

Host prerequisite: QEMU/binfmt for the emulated runtime legs

The sdk/media/android stages cross-compile on amd64 and need no emulation by design — but note the registration is load-bearing even there: nested NATIVE tool sub-builds (e.g. IREE’s bundled-LLVM tblgen) historically only survived because qemu silently executed a wrong-arch binary. That specific case is fixed at the source (CROSS_TOOLCHAIN_FLAGS_NATIVE pins the host compilers), but keep binfmt registered — it is the safety net for the whole class.

Lifetime: the registration lives in the rootlesskit namespace and dies on host reboot and on systemctl --user restart containerd (this bit the 2026-08-08 foreign chain: the shim-failure restart earlier that day had silently wiped it, and media-arm64’s IREE build failed with Exec format error). --install-service below installs a systemd –user unit so it re-registers automatically. The runtime stage is different: build-runtime-manifest.sh builds the per-arch base → package → torch wrappers on the real target platform (nerdctl build --platform linux/arm64|riscv64). For foreign architectures those RUN steps (e.g. base-image.sh bootstrap-ca, copy-media-payloads.sh, apt, dpkg) execute under QEMU user-mode emulation, which requires QEMU emulators registered in binfmt_misc in the namespace where builds run.

Rootless setup (this host — no sudo)

Run the helper once per boot (it is idempotent, and --install-service makes it persistent via a systemd –user unit):

linux/scripts/setup-rootless-binfmt.sh                    # register arm64,riscv64 now
linux/scripts/setup-rootless-binfmt.sh --install-service  # + auto-register on every login/boot
linux/scripts/setup-rootless-binfmt.sh --verify           # check current state

You normally don’t run it by hand: build-runtime-manifest.sh invokes it for you (no sudo) right before the per-arch runtime-image smokes, for whichever target arches are non-native (ensure_foreign_binfmt). Set RUNTIME_REGISTER_BINFMT=0 to skip that auto-registration (e.g. a rootful/CI host where qemu is already registered via docker run --privileged tonistiigi/binfmt or update-binfmts). The standalone invocations above are for registering ahead of time or debugging.

Why the helper is needed (and why the “obvious” commands don’t work rootless):

• sudo apt install qemu-user-static is not required and not wanted here — this host runs rootless containerd + BuildKit and must stay sudo-free.

• A plain nerdctl run --privileged --rm tonistiigi/binfmt --install all does NOT work even though it prints arm64 OK. A rootless --rm container registers binfmt inside its own ephemeral user namespace, which is destroyed on exit — the registration never reaches the namespace where real containers/builds run. Symptom: exec format error on any nested exec, e.g. a -d container that returns an ID but immediately exits 255 with exec /docker-entrypoint.sh: exec format error, or a build step dying at uname / apt.

• The key insight this host relies on: buildkitd is launched nsenter’d into containerd’s rootlesskit namespace (systemctl --user cat buildkit.service → ExecStart=... containerd-rootless-setuptool.sh nsenter -- buildkitd ...), so nerdctl run and nerdctl build share one persistent rootless namespace. The helper registers QEMU once in that shared namespace (entering it the same way, via containerd-rootless-setuptool.sh nsenter), so both emulate correctly. Because binfmt_misc is user-namespace-mountable on this kernel, the helper overmounts a fresh, namespace-owned (writable) binfmt_misc there without any host privilege.

Registration flags matter — the helper uses POCF:

flag	meaning	why it’s needed
P	preserve-argv[0]	critical — without it qemu drops argv[1]; sh -c CMD loses -c and dash treats CMD as a filename (cannot open …: No such file)
O	open-binary as fd	lets qemu run a target that isn’t on the interpreter’s path
C	credentials	setuid/setgid handling
F	fix-binary	kernel opens the interpreter fd at registration time, so it is inherited into nested build/run namespaces where the qemu path isn’t mounted

Symptom in an orchestrator run when binfmt is missing/misregistered: the runtime stage’s arm64/riscv64 legs die with error: failed to solve: process "/dev/.buildkit_qemu_emulator ... bootstrap-ca ..." did not complete successfully: exit code: 1, while amd64 (native, no emulation) succeeds.

Verify emulation actually works (not just “registered”) before a runtime run:

# both should print the target machine, NOT "Exec format error"
nerdctl run --rm --platform linux/arm64  ubuntu:26.04 uname -m   # -> aarch64
nerdctl run --rm --platform linux/riscv64 ubuntu:26.04 uname -m  # -> riscv64

tonistiigi/binfmt’s “OK” output means “a registration was written in my namespace”, not “emulation works”. Always confirm with the run test above — a -d container that returns an ID can still have exited immediately with exec format error.

Rootful hosts

On a rootful Docker/containerd host the standard docker run --privileged --rm tonistiigi/binfmt --install all (or apt install qemu-user-static) registers in the host binfmt_misc and works directly, because containers there share the host (init) user namespace. The rootless helper above is only needed when the daemon runs rootless.

The per-arch latest-base-*, latest-package-*, and latest-* tags are internal publish tags used to assemble the public latest manifest. Variant manifests follow the same shape: :latest-<variant> (e.g. nvidia, rocm) over per-arch :latest-<variant>-<arch> wrappers. Hailo has no variant tag — it ships in the standard amd64/arm64 wrappers (Dockerfile.torch). :latest-cross, the manifest’s name before 2026-09-22, is retired: nothing publishes it again, and its registry tags were deleted on 2026-09-29. Prefer the runtime helpers (the canonical commands are in § Runtime lane helper commands). Run with --dry-run to print the commands without building.

Runtime helper scripts

Two helpers manage the base → package → torch → wrapper → manifest chain:

• build-runtime-manifest.sh — builds the full chain and publishes the multi-arch manifest.

• build-runtime-artifacts.sh — builds the chain and exports the final wrapper rootfs instead of creating a manifest.

Both accept --target-arches, TARGET_ARCHES, or TARGET_ARCH for architecture selection, and ARTIFACT_BUILD_MODE=cross|native for the package artifact source. In cross mode, ARTIFACT_IMAGE_PREFIX is a prefix (e.g. ghcr.io/...:cross-android) that fans out -${TARGET_ARCH}; in native mode it is the exact artifact image ref.

The app wheelhouse (05-frameworks/torch/build-app-wheelhouse.sh) is built on the amd64 host: IREE on every arch, plus torch and torchvision for riscv64, which has no upstream wheels. It reaches the wrapper through /opt/wheels, with the other components’ wheels. The final linux/Dockerfile.torch stage runs on the real target platform in both modes so /opt/venv is correct for the target architecture.

Local handoff behavior:

• When images stay local, base is exported as a plain rootfs directory, package and torch as OCI layouts consumed through named build contexts.

• ARTIFACT_CONTEXT_ROOT lets helpers consume previously saved artifacts from disk instead of pulling from a registry.

• ARTIFACT_CONTEXT_MODE=oci resolves each <arch> within ARTIFACT_CONTEXT_ROOT as oci-layout://... (verified path for out/local-oci/android/{arm64,riscv64}).

• One build still fails when consuming two named OCI contexts at once; the workaround is runtime_artifact as OCI layout + runtime_base as plain rootfs directory.

• Each local stage context is deleted after the downstream build consumes it.

• --manifest-only (alias --repair) creates/pushes the manifest without rebuilding images — the recommended way to repair :latest from existing per-arch wrappers.

Torch pin enforcement on GPU lines

assemble-torch-app.sh’s enforce_torch_version_pins re-installs the pinned torch pair when the venv’s versions differ from versions.env. On the GPU lines three facts make the install non-obvious (rocm lane 2026-09-27):

1. The pin names the LOCAL VERSION (torch==2.14.0+rocm7.14, not torch==2.14.0). With PyPI as the --extra-index-url, a bare pin matched both the rocm7.14 wheel and PyPI’s plain 2.14.0 — and uv picked PyPI’s CUDA build, swapping a ROCm torch for one whose torch.version.hip is None. The GPU-tagged wheel is unambiguous.

2. --index-strategy unsafe-best-match is required for that local version: uv’s first-index-wins guard sees torch on PyPI first and refuses any version PyPI does not carry, without ever consulting the pytorch index. The guard exists to stop cross-source confusion; the +tag already names the source, so relaxing it is safe. Proven by a dry-run inside the android image.

3. With deps, never --no-deps: the CUDA index swapped a CUDA torch back to CPU, and --no-deps would keep the old torch’s pinned nvidia-cudnn/nccl/triton (they move between torch minors).

CUDA resolves the same way (+cu130), so one code path serves both GPU lanes.

The wrapper’s wheelhouse: two deliveries

The torch RUN in Dockerfile.torch builds /opt/venv from /opt/wheels: 8 to 10 wheels, about 83 MB on arm64 and 165 MB on riscv64. Those wheels live in the android image, which is about 12.7 GB in 96 layers. RUNTIME_WHEELS_SOURCE picks how the RUN gets them. Both deliveries read the same image. They differ in what BuildKit must hold while the wrapper builds.

RUNTIME_WHEELS_SOURCE	What happens
auto (default)	The same as image, for now.
image	The delivery every image was built with before 2026-09-24, unchanged. The torch RUN bind-mounts /opt/wheels from the android image (stage wheels-source). A pushing run hands BuildKit the digest-pinned ref. Under ARTIFACT_CONTEXT_ROOT (a --no-push chain) runtime_wheels_context_dir extracts the directory from containerd with nerdctl export | tar. When the android tag is published, that ref is the registry’s digest: a --no-push chain threads no pin, so runtime_android_pin falls back to the registry. The extraction then reads the PUBLISHED android (nerdctl pulls it when it is not local), and the venv gets its wheels while the package copies from this run’s layout. That predates the knob, and neither mode changes it.
export	Before each arch’s base and package builds, BuildKit builds only Dockerfile.torch’s wheels-export stage with --output type=local. That copies /opt/wheels and the free-threaded twins in /opt/wheels-free-threaded out of the same image into <RUNTIME_CONTEXT_ROOT>/runtime-flow.wheels.*/<arch>, and a sha256 manifest of both is written beside it. The wrapper build re-checks the manifest, then mounts the directory (WHEELS_IMAGE=runtime_wheels, --build-context runtime_wheels=<dir>). The directory is removed after the wrapper.

How to pick one.

• Keep the default unless you are measuring or have measured. Every published image was built through image.

• Put RUNTIME_WHEELS_SOURCE=export in front of build-cross-chain.sh, build-runtime-manifest.sh or build-runtime-artifacts.sh for one run. Unset it, or set image, to go back. A switch costs no rebuild of its own: the only step it re-keys, the torch RUN, re-runs on every lane anyway.

• Not on a --no-push chain whose android tag is published, which is every default or variant --no-push chain on the amd64 cross host: export is refused there (below). Use image for those.

• A native host takes either one: a Jetson with CROSS_BUILD_PLATFORM=linux/arm64, the X100 with linux/riscv64. The export stage is one COPY and runs no target code, so it behaves the same on an amd64 cross host and on a native host. The refusal above applies there too, once that host has pushed its -host<arch> android tag.

• Any other value stops the lane with exit 2, and the chain checks it before its first stage.

What export guarantees, and what it refuses.

• The same image. The export resolves its ref with runtime_wheels_image_ref, the function image mode uses. It passes the same WHEELS_IMAGE, or none when the ref is empty, so both fall back to the Dockerfile default. It uses the wrapper build’s --platform and pull policy.

• Under ARTIFACT_CONTEXT_ROOT (a --no-push chain) the export reads the android OCI layout the package build reads, but only when the layout’s one manifest digest equals the digest the ref has in containerd (nerdctl image inspect --mode=native). Otherwise it stops and names both. ARTIFACT_CONTEXT_MODE=dir is refused, because there is no digest to compare.

• So a --no-push chain whose android tag is published always stops there. The ref is then the registry’s digest (see the image row above), and this run’s layout never has it. The arch fails at the runtime lane, before its base build, hours after the chain started; the chain does not check this at its start. The remedy is RUNTIME_WHEELS_SOURCE=image. Never re-export the layout from the ref the error names to make the two agree: the package build copies from that layout, so it would switch to the published android too. The check can pass only while the android tag has never been pushed, so that the ref is this run’s containerd tag: a native host’s -host<arch> tag. Reading this run’s tag on the cross host instead would give up “the same image as image mode”, so it is left to the owner.

• A failed export build, or one without a wheel, fails the arch before its base build.

• At wrapper time the build fails if the directory or its manifest is missing, or if any file changed, appeared or vanished since the export. Nothing falls back from export to image; going back is the operator’s switch.

• Not proven by the unit suite (tests/test-runtime-wheels-source.sh, which runs no BuildKit): the named-context override on the build host’s rootless OCI worker, and that both modes mount identical bytes there. The A/B recipe in cross-build-verification.md checks both.

Why it exists. In the 2026-09-22 lane the torch RUN waited 161 / 565 / 661 s (amd64 / arm64 / riscv64) before uv venv wrote its first file. On 2026-09-21 it waited 96 / 1 / 2 s with the same scripts. The likely cause is BuildKit pulling, extracting and hashing the whole android snapshot for that one bind mount, after something evicted it between the package build and the wrapper. export moves the dependency to before the package build, which needs the same snapshot anyway. That cause is inferred, not proven. The sharing=locked apt cache mounts are not ruled out, and amd64’s recurring 96-161 s has no explanation yet. Count savings per arch, and do not count on the amd64 share.

Where it lives. linux/scripts/lib-runtime-wheels.sh, beside lib-orchestrator.sh and outside every image closure. Changing it, including a later switch of auto to export, re-keys no stage. 01-core/runtime-build-fns.sh holds only the ref resolver, the image arm (_append_wheels_image_args) and a two-way branch in append_wrapper_build_args, because that is where the wrapper’s arguments are built.

Verified local foreign-architecture rebuild

ARTIFACT_CONTEXT_ROOT="$PWD/out/local-oci/android" \
ARTIFACT_CONTEXT_MODE=oci \
RUNTIME_CONTEXT_ROOT="$PWD/out/local-oci/runtime-contexts" \
bash linux/scripts/build-runtime-artifacts.sh \
  --target-arches arm64,riscv64 \
  --image-prefix docker.io/library/opencode-local:latest \
  --artifact-image-prefix docker.io/library/opencode-local:cross-android \
  --artifact-build-mode cross \
  --fast-ubuntu-mirror \
  --fast-ubuntu-mirror-url http://de.archive.ubuntu.com/ubuntu/ \
  --fast-ubuntu-ports-mirror-url http://ports.ubuntu.com/ubuntu-ports/

Previously validated for both arm64 and riscv64 (2026-08 runs, before the LLVM 23.1.0 bump): gcc 16.2.0, clang 22.1.8, /usr/bin/cc → /etc/alternatives/cc → /opt/gcc-16.2.0/bin/gcc, and optional runtime payloads under /usr/local/lib/onnxruntime-*, /usr/local/include/tflite, /usr/local/include/tensorflow, /usr/local/lib/pkgconfig/litert.pc.

After the runtime helper cleanup, validated for amd64 with:

RUNTIME_CONTEXT_ROOT="/tmp/opencode/runtime-contexts" \
bash linux/scripts/build-runtime-artifacts.sh \
  --target-arches amd64 \
  --output-root /tmp/opencode/runtime-smoke \
  --image-prefix docker.io/library/opencode-local:latest-smoke \
  --artifact-image-prefix ghcr.io/kataglyphis/kataglyphis_beschleuniger:cross-android \
  --artifact-build-mode cross \
  --fast-ubuntu-mirror \
  --fast-ubuntu-mirror-url http://de.archive.ubuntu.com/ubuntu/ \
  --fast-ubuntu-ports-mirror-url http://ports.ubuntu.com/ubuntu-ports/

Result at the time (previously, on the LLVM 22 pin): gcc 16.2.0, clang 22.1.8, target x86_64-unknown-linux-gnu, /usr/bin/cc → /etc/alternatives/cc → /opt/gcc-16.2.0/bin/gcc, /usr/bin/clang → /etc/alternatives/clang → /usr/local/llvm-target/bin/clang.

Local wrapper smoke validation

mkdir -p ./out/build-logs && \
nerdctl build --platform linux/amd64 \
  -t local/kataglyphis:latest-wrapper-smoke-amd64 \
  -f linux/Dockerfile.package \
  --target wrapper-smoke \
  --build-arg BASE_IMAGE=ghcr.io/kataglyphis/kataglyphis_beschleuniger:base \
  --build-arg ARTIFACT_IMAGE=ghcr.io/kataglyphis/kataglyphis_beschleuniger:cross-android-amd64 \
  --build-arg ARTIFACT_PLATFORM=linux/amd64 \
  --build-arg TARGET_ARCH=amd64 \
  --build-arg BUILD_MODE=cross \
  --build-arg GCC_VERSION=16.2.0 \
  --build-arg LLVM_RELEASE=23.1.3 \
  --build-arg USE_FAST_UBUNTU_MIRROR=true \
  --build-arg FAST_UBUNTU_MIRROR_URL=http://de.archive.ubuntu.com/ubuntu/ \
  --build-arg FAST_UBUNTU_PORTS_MIRROR_URL=http://ports.ubuntu.com/ubuntu-ports/ \
  . 2>&1 | tee ./out/build-logs/wrapper-smoke.log

Centralized Version Management

All version numbers are now tracked in a single file: linux/scripts/01-core/versions.env. Update this file when bumping versions — do NOT scatter version changes across individual Dockerfiles.

common.sh and artifact-common.sh both source versions.env at load time with set -a, so all build scripts and orchestrators automatically receive canonical values. The per-Dockerfile ARG defaults are kept as safety nets and should match versions.env.

After bumping versions, run python3 docs/scripts/sync_versions.py --write to update the version snapshot in README.md.

versions.env feature toggles (Linux lane)

Besides pins, versions.env carries feature switches for optional, build-cost-heavy capabilities. They are probe-gated: turning one on never hard-fails a build where the dependency is genuinely unavailable — the probe falls back to disabled. Current toggles:

Toggle	Effect	Notes
FFMPEG_ENABLE_X265	libx265 (HEVC) encoding in FFmpeg	Default ON (1), on all arches, and still probe-gated. It was off for a while over a source-x265 compile break.
FFMPEG_ENABLE_TF	FFmpeg TensorFlow DNN backend (amd64 only)	Default OFF (2026-08-14). When off, the TF C SDK is never downloaded and ~500 MB of libtensorflow* never enters the image; the ONNX DNN backend stays always-on regardless. Set =1 to restore it.
ORT_ENABLE_WEBGPU	ONNX Runtime WebGPU EP (Dawn)	Master switch, default ON; Dawn needs the GCC-16 -Wno-invalid-constexpr fix (2026-07-20).
ORT_WEBGPU_ALLOW_CROSS	Allow the WebGPU EP on cross arches	Default ON. Dawn cross-build is the risky part; set it to false and WebGPU is amd64-only.
ORT_ENABLE_LTO	--enable_lto for every ONNX Runtime build that goes through the shared helper (CPU, AMD, NVIDIA)	Default OFF (AP6, 2026-08-24), on all arches at once. A measuring knob, not probe-gated.
GENAI_ALLOW_RISCV64	riscv64 self-builds onnxruntime-genai (GEN1)	Default ON; anything but true backs GEN1 out. Not probe-gated. gen1-riscv64-genai.md

QNN EP (Qualcomm QAIRT SDK, backlog QNN-LINUX): opt-in for the arm64 lane, targeting Snapdragon NPU. No environment toggle — staging a zip in linux/qnn-sdk/ is the switch. When a Linux AArch64 SDK zip is present, resolve_qnn_sdk (shared module 01-core/qnn-sdk.sh, loaded by media_common_init) extracts it, verifies the SHA-256 against QNN_SDK_LINUX_ZIP_SHA256 in versions.env (if pinned), checks version compat (QNN_OP_STFT canary), and each framework wires its own flag (mirroring Windows #121): ORT onnxruntime_USE_QNN=ON + onnxruntime_QNN_HOME=<root> + QNN_ARCH_ABI=aarch64-oe-linux-gcc11.2 (ORT CMake defaults to aarch64-android on Linux aarch64, overridable via -D — it is a cache var guarded by if(NOT QNN_ARCH_ABI)). ORT is the only framework with a build-time QNN flag — corrected 2026-08-31, see Windows backlog #154: LiteRT’s TFLITE_ENABLE_QNN, TVM’s USE_QNN and IREE’s IREE_TARGET_BACKEND_QNN do not exist upstream, so CMake dropped all three silently while the scripts logged success. LiteRT’s real switch is LITERT_ENABLE_QUALCOMM, auto-forced ON by QAIRT_HEADERS_DIR=<root>/include/QNN (the dir holding Qnn*.h — qnn-linux.md explains why the include root does not work), which this lane now passes; TVM and IREE have no Qualcomm path at all. Backend .so + hexagon-v* skel dirs are staged beside each framework’s install by stage_qnn_runtime. No zip = QNN off with a notice. Different SDK from the Windows lane (aarch64-oe-linux-gcc11.2/, not aarch64-windows-msvc). See linux/qnn-sdk/README.md. PROVEN 2026-08-30 on a staged QAIRT v2.49.0.260730 zip — cross-media-arm64 build GREEN, ORT provider wired (build results in refactoring-backlog-archive-2026-09-03.md § A2. QNN-LINUX). Framework fan-out to GenAI/LiteRT/TVM/IREE is WIRED (same 2026-08-30 change), and the validation build ran 2026-09-03 against a real v2.49.0.260730 staged on arm64 — see qnn-linux.md, which owns the results and the QAIRT_HEADERS_DIR defect that run exposed.

Because versions.env sits in the media build’s cache-key closure, toggle flips re-run the affected media compiles — batch them with planned pin bumps (AGENTS.md § Caching discipline, rule 1). After a toggle flip, force a fresh runtime wrapper build with RUNTIME_NO_CACHE=1 (scoped to the runtime package + wrapper; lighter than whole-chain NO_CACHE=1) so the shipped image actually reflects the new media. The shipped bytes are now checked automatically: verify-shipped-wrapper.sh runs in its own pass over every arch and asserts the shipped /opt/ffmpeg lib set matches the versions.env toggles (so FFMPEG_ENABLE_TF=0 ⇒ libtensorflow* must be absent). Set WRAPPER_CONTENT_GATE=0 to downgrade that gate to advisory.

Why the explicit RUNTIME_NO_CACHE: the runtime wrapper once shipped :latest-cross byte-identical five times because nerdctl build --output type=image,name=X never creates a local containerd tag on this rootless host, so push + manifest kept resolving a stale prior tag (root cause RTCACHE3; the earlier “registry cache-hit” theory, RTCACHE1, was a red herring). The fix switched append_runtime_image_output to plain -t; the byte gate above is the belt-and-suspenders check. Always verify shipped bytes, not just that the manifest pushed.

TVM is pinned by COMMIT, not by tag

TVM_REF names the release and TVM_COMMIT in versions.env is what the build clones: it wins over the tag (tvm.sh’s ${TVM_COMMIT:-$ref}, Windows’ Get-SourceBuildVersion order). Since v0.27.0 (2026-10-07) it is the tag’s own peeled commit, and it must never be emptied: apache/tvm also carries a branch named v0.27.0 (the 0.27.0.post1 commit 7b5acb32), and git clone --branch v0.27.0 takes the branch, so an empty pin builds a commit the tag does not name and that can move. Check for a same-named branch (git ls-remote refs/heads/<tag>) at every bump.

The history: until v0.27.0 the pin was upstream main (994e0216, #20189), because TVM v0.26.0 does not compile against LLVM 23 (LLVM_RELEASE) — LLVM 23 dropped TargetOptions::{NoInfsFPMath,NoNaNsFPMath}, renamed SubtargetSubTypeKV::Key/SubtargetFeatureKV::Key to key(), and changed getMCSubtargetInfo() from pointer to reference, across three files. amd64 escaped it at first only because it linked the distro llvm-config-21; tvm-detect.sh now picks the LLVM_RELEASE major on every arch and asserts it, so the commit pin matters on amd64 too.

Upstream main carried TVM_LLVM_VERSION >= 230 guards for all of it and no tagged release did before v0.27.0, so the commit was pinned rather than the port reproduced. A hand-written patch was tried first and removed: measured on a real arm64 build it fixed four of five sites in one of the three files while logging success.

Two things follow, both proven on 2026-08-27:

• RUNTIME_CONTEXT_KEEP_HOURS (default 24) governs the orphaned-stage-context sweep — unrelated to TVM, but the other knob added the same day.

• Setting TVM_COMMIT for the first time exposed that the SHA clone path never initialised submodules: it clones without --recursive and the submodule update only fired when the checkout MOVED HEAD. Pinning the default-branch HEAD moves nothing, so CMake died on an empty 3rdparty/tvm-ffi. Fixed.

v0.27.0 ships the guards (994e0216 is its ancestor), so the pin moved to the tag’s commit 4d0ac168 rather than back to empty, for the branch reason above.

TVM cross wheels: what must not be simplified away

import tvm needs both wheels: its first statement imports tvm_ffi, and the app venv installs the tvm family --no-deps, so a lone apache-tvm wheel ships an unimportable package. USE_LLVM=ON works cross only because tvm.sh rewrites FindLLVM’s fallback to link the imported libLLVM dylib from /opt/llvm-cross/<triplet> instead of executing a foreign llvm-config, and that soname resolves at runtime through /etc/ld.so.conf.d/000-llvm-target.conf; dropping either half breaks the configure or the import. apache-tvm-ffi builds its Cython core WITH_SOABI, so it has the IREE/LiteRT host-SOABI problem, and tvm-python.sh withdraws staged wheels that do not carry the target suffix.

The wheels reach /opt/venv only through Dockerfile.torch’s /opt/wheels mount, but the native libtvm*.so that Dockerfile.media copies to /usr/local/lib is dropped at the package boundary: copy-media-payloads.sh copies an explicit allowlist with no libtvm entry. If the wheel turns out not to bundle its own runtime, fix that allowlist; a COPY glob in Dockerfile.torch would fail every build while the file is absent.

Operational env knobs (not versions.env)

Runtime/orchestration switches that are not pins or feature toggles. The cache-related ones are only half the story — what each cache tier covers, what destroys it, and what replacing it costs is in docs/build-cache-tiers.md:

Knob	Effect
NO_CACHE=1	--no-cache across the whole chain.
RUNTIME_NO_CACHE=1	--no-cache on just the runtime package + wrapper builds (use after a media toggle flip).
RUNTIME_WHEELS_SOURCE=auto|image|export	Where the wrapper’s torch RUN gets /opt/wheels: a bind mount of the android image (image; auto, the default, means image), or a sealed directory exported from that same image before each package build (export). Both paths.
CROSS_NO_LOCAL_CACHE_EXPORT=1	Write-only cache (skip the local cache export; disk relief on big rebuilds).
NO_CACHE_EXPORT=1	Drop everything registry-facing (inline cache export + registry cache reads). The manual recovery for the ghcr DeadlineExceeded/httpReadSeeker import flake — which the chain now also does by itself after two such failures.
CROSS_REGISTRY_CACHE=max	DESIGN ONLY — setting it is a silent no-op. The T4 pilot was declined and its implementation reverted; the --cache-to type=registry it describes no longer exists, and verify-critical-fixes.sh actively forbids the dead <tag>-buildcache ref from returning. Kept as a row only so the name resolves to its history — see build-cache-tiers.md § 4.
WRAPPER_CONTENT_GATE=0	Downgrade the shipped-wrapper byte gate to advisory.
MEDIA_STRIP=0	Disable the media-prefix symbol-strip pass (default on; ffmpeg/gstreamer/libcamera).
VULKAN_CROSS_STRICT=1	Promote the Vulkan cross-components gate (vulkan.sh) from advisory WARN to fatal. It fires when all three cross-components — loader/SPIRV-Tools/glslang — failed, an env-shaped toolchain cause.
CROSS_IMAGE_PRUNE=0	Leave the containerd IMAGE store alone (01-core/disk-guard.sh). The lever runs only where nothing is in flight — between stages and at runtime-lane entry — and removes dangling images plus this chain’s own stage tags the run no longer needs.
VULKAN_CROSS_REQUIRED=''	Empty the list of Vulkan components whose cross-build failure is fatal (vulkan.sh, default: loader, SPIRV-Tools, glslang, shaderc, Vulkan-Tools, ValidationLayers). Everything else in _VK_TARGET_COMPONENTS is non-fatal either way.
WHEEL_SOABI_STRICT=1	Promote the vendored-wheel SOABI gate (verify-wheels.sh) from advisory WARN to fatal. It fires when a vendored wheel’s native .cpython-*.so carries a SOABI for a different arch than the target triple — a host-SOABI leak that only fails at import. The triple is derived from TARGET_ARCH, not the running interpreter.
HOST_PAYLOAD_ARCHES=amd64	The arches whose package image keeps the x86_64-host Android SDK, its payloads and a Rust toolchain built for another host: a comma list, or none. The rest get HOST_PAYLOAD=off and ship those trees empty. amd64,arm64,riscv64 builds the images as published until 2026-10-13. Why.
WEB_LANE_TOOLS_SOURCE=auto|cross|native|legacy	Where riscv64’s from-source wasm-pack / flutter_rust_bridge_codegen come from: android’s cross build when usable (auto, default), only that (cross), a gated, cached compile in the package stage with rv64gc Rust and vendored C (native), or the package stage’s pre-2026-09-23 cargo install verbatim (legacy). Contract.
WEB_LANE_TOOLS_CACHE=on|refresh|off	Their package-stage binary cache: read and write (default), rebuild and re-store, or bypass.
WEB_LANE_TOOLS_CROSS_ARCHES=riscv64	Targets the android producer cross-builds them for: a comma list, or none to turn it off.
HAILO_NESTED_CACHE=carry|off	Whether the protobuf build HailoRT runs inside its own configure uses the compiler cache (carry, default) or runs uncached as before 2026-09-24 (off). Both paths.
HAILO_PYHAILORT_IPO=off|upstream	pyhailort with upstream’s forced LTO patched out, a real module (off, default), or as upstream ships it, which lld links empty (upstream).

The three WEB_LANE_TOOLS_* toggles and the two HAILO_* switches are Dockerfile ARGs. lib-orchestrator.sh appends their names to the forwarded set, so an exported value reaches every stage the way a versions.env key does, and an unset one sends no --build-arg at all. They are kept out of versions.env because any byte there re-keys the chain from base. The chain and both runtime orchestrators refuse an unknown HAILO_* value (exit 2) before they build anything.

HOST_PAYLOAD_ARCHES is no Dockerfile ARG. The runtime lane resolves it per arch into Dockerfile.package’s HOST_PAYLOAD (runtime_host_payload), and refuses a bad list the same way.

Clang cross wrappers

install_cross_clang_wrappers (02-toolchain/llvm.sh) writes /usr/local/bin/clang-<arch> and clang++-<arch>, each a one-line exec of the selected host clang with --target=<triplet>, --sysroot and --gcc-toolchain=<gcc_prefix> already baked in, where gcc_prefix is gcc_toolchain_prefix() (/opt/gcc-$GCC_VERSION) or /usr when that directory is missing. These wrappers are the only path in the build where clang IS the compiler; nothing in the tree sets a bare CC=clang. They live in the build-stage images (the compiler stage installs them); Dockerfile.package does not copy them, so the runtime image has none, and a consumer compiling with clang in :latest passes --gcc-toolchain itself (gcc_toolchain_prefix() in 01-core/cross-gcc.sh names the root). From the image after CON11 it need not: the package stage writes <native-triple>-clang.cfg and -clang++.cfg beside the compiler, holding --gcc-toolchain=${GCC_PREFIX} (BACKLOG CON16), which clang reads for a native build only; a --target build loads neither.

Clang looks for the pair in the directory it was reached through, not the one it resolves to. The compiler resolves /proc/self/exe, so a bare clang++ finds the pair beside /usr/local/llvm-target/bin/clang-23; clang-tidy builds its driver from the compile database’s /usr/bin/clang++ unresolved and looked in /usr/bin, where the first image after CON11 had no pair: Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/16 (BACKLOG CON39, measured 2026-09-30). write_clang_gcc_toolchain_cfg therefore writes the pair into every directory of /usr/bin and /usr/local/bin holding a link to the driver (find -L … -samefile), and validate-compilers.sh smoke grades clang-tidy through /usr/bin/clang and /usr/bin/clang++. A database naming a bare clang++ (no directory) still reaches no pair; CMake always writes the full path.

CROSS_GCC_TOOLCHAIN_PATH and export_clang_gcc_toolchain_env were deleted on 2026-09-05 (backlog CL3). The function exported --gcc-toolchain into CFLAGS/CXXFLAGS/LDFLAGS and had no caller in the build, so the knob changed nothing: what points clang at the source-built GCC is the wrapper’s own exec line, from the same gcc_toolchain_prefix(). The two also disagreed on the missing-prefix case (the wrapper falls back to /usr, the function warned and exported nothing), so keeping both meant two policies for one question. An operator still exporting the name gets no effect and no warning. tests/test-clang-cross-wrappers.sh now asserts what the wrapper bakes; that was untested while the dead knob had a suite of its own.

IREE (Linux lane)

IREE (IREE_VERSION in versions.env, currently v3.12.0) ships 3-arch in the media image since 2026-07-14, with a deliberately split strategy per arch.

No arch installs an upstream wheel: PyPI ships iree-base-{compiler,runtime} as cp312-abi3 for x86_64+aarch64 only, riscv64 has none, and we need version-specific cp314 everywhere — so every arch source-builds, with IREE’s abi3 tagging patched out (_iree_patch_setup_py_abi3 in build-app-wheelhouse.sh). What differs is which wheels come out:

• amd64 — NATIVE build (target arch == build arch), IREE_BUILD_COMPILER=ON (_iree_build_target_native): ships both iree_base_compiler and iree_base_runtime.

• arm64 / riscv64 — CROSS build, RUNTIME-ONLY (-DIREE_BUILD_COMPILER=${IREE_CROSS_BUILD_COMPILER}, default OFF, _iree_build_target_cross), so only iree_base_runtime ships. This is not a preference: IREE imports host tools only under if(IREE_HOST_BIN_DIR AND NOT IREE_BUILD_COMPILER), so COMPILER=ON makes the target ignore IREE_HOST_BIN_DIR, build its own iree-tblgen for the target arch, and then run it on the amd64 host — Exec format error, [code=126] on VMOpEncoder.cpp.inc. Set IREE_CROSS_BUILD_COMPILER=ON to re-try both wheels once IREE supports the combination. Models are compiled on amd64 and executed on the cross arches.

The cross path’s companion host stage (amd64 has none — one cmake configure does everything) supplies iree-c-embed-data, iree-flatcc-cli and iree-tblgen via IREE_HOST_BIN_DIR (_iree_build_host_stage). Only IREE_BUILD_COMPILER=ON installs iree-tblgen, so with the default runtime-only target the host stage builds ON directly (the full-LLVM compile you see in the app-wheelhouse stage). Under IREE_CROSS_BUILD_COMPILER=ON the target needs no host iree-tblgen; there the host stage tries OFF first and escalates to ON only when a required tool is missing. iree-two-stage-build.md has the reasoning.

Every target tree configures twice before it builds (_iree_configure_settled, CON81, 2026-10-07), the native one and the cross one alike. IREE v3.12.0’s top-level CMakeLists.txt adds runtime/ (line 1199), where iree::base links ${IREE_LIBBACKTRACE_TARGET}, before build_tools/third_party/libbacktrace (line 1236) caches that variable. So only a second configure gives every object that sees iree::base IREE_HAVE_LIBBACKTRACE=1: 1187 of the amd64 compiler tree’s 6700 compile lines, 757 of them compiler sources. Any later reconfigure, such as the cp314t twin’s, would otherwise recompile all of them and relink libIREECompiler.so. The order is IREE’s, not the toolchain’s: an aarch64 cross configure of the runtime moves 377 of its 474 compile lines the same way, so the arm64 and riscv64 target trees settle too.

• The bytes do not change. Release builds run IREE_STATUS_MODE 2, which compiles the stack-trace path out, so neither configure links libbacktrace code. A settled amd64 build’s 28 native files are byte-identical to those of :latest, which was built before the settle. That build was v3.11.0 (the published image’s ENV IREE_VERSION wins over the tree’s pin in a local run), which has the same order at lines 1149/1186.

• What it buys is a twin pass that recompiles only what sees Python (consumer-image-contract.md § The free-threaded wheels).

• The host stage configures once. It ships nothing and is never reconfigured.

Build home: linux/scripts/05-frameworks/torch/build-app-wheelhouse.sh (build_iree_wheels), which stages host tools + target runtime and is smoked both natively and via the Python import path. The riscv64 builder iterates on real rebuilds — treat first-failure there as expected tuning, not regression.

verify-parity.sh (on-demand diagnostic, not a gate)

06-packaging/verify-parity.sh <native-image> <cross-image> diffs two BUILT images across packages,python,versions,files,libs,imports. It needs two images, so it can never be a preflight gate (preflight’s artifact-parity slug is the UNRELATED verify-artifact-copy-parity.sh). Use it when a cross image misbehaves where the native one doesn’t — it localizes the divergence in minutes.

Five Critical Fixes To Maintain

To prevent regressions during updates, always preserve the following five vital fixes in the Linux cross pipeline:

1. Fix 1 (gst-python staged libpython): In build_python.sh, python_stage_finalize() runs fix_python_pc_file() over the staged python-<mm>.pc and -embed.pc so their libdir/includedir point at the compiler’s cross directory, and symlinks python3.pc to them, so gst-python builds succeed.

2. Fix 2 (libcamera abseil): build-litert.sh (_install_manual_abseil) must install the Abseil headers the TFLite headers include (absl/types/span.h among them) through the shared install_abseil_headers (01-core/abseil-headers.sh, which build-libcamera.sh uses too), to prevent downstream libcamera build errors.

3. Fix 3 (cross lib-dynload dangling symlinks): In build_python.sh (build_cross_target_python_payload()), standard CPython build steps create standard cross-build library symlinks that end up dangling when packaged. We use cp -a -L to dereference those symlinks, copy the safety-net Modules, and enforce a hard-fail guard find ... -xtype l to ensure absolutely zero dangling symlinks remain in the target’s lib-dynload subdirectory. This prevents C-extension import failures (e.g. import _struct failing under QEMU/binfmt). Since target-packaged Python is staged into the compiler-cross image, the compiler itself must be rebuilt if this helper logic is changed.

4. Fix 4 (cross GCC architecture guard): In Dockerfile.package, GCC alternatives wire /opt/gcc-16.2.0/bin/gcc as cc/c++. On amd64, GCC is built natively. On arm64/riscv64, it is Canadian-cross-compiled; Dockerfile.android swaps the amd64-hosted GCC for the target-native binary. The build hard-fails with three layered guards: (a) cc -dumpmachine must match TARGET_ARCH; (b) readelf -h on the cc binary itself checks ELF machine type (the real discriminator — -dumpmachine only reports the target triple, not the host arch); and (c) a cc1 compile-to-object smoke plus ELF check on the produced object, run under the target platform (QEMU for foreign arches). wrapper-smoke (Dockerfile.package target) runs validate-compilers.sh, smoke-media.sh, smoke-torch-venv.sh and smoke-cross-all-arches.sh for end-to-end verification.

5. Fix 5 (OpenCV 5 GStreamer compat): patch-gstreamer-sources.sh → patch_gstreamer_sources() patches the GStreamer gst-plugins-bad opencv plugin sources at build time for OpenCV 5.x compatibility. Three API changes are handled: (a) contourArea/approxPolyDP/convexHull moved to new geometry module → adds #include <opencv2/geometry.hpp> to gstsegmentation.cpp; (b) chessboard/circles-grid detection (findChessboardCorners/findCirclesGrid/CALIB_CB_*) moved to objdetect module → adds #include <opencv2/objdetect.hpp> to gstcameracalibrate.cpp; (c) cv::CascadeClassifier removed from OpenCV 5 → drops the three cascade-dependent GStreamer elements (faceblur, facedetect, handdetect) from the monolithic libgstopencv.so. Additionally, build-opencv.sh creates an opencv4.pc → opencv5.pc compatibility alias because GStreamer’s meson dependency lookup queries dependency('opencv4'). All patches are idempotent (guarded with grep before applying). When changing OpenCV or GStreamer versions, verify the patch still applies correctly.

Cross env contract

The cross environment set up by linux/scripts/01-core/cross-env.sh (setup_linux_cross_env) is organized in three tiers. Run linux/scripts/01-core/cross-env-doctor.sh <arch> (or source it and call cross_env_doctor) to validate the contract, print the effective configuration, and compile-smoke-check that $CC really emits target-arch ELF objects.

Tier 1 — core toolchain contract

Always exported when a cross build is active: TARGET_ARCH, TARGETARCH, TARGETPLATFORM, BUILDARCH, CROSS_TARGET_TRIPLET, and the tool variables CC, CXX, AR, AS, LD, NM, RANLIB, STRIP, OBJCOPY, plus PKG_CONFIG_LIBDIR, PKG_CONFIG_SYSROOT_DIR, PKG_CONFIG_ALLOW_CROSS. CC/CXX must be absolute paths to existing executables. Consumers must use these variables — never bare cc/gcc from PATH — for target-side compiles.

Tier 2 — rust / cmake derivations

Derived from Tier 1: CROSS_RUST_TARGET, CARGO_BUILD_TARGET, CARGO_TARGET_DIR, CARGO_TARGET_<TRIPLE>_LINKER / _AR, the cc-crate vars CC_<triple> / CXX_<triple> / AR_<triple> / RANLIB_<triple> (for both target and build triples), and the CMAKE_* toolchain variables (CMAKE_SYSTEM_NAME/PROCESSOR, CMAKE_C/CXX_COMPILER, CMAKE_AR, CMAKE_RANLIB, CMAKE_FIND_ROOT_PATH_MODE_*, …).

Tier 3 — PATH policy and bare tool names (opt-in)

/opt/cross-bin is prepended to PATH but contains only triplet-prefixed tool names (<triplet>-gcc, <triplet>-ld, …), which can never shadow the host toolchain. Bare names (gcc, cc, as, ld, …) live in /opt/cross-bin/bare, which is deliberately not on PATH — bare cross names fronting PATH historically broke every host-side compile (e.g. the riscv64 host-protoc “Exec format error” bug). The few consumers that genuinely need bare names (gcc -B tool lookup, rust cc-crate fallbacks) opt in per scope via cross_bare_bin_path():

bare="$(cross_bare_bin_path)" && exec "${CC}" -B"${bare}/" "$@"

The Android ABI is a target, not the build host

ANDROID_TARGET_ABI (01-core/versions.env, default arm64-v8a) names the ABI the prebuilt SDKs under /opt/android are compiled for. It used to be derived from arch_oci — the build host — and every cross stage in this repo builds on amd64. The whole Android layer therefore shipped as x86_64, which is emulator-only, while the app asked for arm64-v8a. Nothing failed loudly; the payload was simply for the wrong machine.

android_target_arch now prefers the declared ABI and falls back to arch_oci only when nothing declared one:

ABI	OCI arch
arm64-v8a	arm64
x86_64	amd64
x86	386
riscv64	riscv64

arch_for_android_abi returns non-zero on anything else rather than guessing, so an unmappable ABI is a build failure and not a silently mis-targeted layer.

One ABI per image. The ABI is part of the cerbero cache-mount id in Dockerfile.android, so changing it rebuilds the GStreamer Android payload rather than reusing another ABI’s state — set it and rebuild the layer to target a different device.

Android SDK environment in the runtime image

Until CON96 (2026-10-13) Dockerfile.package COPYed /opt/android-sdk unconditionally, so the tree was in all three shipped images; since then only amd64 keeps it, unless HOST_PAYLOAD_ARCHES names another arch. Measured 2026-09-04: 3.6 GB, byte-identical on amd64/arm64/riscv64, with build-tools/36.0.0, cmdline-tools/latest, licenses, ndk/29.0.14206865, platforms/android-36 and platform-tools all present. Since 2026-09-17 platforms/android-37.0 and build-tools/37.0.0 are installed alongside them (CON5): API 37’s base platform package is platforms;android-37.0 in Google’s repository — there is no platforms;android-37, and only API 36 kept the un-suffixed hash — so a consumer raising compileSdk to 37 finds the level without the image dropping 36. Nothing advertised it until then, so flutter build apk stopped at [!] No Android SDK found; under CodeQL’s database create --command=… that exit aborts before the database is finalized and the lane finally reports a missing bundle directory, three steps from the cause. The image now sets:

variable	value	true on
ANDROID_HOME	/opt/android-sdk	amd64; empty on arm64 and riscv64 since 2026-10-13
ANDROID_SDK_ROOT	/opt/android-sdk	amd64; empty on arm64 and riscv64 since 2026-10-13

PATH gains ${ANDROID_HOME}/cmdline-tools/latest/bin and ${ANDROID_HOME}/platform-tools, appended, where Dockerfile.android prepends. Appending is the whole non-shadowing argument: platform-tools ships its own mke2fs, make_f2fs and sqlite3, and fronting PATH with it would put an x86-64 mke2fs ahead of /usr/sbin/mke2fs on every arch. Appended, those directories can only win names nothing else provides — adb, sdkmanager, avdmanager, apkanalyzer, none of which were on PATH before. build-tools is deliberately left off: it carries a renderscript-era lld that would shadow /usr/bin/lld on a compiler image. Gradle and AGP find build-tools and the NDK under $ANDROID_HOME without help. Measured on all three shipped images with the new value injected: mke2fs still resolves to /usr/sbin/mke2fs and lld to /usr/bin/lld, while adb, sdkmanager, avdmanager and apkanalyzer resolve for the first time. The one other new name is sqlite3, which nothing on PATH provided before and which — like adb — only executes in the amd64 image.

The SDK’s host tooling is linux-x86_64 only, on every arch. adb, fastboot, aapt2 and zipalign are x86-64 ELF and the NDK’s only prebuilt toolchain is linux-x86_64, so on arm64/riscv64 they are data, not programs (adb there exits cannot execute: required file not found). That is upstream’s shape, not a packaging defect, and it is why /opt/android-sdk is already an _RT_TREE_ARCH_EXEMPT tree — artifact-copy-completeness.md. Build Android artifacts from the amd64 image. Since 2026-10-13 the arm64 and riscv64 images ship the SDK directories empty and the variables above empty, so PATH gains nothing there (why).

Non-amd64 build hosts

The paragraph above is about the target arch. There is a second, independent question: what happens when the build host itself is not amd64 — a native arm64 or riscv64 machine running the chain. Google publishes the NDK only as toolchains/llvm/prebuilt/linux-x86_64, so on such a host the payload cannot be installed at all, and commandlinetools-linux-<ver>_latest.zip has no riscv64 counterpart either.

Turning the payload off must not turn the build red. The android stage still builds, still tags, still captures its pin and still exports its OCI layout — it just carries an empty payload. The stage graph never changes shape, so runtime, package and wrapper keep their single artifact source and no COPY has to become conditional.

android_build_host_supported() (platform.sh) is the one owner of the decision: [ "$(build_arch_oci)" = "amd64" ]. Everything else reads its answer rather than asking the arch again.

Consumer	amd64 build host	non-amd64 build host
android-sdk.sh	installs SDK + NDK	writes /opt/android/.android-payload-off, exits 0
android-build-preamble.sh (5 lib stages)	builds	exits 0 via the same predicate
Dockerfile.android	populated /opt/android/*	the same directories, empty (they are mkdir’d after the script returns)
smoke-android.sh	runs all seven checks	reads the marker, prints SKIP per check, exits 0
runtime_host_payload (runtime lane)	HOST_PAYLOAD per HOST_PAYLOAD_ARCHES	off for every arch: its ARTIFACT_PLATFORM is not linux/amd64, so the artifact has no SDK, and Dockerfile.package leaves ANDROID_HOME empty
swap-native-gcc.sh	swaps in the Canadian cross GCC	target == build host, so the host GCC already IS native
cross_android_tag	:cross-android-<arch>	:cross-android-host<host>-<arch>

Two of those rows are the parts that are easy to get wrong:

The marker is written, not re-derived. smoke-android.sh deliberately does not source platform.sh — it reads the file. Its NDK check hard-codes the prebuilt/linux-x86_64 path, which on riscv64 can never exist; asking the arch a second time there would also flip its ELF helpers away from the inline fallbacks smoke-arch-parity pins.

The GCC swap keys on the build host, not on the literal amd64. gcc.sh builds the Canadian /opt/gcc-<ver>-native-<arch> only when target != build_arch_oci; when they are equal it takes link_amd64_host_as_cross and produces no such prefix. A consumer testing the literal therefore demanded, on a native arm64 host, exactly the prefix its own toolchain image deliberately does not build.

The tag carries the build host because every cross stage is always pushed (--no-push is the real toggle). Without the infix, a native arm64 run would overwrite the amd64 lane’s real android artifact under the same name. The infix is empty on amd64, so that lane’s tags are byte-identical to before.

What this does not deliver. android stops being the reason a non-amd64-hosted run fails; it does not make the chain finish. On riscv64 apt.llvm.org publishes no packages at all, so the compiler stage builds LLVM/Clang from source there (02-toolchain/llvm.sh).

Superseded, 2026-09-15. This paragraph used to name two more blockers — that the runtime lane “still pins --platform linux/amd64 for its artifact source” and “asks binfmt for a handler name that does not exist”. Both were fixed the SAME DAY this was written (14:16) and the sentence was never updated: ec69a976 (17:00) made runtime_artifact_platform ask cross_build_platform instead of freezing the literal, and d3566550 (20:08) taught _binfmt_qemu_name that the amd64 handler is qemu-x86_64. test-tag-naming.sh and test-native-build-host.sh pin both today.

Two further arm64-host blockers were fixed on 2026-09-15 and are worth knowing because each one failed BEFORE any build started:

• lint-shell.sh’s shellcheck bootstrap had no Linux/aarch64 arm, so it err’d with “Unsupported platform” — make preflight and make lint, the repo’s own entry gate, could not run at all on an ARM host.

• setup-rootless-binfmt.sh defaulted to the literal arm64,riscv64. Run by hand on an arm64 host — the exact command verify_foreign_binfmt’s error tells you to run — that registered a QEMU handler for the host’s OWN arch (binfmt_misc is consulted for native ELF too) and left amd64, the one arch that host must emulate, unregistered. The default is derived from uname -m now; the amd64 lane’s answer is unchanged.

Found only by RUNNING it (2026-09-15, Jetson AGX Orin). The two fixes above were unit-tested and still could not work. setup-rootless-binfmt.sh unpacks its qemu-user emulators from tonistiigi/binfmt, and extract_emulators asked for --platform linux/amd64 unconditionally. A qemu-user binary is a host-arch executable that interprets foreign code, so on arm64 that is wrong twice over: every extracted binary is x86-64 ELF and cannot exec (readelf reported Advanced Micro Devices X86-64), and the amd64 image ships no qemu-x86_64 at all — an x86_64-on-x86_64 emulator is pointless, so upstream ships aarch64/arm/i386/riscv64 instead, and qemu-x86_64 is precisely the one an arm64 host needs. Two more surfaced behind it:

• the cleanup trap ... EXIT dereferenced a function local, which is out of scope when the trap fires at script exit — under set -u that killed the trap with tmp: unbound variable and masked the real error;

• the pull was guarded by nerdctl image inspect, which answers “is this REFERENCE present”, not “is this PLATFORM present”. With a multi-arch index and one platform already cached it skipped the pull, and image save then died with content digest sha256:…: not found.

All three are pinned by test-native-build-host.sh, including tripwires that the frozen linux/amd64 literal and the local-in-trap cannot return. Verified afterwards on that host: nerdctl run --platform linux/arm64 → aarch64, linux/amd64 → x86_64, linux/riscv64 → riscv64.

Run end to end, 2026-09-21/22 (Jetson AGX Orin). The chain built natively for arm64 only, base through runtime, with the NVIDIA layer inserted by hand, and the runtime image ran on the Orin’s GPU. The sequence, the knobs and what is still open (no NVIDIA stage in the chain, no Android payloads on an arm64 host): linux-accelerator-images.md § NVIDIA on arm64 (SBSA).

A JDK ships on every arch. Dockerfile.package installs JDK_PACKAGE (versions.env, openjdk-21-jdk-headless) and sets JAVA_HOME=/usr/lib/jvm/default-java, so the SDK’s Java wrappers (sdkmanager, avdmanager, apkanalyzer, d8, apksigner) and flutter build apk find one. The image used to carry the SDK and no Java at all: consumer-image-contract.md § The Android lane needs a JDK.

The built image is asserted by smoke-runtime-image.sh’s android-home and jdk consumer-contract rows; that the Dockerfile ever sets ANDROID_HOME, and appends rather than fronts, is asserted by tests/test-dockerfile-env-order.sh.

Cross Python wheels (setuptools knobs)

A hand-rolled setup.py bdist_wheel cross-builds correctly only when these are set — setuptools itself reads them (setuptools/_distutils/compilers/C/unix.py configure_system, setuptools/command/build_ext.py get_ext_filename). Users: 03-media/build/pyav/build-pyav.sh, 05-frameworks/torch/build-app-wheelhouse.sh, 03-media/build/onnxruntime/build/60-build-genai.sh.

onnxruntime-genai on riscv64 (GEN1, 2026-08-31)

Full reference: gen1-riscv64-genai.md — the patch, the toggle wiring, the smoke tiers, the first-build watch list, and what is MEASURED versus REASONED. This section is the summary.

The riscv64 cross lane BUILDS onnxruntime-genai from source. Upstream ships no riscv64 wheel, runs no riscv64 CI, and closed its one RISC-V field report (onnxruntime-genai#594, Phi-3 int4 emitting nonsense on a XuanTie C910) as not-planned — so this is a self-build. Validated 2026-09-03: it builds, and generate() on the shipped image emits ids identical to an amd64 control, so #594 does not reproduce here — gen1-riscv64-genai.md.

• Toggle: GENAI_ALLOW_RISCV64 (versions.env, default true; Dockerfile.media ARG → ENV on the onnxruntime stage). Anything but true restores the pre-GEN1 behaviour exactly: 60-build-genai.sh creates a placeholder output tree and exits 0, and verify-media-artifacts.sh SKIPs the genai stage in agreement. Deliberately riscv64-scoped — BUILD_GENAI is the all-arch master switch and would take amd64/arm64 down with it.

• Upstream patch: linux/scripts/patches/onnxruntime-genai/001-riscv64-target-platform.patch adds a riscv64 arm to cmake/target_platform.cmake, whose Linux branch otherwise FATAL_ERRORs at configure time on an unknown CMAKE_SYSTEM_PROCESSOR. The variable it sets (genai_target_platform) is read only by MSVC/WinML/Java code, so the patch is inert on this lane apart from removing the abort. Applied only when ARCH=riscv64.

• --use_guidance stays ON (llguidance via Corrosion — riscv64gc-unknown-linux-gnu is a rustc Tier-2-with-host-tools target and install-rust.sh adds its std for every CROSS_TARGETS arch). A riscv64-only preflight drops the flag if rustup target list --installed positively lacks the triple, so a missing std warns instead of aborting a multi-hour stage.

• Performance note: ORT’s riscv64 MLAS takes its RVV kernels only under onnxruntime_USE_RVV and an rv64gcv compiler. When GEN1 landed this repo set neither, so MLAS ran its SCALAR reference kernels. The RVA23 baseline now sets both on riscv64 (riscv64-rva23-baseline.md).

• Gates: the producer’s own cross wheel check (assert_elf_arch over every .so in the wheel + the target EXT_SUFFIX assert), verify-media-artifacts.sh onnxruntime-genai, smoke-torch-venv.sh’s pin assertion (riscv64 absence is now a FAILURE unless the toggle is off), the ARCH-PARITY table (its riscv64:onnxruntime_genai exemption is deleted), and smoke-runtime-image.sh’s check_genai_binding.

Variable	Why it is load-bearing
CC	Replaces the interpreter’s compiler. May be multi-word (ccache <cross-gcc>) — set_executables shlex-splits it.
LDSHARED	Explicit <cross-gcc> -shared. Without it the link command is derived from the host interpreter’s sysconfig.
CFLAGS	Replaces (does not append to) the sysconfig CFLAGS, and lands before the extension’s own -I dirs — which is what makes the TARGET Python headers win over the host ones build_ext always appends. Must carry -O2 itself, since the sysconfig optimisation flags are replaced with it.
SETUPTOOLS_EXT_SUFFIX	The target SOABI suffix. Without it extensions are named .cpython-<mm>-x86_64-linux-gnu.so, which the target interpreter never even considers at import time (importlib EXTENSION_SUFFIXES) — the wheel installs and the import dies. runtime/verify-wheels.sh asserts the suffix.
_PYTHON_HOST_PLATFORM	Makes get_platform(), and hence the wheel platform tag, the target one — so the wheel is born correctly tagged and runtime/repair-wheels.sh’s blanket retag is only a safety net. uv must never see it: uv inspects the interpreter through sysconfig.get_platform() and refuses Unknown operating system: linux_riscv64 at both uv venv and uv build, so the cross path builds the wheel with the venv’s python -m pip wheel and sets the variable on that command alone (the sdist stays on uv build --sdist).

Runtime lane helper commands

Building and publishing the per-arch wrappers and the multi-arch manifest, plus manifest repair. The canonical chain commands are in § The command reference AGENTS.md used to carry; these are the runtime-lane half.

# Build and push per-arch wrappers + manifest
bash linux/scripts/build-runtime-manifest.sh \
  --image ghcr.io/kataglyphis/kataglyphis_beschleuniger:latest \
  --target-arches amd64,arm64,riscv64 \
  --artifact-image-prefix ghcr.io/kataglyphis/kataglyphis_beschleuniger:cross-android \
  --push

# Build local artifacts only (no push)
bash linux/scripts/build-runtime-artifacts.sh \
  --image-prefix ghcr.io/kataglyphis/kataglyphis_beschleuniger:latest \
  --target-arches amd64,arm64,riscv64

# Dry-run: print what would be built without executing
bash linux/scripts/build-runtime-manifest.sh \
  --image ghcr.io/kataglyphis/kataglyphis_beschleuniger:latest \
  --target-arches amd64,arm64,riscv64 --dry-run

# Manifest repair (rebuild manifest from existing per-arch wrappers)
nerdctl manifest rm "ghcr.io/kataglyphis/kataglyphis_beschleuniger:latest" >/dev/null 2>&1 || true
nerdctl manifest create "ghcr.io/kataglyphis/kataglyphis_beschleuniger:latest" \
  "ghcr.io/kataglyphis/kataglyphis_beschleuniger:latest-amd64" \
  "ghcr.io/kataglyphis/kataglyphis_beschleuniger:latest-arm64" \
  "ghcr.io/kataglyphis/kataglyphis_beschleuniger:latest-riscv64"
nerdctl manifest push --purge "ghcr.io/kataglyphis/kataglyphis_beschleuniger:latest"

# Or use the helper: rebuild just the manifest (no image rebuilds)
bash linux/scripts/build-runtime-manifest.sh \
  --image ghcr.io/kataglyphis/kataglyphis_beschleuniger:latest \
  --target-arches amd64,arm64,riscv64 --manifest-only --push-manifest

# Shorthand: --repair is an alias for --manifest-only
bash linux/scripts/build-runtime-manifest.sh \
  --image ghcr.io/kataglyphis/kataglyphis_beschleuniger:latest \
  --target-arches amd64,arm64,riscv64 --repair --push-manifest

The app the wrapper builds

The torch stage builds OrchestrANT’s develop branch (owner decision 2026-09-29). Fixes the build needs go to develop directly; there is no release tag to cut. versions.env APP_REF=develop names what to track, not what to build:

• The commit, resolved once per run. build-runtime-manifest.sh and build-runtime-artifacts.sh call runtime_resolve_app_ref before the per-arch loop. It resolves the ref with git ls-remote and exports APP_REF=<commit>, which the versions.env forwarding then passes to every arch. All arches of one index build the same app, even if develop moves mid-run. A branch wins over a same-named tag, and an annotated tag gives the commit it peels to.

• Why a commit, never the name. BuildKit keys the torch RUN on its build args. With the arg fixed at develop, the cache would reuse the first clone forever and ship a stale app behind a green build. With the commit as the arg, the layer rebuilds exactly when develop moved and stays cached when it did not.

• The fetch. assemble-torch-app.sh fetches a 40-hex APP_REF by itself (git init + fetch --depth 1 <commit> + checkout FETCH_HEAD); clone --branch takes names only. A name still clones, with a warning, for a plain docker build that runs on the Dockerfile default.

• The record. The commit is the image’s org.opencontainers.image.version label. To rebuild an old image exactly, export APP_REF=<that commit> (or a tag) before the run; a 40-hex value is built as given, with no lookup.

• Failure. An unreachable remote or an unknown ref stops a real run before anything builds; --dry-run only warns. A broken develop fails the torch stage near the end of the chain, with every earlier stage cached.

The Windows lane does the same in Resolve-TorchAppRef and Build-TorchApp.ps1: windows-builds.md.

The chain wheels and the app’s lock

OrchestrANT’s lock resolves every package from PyPI and the PyTorch indexes, torch included: the 2026-10-01 chain logged torch pins satisfied on amd64 and arm64, with no override. The chain’s own wheels in /opt/wheels replace the lock’s entries of the same family: ONNX Runtime (under its flavour name), GenAI, ai-edge-litert, IREE, OpenCV and a local torch. install_project_environment runs four steps in this order:

1. uv sync names each replaced family in --no-install-package (collect_locked_local_skip_packages, plus GenAI in build_uv_sync_args). It installs nothing else first: an exact sync removes what it was told not to install, so a wheel installed before it was removed and installed again, the detour of the 2026-10-01 chain (ai-edge-litert, onnxruntime-genai).

2. reconcile_local_wheels is the one place that installs the store, --no-deps.

3. enforce_torch_version_pins.

4. assert_chain_wheels_installed fails the stage unless every staged wheel is what the venv has, by the direct_url.json uv records (failure-modes.md). assert_ort_chain_only then proves the ORT bytes.

Why the lock cannot name the store. A [tool.uv.sources] entry routed to a flat index at /opt/wheels makes uv lock fail everywhere the store is absent, which is OrchestrANT’s CI and every dev host: Failed to read --find-links directory: /opt/wheels (exit 2, measured 2026-10-01 with the image’s uv 0.12.17). A source marker cannot help, since PEP 508 has no “inside the image” marker. And a lock that did resolve there would record each chain wheel’s SHA256, which the next chain rebuilds.

Orchestrator stage selection

Resuming mid-chain, building one stage, and the parallel-arch knobs.

# Resume mid-chain (e.g., after rebuilding compiler)
bash linux/scripts/build-cross-chain.sh --from-stage sdk --target-arches amd64,arm64,riscv64 --log-dir ./out/build-logs

# Build only one stage for one architecture
bash linux/scripts/build-cross-chain.sh --only media --target-arches arm64 --log-dir ./out/build-logs

# Build per-arch stages in parallel (faster on multi-core machines).
# PAR4 (VALIDATED through wave-4, 2026-08-21): cross_build_mem_divisor
# multiplies the arch count by PAR_INTRA_STEP_BUDGET (default 2) for
# PER-ARCH stages, and uses divisor 1 for SHARED stages (base/compiler run
# alone — the ×budget throttled the compiler to 1/3 jobs once). Across ~12
# parallel media rounds: ONE isolated OOM kill, absorbed by retries. If a
# lane OOMs repeatedly raise PAR_INTRA_STEP_BUDGET=3 or use
# PARALLEL_STAGES=sdk,android. Details: docs/build-parallelism-memory-tuning.md.
bash linux/scripts/build-cross-chain.sh --target-arches amd64,arm64,riscv64 --parallel-archs --log-dir ./out/build-logs

# Build a single cross stage standalone (with digest-pinned parent when --push)
bash linux/scripts/build-cross-stage.sh --stage sdk --arch arm64 --push --log-dir ./out/build-logs

The Android ABI is a target, not the build host

/opt/android holds five prebuilt Android SDKs — GStreamer, ONNX Runtime, LiteRT, OpenCV, IREE. Each is compiled for exactly one Android ABI, and until 2026-09-05 that ABI was derived from arch_oci: the architecture of the machine doing the build. Every cross stage builds on linux/amd64, so the answer was always x86_64, on every image, including the arm64 and riscv64 ones.

The symptom reached a consumer as a linker error, not a missing file — the SDKs were all present and all wrong:

ld.lld: error: /opt/android/gstreamer/libgstreamer-1.0.a(gst.c.o)
        is incompatible with aarch64linux

x86_64 is the emulator ABI. An app targeting real phones asks for arm64-v8a, so the layer was useful to almost nobody.

ANDROID_TARGET_ABI in versions.env now names the target, defaulting to arm64-v8a. android_target_abi() reads it, and android_target_arch() maps it BACK to a Debian arch through arch_for_android_abi so the two never disagree — the preamble’s TARGET_ARCH drives cerbero’s CERBERO_TARGET_ARCH, the API-level floor, and ONNX Runtime’s riscv64 skip. With the knob unset the old host-derived behaviour is unchanged.

One ABI per image. Switching costs a full rebuild of the android stage, which is why the five cerbero cachemounts carry abi${ANDROID_TARGET_ABI} in their id: without it a switched ABI would resume the previous ABI’s cerbero tree and spend hours producing the wrong thing, the same class of trap the NDK version key already guards.

To build the emulator ABI instead:

ANDROID_TARGET_ABI=x86_64 bash linux/scripts/build-cross-chain.sh --only android