Linux Cross Builds

Linux Cross Builds#

See also: docs/linux-build-basics.md for build fundamentals, caching, and troubleshooting. AGENTS.md for agent guardrails and the full repo map.

Build-time download speed: the cross-compiler/SDK builds fetch the LLVM source with git inside a RUN step. On this host that is fast because rootless BuildKit runs with --oci-worker-net=host (host networking for RUN steps). Registry mirrors do not help that git fetch; the host-net setting does. See docs/project-info.md for the drop-in config and AGENTS.md for the do-not-regress note. For repeated LLVM rebuilds, prefer caching the source on the host over re-fetching.

Build logging: build-cross-chain.sh and build-cross-stage.sh accept --log-dir ./out/build-logs to write per-stage build logs. The other orchestrators (build-cross-compiler.sh, build-runtime-manifest.sh, build-runtime-artifacts.sh) do not — capture their output, and any manual nerdctl build, with 2>&1 | tee ./out/build-logs/<name>.log. The standard location for build logs is out/build-logs/.

Cross-Compiler builder (nerdctl, amd64 host; amd64/arm64/riscv64 targets)#

The existing multi-platform build above stays unchanged. Treat it as the compatibility lane for the current QEMU/binfmt-based end-to-end build.

The cross-compiler path below is additive. It does not replace the existing QEMU workflow. Instead, it prepares a single amd64-hosted builder image that contains cross toolchains for amd64, arm64, and riscv64 for a future artifact-based multi-architecture end-to-end build.

This lane intentionally builds only a linux/amd64 container image. The three architectures are the compiler targets installed inside that image via CROSS_TARGETS=amd64,arm64,riscv64, not three separate compiler container manifests. This image is a single amd64 builder image, not a replacement for the full multi-platform Linux chain yet. It keeps the current native/emulated flow intact while adding source-built GCC 16 target compilers like x86_64-linux-gnu-gcc, aarch64-linux-gnu-gcc, and riscv64-linux-gnu-gcc, plus convenience wrappers such as clang-amd64, clang-arm64, and clang-riscv64 for host-side cross builds.

For the cross-compiler path, the helper can bootstrap the base image locally when needed, so you do not have to rely on a remote base intermediate tag surviving in GHCR.

Fastest entry point:

./linux/scripts/build-cross-compiler.sh --cross-targets amd64,arm64,riscv64 --fast-ubuntu-mirror \
  --fast-ubuntu-mirror-url http://de.archive.ubuntu.com/ubuntu/

Use --fast-ubuntu-mirror-url URL to override the default mirror (https://archive.ubuntu.com/ubuntu/). For example: --fast-ubuntu-mirror-url http://de.archive.ubuntu.com/ubuntu/.

The helper script only uses nerdctl. It first tries to reuse a local image, then tries to pull from the registry, and if that fails it rebuilds the base image locally before building the compiler image. It only pushes when you pass --push. Internally the script delegates to the shared stage graph (stage-defs.sh) and build helpers — the same infrastructure used by the full orchestrator. The --image-repo flag switches the registry prefix; there are no legacy env var overrides.

If you only need the downstream SDK or media cross stages and want to reuse the published compiler image, pull it first:

mkdir -p ./out/build-logs && \
nerdctl pull --platform linux/amd64 \
  ghcr.io/kataglyphis/kataglyphis_beschleuniger:cross-compiler-amd64 \
  2>&1 | tee ./out/build-logs/pull-compiler.log

Build the local amd64 base image:

LOG_DIR="logs/$(date -u +'%Y%m%dT%H%M%SZ')-cross-base"
mkdir -p "${LOG_DIR}"

nerdctl build --platform linux/amd64 -t ghcr.io/kataglyphis/kataglyphis_beschleuniger:base \
  --output 'type=image,name=ghcr.io/kataglyphis/kataglyphis_beschleuniger:base,push=true' \
  -f linux/Dockerfile.base \
  --build-arg USE_FAST_UBUNTU_MIRROR=true \
  --build-arg FAST_UBUNTU_MIRROR_URL=http://de.archive.ubuntu.com/ubuntu/ \
  . 2>&1 | tee "${LOG_DIR}/base.log"

Then build the dedicated amd64-hosted compiler image in cross mode for amd64, arm64, and riscv64 targets:

LOG_DIR="logs/$(date -u +'%Y%m%dT%H%M%SZ')-cross-compiler"
mkdir -p "${LOG_DIR}"

nerdctl build --platform linux/amd64 -t ghcr.io/kataglyphis/kataglyphis_beschleuniger:cross-compiler-amd64 \
  --output 'type=image,name=ghcr.io/kataglyphis/kataglyphis_beschleuniger:cross-compiler-amd64,push=true' \
  -f linux/Dockerfile.toolchain \
  --build-arg BASE_IMAGE=ghcr.io/kataglyphis/kataglyphis_beschleuniger:base \
  --build-arg USE_FAST_UBUNTU_MIRROR=true \
  --build-arg FAST_UBUNTU_MIRROR_URL=http://de.archive.ubuntu.com/ubuntu/ \
  --build-arg BUILD_MODE=cross \
  --build-arg CROSS_TARGETS=amd64,arm64,riscv64 \
  . 2>&1 | tee "${LOG_DIR}/cross-compiler-amd64.log"

The explicit nerdctl build --output ... push=true commands above already push the intermediary images to GHCR. Only the helper script keeps the images local by default unless you pass --push.

Expected compiler result inside that image:

  • gcc and g++ resolve to /opt/gcc-16.2.0/bin/* and report GCC 16.x on the amd64 host compiler path.

  • x86_64-linux-gnu-gcc, aarch64-linux-gnu-gcc, and riscv64-linux-gnu-gcc resolve to /opt/gcc-16.2.0/bin/* and report GCC 16.x.

  • clang-amd64, clang-arm64, and clang-riscv64 still exist, but now point Clang at /opt/gcc-16.2.0 as the GCC toolchain root.

Expected result: the build log ends with ghcr.io/kataglyphis/kataglyphis_beschleuniger:cross-compiler-amd64. That is correct for this cross lane because the builder container itself runs on amd64 while shipping source-built GCC 16 host and cross compilers for all three target architectures.

Or let the helper do the push too:

./linux/scripts/build-cross-compiler.sh --cross-targets amd64,arm64,riscv64 --fast-ubuntu-mirror --push

Manual staged build (low-level reference)#

For full hands-off builds, prefer the orchestrator build-cross-chain.sh with digest-pinned stage handoff. Each cross stage maps to one Dockerfile and one tag; the orchestrator and build-cross-stage.sh handle the per-arch fan-out, build arg assembly, and pin capture for you. Use the helpers unless you are debugging a specific stage in isolation.

If you must drive individual builds manually, refer to the helper scripts for the canonical argument set (or run them with --dry-run to print the commands they would execute). The essential pattern for each cross-lane stage is:

nerdctl build --platform linux/amd64 --pull=true \
  --output 'type=image,name=<tag>,push=true' \
  -f <dockerfile> \
  --build-arg BASE_IMAGE=<parent_tag_or_pinned_digest> \
  --build-arg BUILD_MODE=cross \
  [--build-arg TARGET_ARCH=<arch> if per-arch] \
  .

For the runtime lane, prefer build-runtime-manifest.sh. The helper handles the base -> package -> torch chain and multi-arch manifest creation. --manifest-only / --repair can recreate the manifest from existing per-arch wrappers without rebuilding any images.

linux/Dockerfile.sdk serves both the sequential SDK build and the amd64-hosted cross SDK artifact lane. The cross path consumes one TARGET_ARCH per nerdctl build, fanned out per architecture.

linux/Dockerfile.package is the handoff point where amd64-hosted cross artifacts are copied into a clean target-native root filesystem. For foreign-architecture images, the package stage must receive:

  • A target-native /opt/llvm-target tree, wired to /usr/bin/clang

  • A target-native /opt/gcc-16.2.0 (cross-compiled from source via Canadian cross, swapped in by Dockerfile.android)

  • A hard-fail CC validation guard (dumpmachine, ELF type, cc1 smoke test)

linux/Dockerfile.torch produces the final :latest-cross-<arch> wrapper images (torch venv, app, runtime scripts, entrypoint). The per-arch wrappers are assembled into the :latest-cross multi-arch manifest.

OpenCV 5.x GStreamer compatibility (applies to all architectures)#

OpenCV 5.x reorganized several modules relative to OpenCV 4.x. GStreamer’s bundled gst-plugins-bad “opencv” plugin (1.29.x) still targets the 4.x layout, so it fails to compile against the source-built OpenCV 5 in this image. The build system applies an automatic source patch via patch-gstreamer-sources.sh → patch_gstreamer_sources() that addresses three upstream API changes:

  1. contourArea/approxPolyDP/convexHull moved from imgproc to the new geometry module → adds #include <opencv2/geometry.hpp> to gstsegmentation.cpp.

  2. findChessboardCorners/findCirclesGrid/drawChessboardCorners + CALIB_CB_* moved from calib3d into objdetect → adds #include <opencv2/objdetect.hpp> to gstcameracalibrate.cpp.

  3. cv::CascadeClassifier + CASCADE_* (legacy Haar cascade detection) were removed from OpenCV 5 → the three cascade-dependent GStreamer elements (faceblur, facedetect, handdetect) are dropped from the monolithic libgstopencv.so. The remaining 22 elements (dilate, sobel, smooth, edgedetect, tracker, grabcut, retinex, segmentation, cameracalibrate, etc.) build and function normally.

Additionally, build-opencv.sh creates an opencv4.pc → opencv5.pc compatibility alias because GStreamer’s meson dependency lookup queries dependency('opencv4', '>= 4.0.0').

SDK rootfs artifacts (first host-side build step)#

The first additive artifact path is now the SDK stage. It reuses linux/Dockerfile.sdk in BUILD_MODE=cross, builds target-specific SDK root filesystems for amd64, arm64, and riscv64 on a fast amd64 host, and exports them to disk while the existing QEMU/binfmt multi-platform build above remains unchanged.

Build the first SDK artifacts for amd64, arm64, and riscv64 while saving this run under one timestamped logs/ directory:

set -o pipefail
LOG_DIR="logs/$(date -u +'%Y%m%dT%H%M%SZ')-sdk-artifacts"
mkdir -p "${LOG_DIR}"

./linux/scripts/build-sdk-artifacts.sh --target-arches amd64,arm64,riscv64 --fast-ubuntu-mirror \
  --fast-ubuntu-mirror-url http://de.archive.ubuntu.com/ubuntu/ \
  2>&1 | tee "${LOG_DIR}/build-sdk-artifacts.log"

For individual SDK artifact builds, use build-cross-stage.sh:

for arch in amd64 arm64 riscv64; do
  bash linux/scripts/build-cross-stage.sh --stage sdk --arch "${arch}" --push --log-dir ./out/build-logs
done

If you want this helper to reuse the published compiler image instead of bootstrapping it locally, pull the compiler tag first:

mkdir -p ./out/build-logs && \
nerdctl pull --platform linux/amd64 \
  ghcr.io/kataglyphis/kataglyphis_beschleuniger:cross-compiler-amd64 \
  2>&1 | tee ./out/build-logs/pull-compiler.log

The helper accepts TARGET_ARCHES=amd64,arm64,riscv64, TARGET_ARCH=amd64,arm64,riscv64, or --target-arches amd64,arm64,riscv64 and then fans that list out into one TARGET_ARCH=<arch> build per target.

Expected output layout:

out/linux-sdk/amd64/rootfs/
out/linux-sdk/amd64/artifact.env
out/linux-sdk/arm64/rootfs/
out/linux-sdk/arm64/artifact.env
out/linux-sdk/riscv64/rootfs/
out/linux-sdk/riscv64/artifact.env

This helper uses linux/Dockerfile.sdk with BUILD_MODE=cross and the amd64-hosted cross compiler image. During successful cross SDK builds, CMake should identify the active C++ compiler as GNU 16.2.0 rather than the Ubuntu 26.04 system GCC toolchain. It is the first real host-side rootfs export step toward a full multi-architecture non-QEMU end-to-end build, but it does not yet replace the full :latest pipeline.

linux/Dockerfile.sdk also forwards the checked-in LLVM_RELEASE pin into the target-clang step, so rebuilding an SDK artifact from an older cross-compiler-amd64 base still refreshes /opt/llvm-target to the repository pin instead of inheriting a stale base-image environment value.

Cross packaging to multi-arch manifest (experimental)#

Overview#

The new end-goal path keeps the existing QEMU lane for compatibility while adding:

  1. Cross-compile target artifacts host-side with the cross builder.

  2. Assemble one runtime image per architecture from a clean per-arch linux/Dockerfile.base plus the target-built payload from cross-android-${TARGET_ARCH}.

  3. Publish a single multi-architecture manifest.

linux/Dockerfile.package is the shared runtime packaging layer. It starts from a clean per-arch base, copies the selected target payload from the artifact image, replays final runtime dependency setup, and becomes BASE_IMAGE for linux/Dockerfile.torch. In cross mode the artifact image runs on amd64 (cross-android-${TARGET_ARCH}); in native mode it uses the target-platform sequential image directly.

Host prerequisite: QEMU/binfmt for the emulated runtime legs#

The sdk/media/android stages cross-compile on amd64 and need no emulation by design — but note the registration is load-bearing even there: nested NATIVE tool sub-builds (e.g. IREE’s bundled-LLVM tblgen) historically only survived because qemu silently executed a wrong-arch binary. That specific case is fixed at the source (CROSS_TOOLCHAIN_FLAGS_NATIVE pins the host compilers), but keep binfmt registered — it is the safety net for the whole class.

Lifetime: the registration lives in the rootlesskit namespace and dies on host reboot and on systemctl --user restart containerd (this bit the 2026-08-08 foreign chain: the shim-failure restart earlier that day had silently wiped it, and media-arm64’s IREE build failed with Exec format error). --install-service below installs a systemd –user unit so it re-registers automatically. The runtime stage is different: build-runtime-manifest.sh builds the per-arch base → package → torch wrappers on the real target platform (nerdctl build --platform linux/arm64|riscv64). For foreign architectures those RUN steps (e.g. base-image.sh bootstrap-ca, copy-media-payloads.sh, apt, dpkg) execute under QEMU user-mode emulation, which requires QEMU emulators registered in binfmt_misc in the namespace where builds run.

Rootless setup (this host — no sudo)#

Run the helper once per boot (it is idempotent, and --install-service makes it persistent via a systemd –user unit):

linux/scripts/setup-rootless-binfmt.sh                    # register arm64,riscv64 now
linux/scripts/setup-rootless-binfmt.sh --install-service  # + auto-register on every login/boot
linux/scripts/setup-rootless-binfmt.sh --verify           # check current state

You normally don’t run it by hand: build-runtime-manifest.sh invokes it for you (no sudo) right before the per-arch runtime-image smokes, for whichever target arches are non-native (ensure_foreign_binfmt). Set RUNTIME_REGISTER_BINFMT=0 to skip that auto-registration (e.g. a rootful/CI host where qemu is already registered via docker run --privileged tonistiigi/binfmt or update-binfmts). The standalone invocations above are for registering ahead of time or debugging.

Why the helper is needed (and why the “obvious” commands don’t work rootless):

  • sudo apt install qemu-user-static is not required and not wanted here — this host runs rootless containerd + BuildKit and must stay sudo-free.

  • A plain nerdctl run --privileged --rm tonistiigi/binfmt --install all does NOT work even though it prints arm64 OK. A rootless --rm container registers binfmt inside its own ephemeral user namespace, which is destroyed on exit — the registration never reaches the namespace where real containers/builds run. Symptom: exec format error on any nested exec, e.g. a -d container that returns an ID but immediately exits 255 with exec /docker-entrypoint.sh: exec format error, or a build step dying at uname / apt.

  • The key insight this host relies on: buildkitd is launched nsenter’d into containerd’s rootlesskit namespace (systemctl --user cat buildkit.service → ExecStart=... containerd-rootless-setuptool.sh nsenter -- buildkitd ...), so nerdctl run and nerdctl build share one persistent rootless namespace. The helper registers QEMU once in that shared namespace (entering it the same way, via containerd-rootless-setuptool.sh nsenter), so both emulate correctly. Because binfmt_misc is user-namespace-mountable on this kernel, the helper overmounts a fresh, namespace-owned (writable) binfmt_misc there without any host privilege.

Registration flags matter — the helper uses POCF:

flag

meaning

why it’s needed

P

preserve-argv[0]

critical — without it qemu drops argv[1]; sh -c CMD loses -c and dash treats CMD as a filename (cannot open …: No such file)

O

open-binary as fd

lets qemu run a target that isn’t on the interpreter’s path

C

credentials

setuid/setgid handling

F

fix-binary

kernel opens the interpreter fd at registration time, so it is inherited into nested build/run namespaces where the qemu path isn’t mounted

Symptom in an orchestrator run when binfmt is missing/misregistered: the runtime stage’s arm64/riscv64 legs die with error: failed to solve: process "/dev/.buildkit_qemu_emulator ... bootstrap-ca ..." did not complete successfully: exit code: 1, while amd64 (native, no emulation) succeeds.

Verify emulation actually works (not just “registered”) before a runtime run:

# both should print the target machine, NOT "Exec format error"
nerdctl run --rm --platform linux/arm64  ubuntu:26.04 uname -m   # -> aarch64
nerdctl run --rm --platform linux/riscv64 ubuntu:26.04 uname -m  # -> riscv64

tonistiigi/binfmt’s “OK” output means “a registration was written in my namespace”, not “emulation works”. Always confirm with the run test above — a -d container that returns an ID can still have exited immediately with exec format error.

Rootful hosts#

On a rootful Docker/containerd host the standard docker run --privileged --rm tonistiigi/binfmt --install all (or apt install qemu-user-static) registers in the host binfmt_misc and works directly, because containers there share the host (init) user namespace. The rootless helper above is only needed when the daemon runs rootless.

The per-arch latest-cross-base-*, latest-cross-package-*, and latest-cross-* tags are internal publish tags used to assemble the public latest-cross manifest. Prefer the runtime helpers (see AGENTS.md § Runtime Helpers for the canonical commands). Run with --dry-run to print the commands without building.

Runtime helper scripts#

Two helpers manage the base → package → torch → wrapper → manifest chain:

  • build-runtime-manifest.sh — builds the full chain and publishes the multi-arch manifest.

  • build-runtime-artifacts.sh — builds the chain and exports the final wrapper rootfs instead of creating a manifest.

Both accept --target-arches, TARGET_ARCHES, or TARGET_ARCH for architecture selection, and ARTIFACT_BUILD_MODE=cross|native for the package artifact source. In cross mode, ARTIFACT_IMAGE_PREFIX is a prefix (e.g. ghcr.io/...:cross-android) that fans out -${TARGET_ARCH}; in native mode it is the exact artifact image ref.

The riscv64 app wheelhouse is built on the amd64 host for torch, torchvision, and opencv-python git dependencies and carried through /opt/wheels. The final linux/Dockerfile.torch stage runs on the real target platform in both modes so /opt/venv is correct for the target architecture.

Local handoff behavior:

  • When images stay local, base is exported as a plain rootfs directory, package and torch as OCI layouts consumed through named build contexts.

  • ARTIFACT_CONTEXT_ROOT lets helpers consume previously saved artifacts from disk instead of pulling from a registry.

  • ARTIFACT_CONTEXT_MODE=oci resolves each <arch> within ARTIFACT_CONTEXT_ROOT as oci-layout://... (verified path for out/local-oci/android/{arm64,riscv64}).

  • One build still fails when consuming two named OCI contexts at once; the workaround is runtime_artifact as OCI layout + runtime_base as plain rootfs directory.

  • Each local stage context is deleted after the downstream build consumes it.

  • --manifest-only (alias --repair) creates/pushes the manifest without rebuilding images — the recommended way to repair :latest-cross from existing per-arch wrappers.

Verified local foreign-architecture rebuild#

ARTIFACT_CONTEXT_ROOT="$PWD/out/local-oci/android" \
ARTIFACT_CONTEXT_MODE=oci \
RUNTIME_CONTEXT_ROOT="$PWD/out/local-oci/runtime-contexts" \
bash linux/scripts/build-runtime-artifacts.sh \
  --target-arches arm64,riscv64 \
  --image-prefix docker.io/library/opencode-local:latest-cross \
  --artifact-image-prefix docker.io/library/opencode-local:cross-android \
  --artifact-build-mode cross \
  --fast-ubuntu-mirror \
  --fast-ubuntu-mirror-url http://de.archive.ubuntu.com/ubuntu/ \
  --fast-ubuntu-ports-mirror-url http://ports.ubuntu.com/ubuntu-ports/

Validated for both arm64 and riscv64: gcc 16.2.0, clang 22.1.8, /usr/bin/cc → /etc/alternatives/cc → /opt/gcc-16.2.0/bin/gcc, and optional runtime payloads under /usr/local/lib/onnxruntime-*, /usr/local/include/tflite, /usr/local/include/tensorflow, /usr/local/lib/pkgconfig/litert.pc.

After the runtime helper cleanup, validated for amd64 with:

RUNTIME_CONTEXT_ROOT="/tmp/opencode/runtime-contexts" \
bash linux/scripts/build-runtime-artifacts.sh \
  --target-arches amd64 \
  --output-root /tmp/opencode/runtime-smoke \
  --image-prefix docker.io/library/opencode-local:latest-cross-smoke \
  --artifact-image-prefix ghcr.io/kataglyphis/kataglyphis_beschleuniger:cross-android \
  --artifact-build-mode cross \
  --fast-ubuntu-mirror \
  --fast-ubuntu-mirror-url http://de.archive.ubuntu.com/ubuntu/ \
  --fast-ubuntu-ports-mirror-url http://ports.ubuntu.com/ubuntu-ports/

Result: gcc 16.2.0, clang 22.1.8, target x86_64-unknown-linux-gnu, /usr/bin/cc → /etc/alternatives/cc → /opt/gcc-16.2.0/bin/gcc, /usr/bin/clang → /etc/alternatives/clang → /usr/local/llvm-target/bin/clang.

Local wrapper smoke validation#

mkdir -p ./out/build-logs && \
nerdctl build --platform linux/amd64 \
  -t local/kataglyphis:latest-cross-wrapper-smoke-amd64 \
  -f linux/Dockerfile.package \
  --target wrapper-smoke \
  --build-arg BASE_IMAGE=ghcr.io/kataglyphis/kataglyphis_beschleuniger:base \
  --build-arg ARTIFACT_IMAGE=ghcr.io/kataglyphis/kataglyphis_beschleuniger:cross-android-amd64 \
  --build-arg ARTIFACT_PLATFORM=linux/amd64 \
  --build-arg TARGET_ARCH=amd64 \
  --build-arg BUILD_MODE=cross \
  --build-arg GCC_VERSION=16.2.0 \
  --build-arg LLVM_RELEASE=22.1.8 \
  --build-arg USE_FAST_UBUNTU_MIRROR=true \
  --build-arg FAST_UBUNTU_MIRROR_URL=http://de.archive.ubuntu.com/ubuntu/ \
  --build-arg FAST_UBUNTU_PORTS_MIRROR_URL=http://ports.ubuntu.com/ubuntu-ports/ \
  . 2>&1 | tee ./out/build-logs/wrapper-smoke.log

Centralized Version Management#

All version numbers are now tracked in a single file: linux/scripts/01-core/versions.env. Update this file when bumping versions — do NOT scatter version changes across individual Dockerfiles.

common.sh and artifact-common.sh both source versions.env at load time with set -a, so all build scripts and orchestrators automatically receive canonical values. The per-Dockerfile ARG defaults are kept as safety nets and should match versions.env.

After bumping versions, run python3 docs/scripts/sync_versions.py --write to update the version snapshot in README.md.

versions.env feature toggles (Linux lane)#

Besides pins, versions.env carries feature switches for optional, build-cost-heavy capabilities. They are probe-gated: turning one on never hard-fails a build where the dependency is genuinely unavailable — the probe falls back to disabled. Current toggles:

Toggle

Effect

Notes

FFMPEG_ENABLE_X265

libx265 (HEVC) encoding in FFmpeg

Probe-gated; historically off because FFmpeg master could fail against bleeding-edge x265.

ORT_ENABLE_WEBGPU

ONNX Runtime WebGPU EP (Dawn)

Master switch; Dawn needs the GCC-16 -Wno-invalid-constexpr fix (2026-07-20).

ORT_WEBGPU_ALLOW_CROSS

Allow the WebGPU EP on cross arches

Dawn cross-build is the risky part; amd64-only unless set.

Not yet a toggle (planned, backlog S2): the FFmpeg TensorFlow DNN backend currently auto-enables whenever the TF C SDK downloads (amd64 only) and bundles ~500 MB of libtensorflow* into the image; FFMPEG_ENABLE_TF (default off, mirroring x265) is queued for the next versions.env window.

Because versions.env sits in the media build’s cache-key closure, toggle flips re-run the affected media compiles — batch them with planned pin bumps (see docs/refactoring-backlog.md, standing rules).

IREE (Linux lane)#

IREE (IREE_VERSION in versions.env, currently v3.11.0) ships 3-arch in the media image since 2026-07-14, with a deliberately split strategy per arch:

  • amd64 / arm64 — upstream PyPI cp312-abi3 wheels, installed into the Python 3.14 venv (abi3 makes the cp312 tag valid there). No source build.

  • riscv64 — no upstream wheel exists; IREE is source-built, RUNTIME-ONLY (IREE_BUILD_COMPILER=OFF for the target — consistent with upstream’s own riscv64 stance). The compiler tools come from a companion host build (IREE_BUILD_COMPILER=ON, full LLVM — the long compile you see in the app-wheelhouse stage); models are compiled on the host and executed on riscv64.

Build home: linux/scripts/05-frameworks/torch/build-app-wheelhouse.sh (build_iree_wheels), which stages host tools + target runtime and is smoked both natively and via the Python import path. The riscv64 builder iterates on real rebuilds — treat first-failure there as expected tuning, not regression.

verify-parity.sh (on-demand diagnostic, not a gate)#

06-packaging/verify-parity.sh <native-image> <cross-image> diffs two BUILT images across packages,python,versions,files,libs,imports. It needs two images, so it can never be a preflight gate (preflight’s artifact-parity slug is the UNRELATED verify-artifact-copy-parity.sh). Use it when a cross image misbehaves where the native one doesn’t — it localizes the divergence in minutes.

Five Critical Fixes To Maintain#

To prevent regressions during updates, always preserve the following five vital fixes in the Linux cross pipeline:

  1. Fix 1 (gst-python staged libpython): In build_python.sh, the rewrite_staged_python_pc() helper rewrites the staged python-3.14.pc file’s libdir and includedir to point correctly at the compiler’s cross directory so gst-python builds succeed.

  2. Fix 2 (libcamera abseil): In build-litert.sh, the build must copy the required Abseil header absl/types/span.h into the LiteRT installation directory to prevent downstream libcamera build errors.

  3. Fix 3 (cross lib-dynload dangling symlinks): In build_python.sh (build_cross_target_python_payload()), standard CPython build steps create standard cross-build library symlinks that end up dangling when packaged. We use cp -a -L to dereference those symlinks, copy the safety-net Modules, and enforce a hard-fail guard find ... -xtype l to ensure absolutely zero dangling symlinks remain in the target’s lib-dynload subdirectory. This prevents C-extension import failures (e.g. import _struct failing under QEMU/binfmt). Since target-packaged Python is staged into the compiler-cross image, the compiler itself must be rebuilt if this helper logic is changed.

  4. Fix 4 (cross GCC architecture guard): In Dockerfile.package, GCC alternatives wire /opt/gcc-16.2.0/bin/gcc as cc/c++. On amd64, GCC is built natively. On arm64/riscv64, it is Canadian-cross-compiled; Dockerfile.android swaps the amd64-hosted GCC for the target-native binary. The build hard-fails with three layered guards: (a) cc -dumpmachine must match TARGET_ARCH; (b) readelf -h on the cc binary itself checks ELF machine type (the real discriminator — -dumpmachine only reports the target triple, not the host arch); and (c) a cc1 compile-to-object smoke plus ELF check on the produced object, run under the target platform (QEMU for foreign arches). wrapper-smoke (Dockerfile.package target) runs validate-compilers.sh, smoke-media.sh, smoke-torch-venv.sh and smoke-cross-all-arches.sh for end-to-end verification.

  5. Fix 5 (OpenCV 5 GStreamer compat): patch-gstreamer-sources.sh → patch_gstreamer_sources() patches the GStreamer gst-plugins-bad opencv plugin sources at build time for OpenCV 5.x compatibility. Three API changes are handled: (a) contourArea/approxPolyDP/convexHull moved to new geometry module → adds #include <opencv2/geometry.hpp> to gstsegmentation.cpp; (b) chessboard/circles-grid detection (findChessboardCorners/findCirclesGrid/CALIB_CB_*) moved to objdetect module → adds #include <opencv2/objdetect.hpp> to gstcameracalibrate.cpp; (c) cv::CascadeClassifier removed from OpenCV 5 → drops the three cascade-dependent GStreamer elements (faceblur, facedetect, handdetect) from the monolithic libgstopencv.so. Additionally, build-opencv.sh creates an opencv4.pc → opencv5.pc compatibility alias because GStreamer’s meson dependency lookup queries dependency('opencv4'). All patches are idempotent (guarded with grep before applying). When changing OpenCV or GStreamer versions, verify the patch still applies correctly.

Cross env contract#

The cross environment set up by linux/scripts/01-core/cross-env.sh (setup_linux_cross_env) is organized in three tiers. Run linux/scripts/01-core/cross-env-doctor.sh <arch> (or source it and call cross_env_doctor) to validate the contract, print the effective configuration, and compile-smoke-check that $CC really emits target-arch ELF objects.

Tier 1 — core toolchain contract#

Always exported when a cross build is active: TARGET_ARCH, TARGETARCH, TARGETPLATFORM, BUILDARCH, CROSS_TARGET_TRIPLET, and the tool variables CC, CXX, AR, AS, LD, NM, RANLIB, STRIP, OBJCOPY, plus PKG_CONFIG_LIBDIR, PKG_CONFIG_SYSROOT_DIR, PKG_CONFIG_ALLOW_CROSS. CC/CXX must be absolute paths to existing executables. Consumers must use these variables — never bare cc/gcc from PATH — for target-side compiles.

Tier 2 — rust / cmake derivations#

Derived from Tier 1: CROSS_RUST_TARGET, CARGO_BUILD_TARGET, CARGO_TARGET_DIR, CARGO_TARGET_<TRIPLE>_LINKER / _AR, the cc-crate vars CC_<triple> / CXX_<triple> / AR_<triple> / RANLIB_<triple> (for both target and build triples), and the CMAKE_* toolchain variables (CMAKE_SYSTEM_NAME/PROCESSOR, CMAKE_C/CXX_COMPILER, CMAKE_AR, CMAKE_RANLIB, CMAKE_FIND_ROOT_PATH_MODE_*, …).

Tier 3 — PATH policy and bare tool names (opt-in)#

/opt/cross-bin is prepended to PATH but contains only triplet-prefixed tool names (<triplet>-gcc, <triplet>-ld, …), which can never shadow the host toolchain. Bare names (gcc, cc, as, ld, …) live in /opt/cross-bin/bare, which is deliberately not on PATH — bare cross names fronting PATH historically broke every host-side compile (e.g. the riscv64 host-protoc “Exec format error” bug). The few consumers that genuinely need bare names (gcc -B tool lookup, rust cc-crate fallbacks) opt in per scope via cross_bare_bin_path():

bare="$(cross_bare_bin_path)" && exec "${CC}" -B"${bare}/" "$@"