Project Information

Project Information#

Prerequisites#

  • nerdctl with BuildKit support.

  • GPU passthrough configured when building Vulkan-enabled images.

Installation#

  1. Clone the repo:

    git clone --recurse-submodules https://github.com/Kataglyphis/ANTfrastructure.git

Tests#

Automated validation in this repository is the no-build gate suite; no workflow here builds an image:

  • linux-x64.yml runs bash linux/scripts/preflight.sh on every push and PR to main/develop (the mutation gate as four shards) and the docs build. The version snapshot (python3 docs/scripts/sync_versions.py --check) is its version-snapshot slug. windows-x64.yml runs the PowerShell lint and the Pester suites when windows/ changes. Every trigger: ci-build-triggers.md § This repository’s lanes.

  • Local container validation is currently documented as targeted smoke builds in docs/linux-build-basics.md and docs/linux-cross-builds.md.

  • The wrapper-smoke target in Dockerfile.package provides cheap packaging validation before publish.

  • build-cross-chain.sh --verify-chain performs a dry-run stale-check of the entire cross chain against registry digests without building anything.

  • verify-cross-chain.sh provides the same staleness check as a standalone script with a lighter footprint. Both use the shared chain-verify.sh module.

  • cross_stage_validate_graph() (in stage-defs.sh) runs automatically before every build to check internal stage graph consistency (parent references, cycle detection).

  • build-cross-chain.sh --describe-chain prints the full stage graph with tag names and parent chains.

  • verify-artifact-copy-parity.sh checks that the artifact COPY lists in Dockerfile.package are consistent.

  • verify-critical-fixes.sh is the host half of the critical-fixes battery (fix5 through fix11); 06-packaging/smoke-critical-fixes.sh probes fix1 to fix4 inside a built image. The original five are in docs/linux-cross-builds.md § Five Critical Fixes To Maintain.

  • build-cross-chain.sh --dry-run prints all build commands without executing them, useful for auditing the stage transitions.

  • There is not yet a single end-to-end CI workflow that builds every Linux, accelerator, and Windows image variant on each change.

Windows Image Chain#

Windows container builds run on a windows/amd64 host, on windows/servercore:ltsc2025. windows/Build-Buildkit.ps1 is the only driver; the full build commands and prerequisites are in windows-builds.md § Build Commands. A run builds into the local store, and -PushRef publishes the final image: ghcr.io/kataglyphis/kataglyphis_beschleuniger:winamd64 by default, :winarm64 from -TargetArch arm64 (a windows/amd64 image carrying the aarch64 bundle), :winamd64-rocm from -Variant rocm.

Stage chain (each FROM the previous stage’s local tag, docker.io/local/kataglyphis:bk-<name>):

Stage

Dockerfile

Produces

Contents

1

windows/Dockerfile.base

bk-windows-base

VS Build Tools 18 (ClangCL toolset), Scoop (LLVM LLVM_WINDOWS_VERSION, CMake, Ninja, NASM, Flutter, Vulkan SDK), WiX 4 (dotnet tool, WIX_VERSION), rustup + Rust (Install-RustToolchain.ps1), Git, Python bootstrapping, versions.env

2

re-tag of base (CPU) or windows/Dockerfile.nvidia (-Gpu, same as -Variant nvidia) or windows/Dockerfile.rocm (-Variant rocm)

bk-windows-sdk

CPU lane simply re-tags bk-windows-base (the former Dockerfile.sdk no-op shim was removed); the GPU lane builds the NVIDIA layer (CUDA CUDA_VERSION + cuDNN CUDNN_VERSION + optional TensorRT TENSORRT_VERSION); the rocm variant puts TheRock ROCm in the same slot, and every tag after base gets a -rocm infix.

3

windows/Dockerfile.toolchain-builder

bk-windows-toolchain

CPython 3.14 source-built with ClangCL via Build-ToolchainAll.ps1 (PCbuild\build.bat)

4

windows/Dockerfile.media-merge-builder (+ per-branch windows/Dockerfile.media-builder)

bk-windows-media

AI/media stack: ONNX Runtime (ONNXRUNTIME_VERSION), ONNX GenAI (ONNXRUNTIME_GENAI_VERSION), FFmpeg (FFMPEG_VERSION; --enable-libonnxruntime, DNN filters ship with the backend, no separate --enable-dnn), OpenCV 5.x, HailoRT, LiteRT (LITERT_VERSION), LiteRT-LM (LITERT_LM_VERSION), TVM (TVM_REF), IREE, GStreamer (GSTREAMER_VERSION) — all source-built with clang-cl in dependency order; the generator per component is in docs/windows-builds.md § Component Build Matrix

4a

windows/Dockerfile.rocm-migraphx, then windows/Dockerfile.rocm-llama

bk-windows-media-migraphx-rocm, bk-windows-media-llama-rocm

rocm variant only: MIGraphX with the ONNX Runtime plugin EP, then llama.cpp (HIP and Vulkan)

5

windows/Dockerfile.torch

bk-windows-torch

The OrchestrANT app environment at APP_REF

6

windows/Dockerfile

bk-winamd64 (-PushRef publishes it as :winamd64)

Final developer image (VsDevCmd entrypoint, HEALTHCHECK, smoke-test script; the Vulkan loader and WiX’s firewall extension, WIX_FIREWALL_EXT_VERSION, so a bump of either rebuilds only this stage)

Container validation uses windows/scripts/build/Test-Container.ps1 (25 numbered sections; its assertion harness lives in windows/scripts/modules/WindowsSmokeTest.Common.psm1) and the Docker HEALTHCHECK defined in windows/scripts/build/Test-Health.ps1.

The smoke test validates (1) build tools, (2) Python 3.14, (3) Rust, (4) LLVM/Clang+Flutter+WiX, (5) VS Build Tools, (6) Vulkan SDK, (7) CUDA+cuDNN (skippable), (8) ONNX Runtime, (9) ONNX GenAI, (10) OpenCV 5, (11) GStreamer, (12) LiteRT, (13) LiteRT-LM, (14) compiler smoke, (15) CMake+Ninja+clang-cl integration, (16) MSBuild+ClangCL, (17) TVM (source-built), (18) FFmpeg (source-built with DNN/ONNX), (19) environment pointer integrity, (20) Python bindings, (21) the OrchestrANT app environment, (22) IREE, (23) the baked C:\temp\scripts surface, (24) Hailo, (25) the ONNX Runtime single source.

Roadmap#

  • Keep the current multi-platform Linux build path working while expanding the amd64-hosted cross artifact lane.

  • Improve validation coverage for Linux sequential, cross, NVIDIA, AMD, and runtime packaging flows.

  • Continue tightening documentation so the source docs and generated site stay aligned with the Dockerfiles and helper scripts.

Troubleshooting#

Caching is weird or files cannot be found#

Symptom: caching is weird or files cannot be found.

Solution: If sccache is interfering with builds, turn it off explicitly. USE_SCCACHE=0 is the blanket switch in linux/scripts/01-core/compiler-cache.sh: setup_sccache returns early, so no Rust wrapper is wired, and setup_ccache picks plain ccache for CMAKE_*_COMPILER_LAUNCHER instead of the sccache launcher — C/C++ stays cached, on the documented ccache fallback. An exported empty RUSTC_WRAPPER is the Rust-only opt-out:

export USE_SCCACHE=0      # no sccache at all; C/C++ falls back to ccache
export RUSTC_WRAPPER=""   # Rust only

Do not write a bare RUSTC_WRAPPER=. An unset RUSTC_WRAPPER is the opposite signal: build-gstreamer-monorepo.sh tests set-ness ([ -z "${RUSTC_WRAPPER+x}" ]), and on unset it points the wrapper at the guarded launcher /opt/scripts/core/sccache-launcher.sh. A bare assignment at a shell prompt also creates an unexported shell variable that the build process never sees. linux/Dockerfile.toolchain and linux/Dockerfile.package both ship the exported-empty ENV RUSTC_WRAPPER="". On the GStreamer lane that alone does not hold: setup-gstreamer.sh calls setup_sccache, which re-exports the launcher unconditionally and overwrites an empty value inherited from the environment — use USE_SCCACHE=0 there.

No space left on this device#

Symptom: no space left on this device.

Solution:

  • Prefer workspace-relative output directories like logs/ and out/ for large build artifacts.

  • The runtime packaging helpers already avoid /tmp by default and use ${XDG_CACHE_HOME:-$HOME/.cache}/opencode/runtime-build-contexts for temporary local stage handoff.

  • The root .dockerignore excludes linux/webserver and out/, so repo-root Linux builds do not keep re-sending them through unrelated build contexts.

  • Keep exported repair trees such as out/runtime-repair-* out of later Docker build contexts too, or routine retries will spend minutes re-uploading them.

  • Clean old local images, caches, and exported rootfs artifacts if repeated BuildKit runs fill the disk.

Local runtime images try to pull from a registry#

Symptom: a local runtime rebuild tries to resolve docker.io/library/opencode-local:* remotely, or localhost/* is treated like a real registry and fails with connect: connection refused.

Solution:

  • On this host, do not rely on plain local image tags as reusable FROM sources for the runtime packaging chain.

  • Keep the helper default local-context handoff for base -> package -> torch, and for saved runtime artifact images pass ARTIFACT_CONTEXT_ROOT=... with ARTIFACT_CONTEXT_MODE=oci instead of expecting FROM opencode-local:* to stay local. The helper still runs the Torch stage natively on linux/<arch> so the final runtime image includes /opt/venv. In cross mode, the media artifact lane now also makes a best-effort riscv64 app wheelhouse on the amd64 host for the locked torch, torchvision, and opencv-python git-source dependencies used by OrchestrANT, and the native Torch install keeps the upstream uv.lock when present so it can reuse those local wheels before falling back to source builds. If a reused cross artifact has an empty /opt/wheels the Torch install step now keeps the packages that uv sync already resolved instead of trying to install a literal /opt/wheels/*.whl glob. The foreign-arch package stage must keep /usr/bin/clang wired to the copied target-native /usr/local/llvm-target/bin/clang while prioritizing the custom /opt/gcc-16.2.0 as the default system native compiler, rather than falling back to the distro clang at /usr/lib/llvm-<major> (major taken from LLVM_RELEASE).

  • docs/linux-cross-builds.md documents the verified mixed OCI artifact + plain rootfs base workaround.

Rebuilt SDK artifact still reports old clang#

Symptom: a rebuilt arm64 or riscv64 SDK artifact still reports an older clang version under /opt/llvm-target even though the repository pin was updated.

Solution:

  • The target-native clang is built in the compiler image (Dockerfile.toolchain installs /opt/llvm-target-<arch> for every arch); linux/Dockerfile.sdk only selects it into /opt/llvm-target (materialize-llvm-target.sh). The SDK re-declares the checked-in LLVM_RELEASE, so it does not inherit a stale value from an older cross-compiler-amd64 base image, but it cannot rebuild the clang.

  • After a pin bump, rebuild the compiler image first, then the SDK artifact.

buildctl or ctr permission denied in rootless troubleshooting#

Symptom: buildctl du --verbose fails with dial unix /run/buildkit/buildkitd.sock: connect: permission denied, or ctr images export cannot access /run/containerd/containerd.sock.

Solution:

  • Some rootless setups expose nerdctl but not the raw BuildKit or containerd sockets.

  • Use nerdctl save, nerdctl create, and nerdctl export for local image export and inspection on this host.

  • Prefer the checked-in runtime helpers over manual rebuild loops when validating or publishing the cross runtime path.

  • Fall back to regular disk usage checks and nerdctl cleanup commands when buildctl or ctr socket access is unavailable.

Slow build-time downloads in rootless nerdctl/BuildKit#

Symptom: A RUN step that downloads a large source tree is extremely slow. The worst offender is the LLVM git fetch in linux/scripts/02-toolchain/build-clang.sh (and linux/scripts/02-toolchain/llvm.sh) during the cross-compiler/SDK builds.

Cause: Rootless BuildKit defaults to --oci-worker-net=bridge, so every in-build git/curl/wget is routed through the user-space rootless bridge/slirp path. Registry mirrors do not help here because this is not an image pull.

Solution (already applied on this host):

  • Switch the rootless BuildKit OCI worker to host networking with the systemd drop-in ~/.config/systemd/user/buildkit.service.d/override.conf. Its canonical copy is linux/host-config/buildkit.service-override.conf (it also raises the 2 MiB step-log clip); install it with bash linux/host-config/apply-host-config.sh, which refuses while a build runs and writes the nerdctl prefix the live units already name in place of @NERDCTL_PREFIX@. Never hand-write a /usr/local prefix there: it reverted a rootless $HOME/.local install once (2026-09-08). The line that matters:

    [Service]
ExecStart=
ExecStart="@NERDCTL_PREFIX@/bin/containerd-rootless-setuptool.sh" nsenter -- buildkitd --oci-worker=true --oci-worker-rootless=true --containerd-worker=false --oci-worker-net=host --allow-insecure-entitlement network.host

  • Make rootless containerd networking explicit and fast with ~/.config/systemd/user/containerd.service.d/override.conf:

    [Service]
Environment=CONTAINERD_ROOTLESS_ROOTLESSKIT_NET=slirp4netns
Environment=CONTAINERD_ROOTLESS_ROOTLESSKIT_MTU=65520
Environment=CONTAINERD_ROOTLESS_ROOTLESSKIT_DETACH_NETNS=true
Environment=CONTAINERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER=builtin

  • Mirror Docker Hub image pulls (only helps FROM ..., not in-build downloads) via ~/.config/containerd/certs.d/docker.io/hosts.toml (referenced by hosts_dir in ~/.config/nerdctl/nerdctl.toml) and ~/.config/buildkit/buildkitd.toml.

  • Apply changes with:

    systemctl --user daemon-reload
systemctl --user restart containerd buildkit

  • With --oci-worker-net=host set, plain nerdctl build already uses host networking; you do not need to pass --network host.

  • For repeated LLVM rebuilds, the host-net change is the main lever. For an even bigger win, cache the LLVM source on the host instead of re-fetching it every build.

Local runtime artifacts: OCI layouts, rootfs, and disk usage#

When feeding locally saved runtime artifacts back into later builds:

  • Keep .dockerignore excluding out/local-oci, out/local-android-dir, out/linux-sdk, out/linux-runtime, and out/runtime-repair-* so exported OCI layouts and rootfs trees do not get sent back as a later build context.

  • Prefer saved OCI layouts such as out/local-oci/android/<arch> for foreign-architecture runtime packaging. The plain directory exports under out/local-android-dir/<arch> are much larger, and an earlier OCI-to-directory conversion dropped /usr/local/lib/onnxruntime-cpu.

  • On this host, the verified local runtime path mixes context types: the heavy runtime_artifact input comes from an oci-layout://... context, while the intermediate runtime_base handoff stays a plain rootfs directory because one build still fails when it consumes two named OCI image contexts at once.

  • readlink -f on symlinks inside out/linux-runtime/*/rootfs resolves absolute links against the host root, so use plain readlink or validate from inside the built image when checking /usr/bin/cc, /usr/bin/clang, /etc/alternatives/cc, and /etc/alternatives/clang.

  • Local BuildKit and containerd stores can still grow very large during repeated runtime rebuilds, so prune old images and exported artifacts if disk pressure returns.

Terminal Freeze or Slowness During Large/Interactive Rebuilds#

Symptom: Running long interactive nerdctl build loops in the foreground causes the terminal to freeze, lag, or experience extremely slow download rates with direct terminal stdout.

Cause: High-volume stdout stream pipelines from concurrent apt or source fetch downloads (under QEMU/binfmt or native compilation) can overwhelm terminal buffers and choke build execution.

Solution: Always build each stage independently and non-interactively in the background using a decoupled session (e.g., setsid bash -c "nerdctl build ... > stage-build.log 2>&1" & disown) and poll/inspect progress via file-based tail, grep, or pgrep checks. This guarantees that direct console rendering does not throttle execution threads.

Contributing#

  1. Fork the project.

  2. Create a feature branch (git checkout -b feature/my-change).

  3. Make your changes. If modifying build scripts or Dockerfiles, run python3 docs/scripts/sync_versions.py --check to verify version consistency.

  4. Commit your changes — the pre-commit hook (linux/host-config/git-hooks/pre-commit, installed by make hooks) runs the fast preflight slugs, staged shellcheck and the doc gates.

  5. Push and open a pull request.

License#

This project’s own code is licensed under the MIT License — see the LICENSE file at the repository root. Every source file carries a matching SPDX-License-Identifier: MIT header, and the published container images declare org.opencontainers.image.licenses="MIT".

Until 2026-08-07 the source headers and two of the three image labels said Apache-2.0 while LICENSE said MIT. That contradiction is resolved in favour of MIT throughout; nothing about the terms was ever intended to differ from LICENSE.

Bundled upstream software keeps its own terms — see Third-Party Licenses. The NOTICES bundle the served site carries is generated by the app that builds that site, in another repository; the licences in it belong to those projects and nothing here edits them. This repository generates only linux/webserver/license-assets/, which the image overlays onto the site at build time.

Repository size and large files#

A factual note, because the alternative is that somebody rediscovers this with git count-objects and reaches for a history rewrite.

The pack is ~154 MB, and almost all of it is one directory. linux/webserver/dist/ — the built Flutter site the :webserver image serves — was tracked from the beginning until 2026-09-15: 82 MB in the working tree, and several times that in history, because minified web output is rewritten in full on every site build and every version of it is a new set of blobs. It is built in another repository, from sources that do not exist here, so nothing in this repo could ever regenerate it.

What is tracked and large today: nothing. After that removal the biggest tracked files are text — docs/scripts/mutations.json (~600 KB on 2026-09-25), docs/scripts/code-dupes.allow (~390 KB), CHANGELOG.md and the documentation archives. The only tracked binary is images/logo.png, at 2 KB. The .gitignore rules under “Binaries and build output do not belong in git” are what keeps it that way; a binary that genuinely belongs here is committed with git add -f and named in this section.

The history is NOT being rewritten, and that is a decision, not an oversight. A filter-repo pass would reclaim most of the 154 MB and would also change every commit id in the repository — which breaks every pin in every consumer (third_party/ANTfrastructure is a gitlink in every consumer but ANThology), every SHA in every changelog entry and every link into this repo that names a commit. 154 MB is a one-time clone cost measured in seconds on any connection this project is developed on. The cost of the rewrite is paid by everyone, repeatedly, for as long as anyone reads an old reference.

Contact#

Jonas Heinle - @Cataglyphis_ - jonasheinle@googlemail.com

Project Link: Kataglyphis/ANTfrastructure

Acknowledgements#

Thanks for free 3D models:

Literature#

Some very helpful literature, tutorials, etc.

Reusable Sphinx theme package#

The shared theme lives in the DocumANTation repo and is vendored here as a submodule.

The shared theme now lives in the DocumANTation repo, vendored here as a submodule at third_party/DocumANTation. requirements.txt installs it editable (-e ./third_party/DocumANTation/sphinx-kataglyphis-theme), so docs/conf.py just imports it:

from sphinx_kataglyphis import setup_theme
setup_theme(globals(), repository_url="https://github.com/org/repo")

setup_theme() provides all shared Sphinx config and loads the canonical CSS from the package’s _static/ directory.

For other projects — add the DocumANTation submodule and the same -e line to their requirements.txt, then use the conf.py snippet above.

The canonical CSS lives in the submodule at third_party/DocumANTation/sphinx-kataglyphis-theme/sphinx_kataglyphis/_static/css/custom.css — edit it in the DocumANTation repo to change the global look. The project’s own docs/_static/css/ can hold additional per-project overrides.