Windows build resources — CPU, memory, cache and GPU

Windows build resources — CPU, memory, cache and GPU#

How much of the host a Windows container build may take, the measured ceilings, and how the compile cache is wired. Every number here was measured on this repo’s build host — treat them as this host’s envelope and re-measure elsewhere.

Neighbours: the lanes themselves are windows-build-lanes.md; consumer-project build performance (container reuse, transports, Dev Drive) is windows-container-build-performance.md; Linux-side job counts are build-parallelism-memory-tuning.md.

GPU acceleration in containers (DirectML on the host GPU)#

On Windows, GPU acceleration in containers is DirectX-only — Direct3D 12 and everything layered on it, which includes DirectML (the ONNX DmlExecutionProvider and onnxruntime-genai’s DML path). CUDA/TensorRT cannot be GPU-accelerated in a Windows container. On this host that is exactly the point: the machine has an AMD Radeon RX 9070 XT (+ iGPU) and no NVIDIA GPU, so DirectML — being vendor-agnostic — is the only GPU path. (The CUDA/TensorRT EPs are still built and smoke-checked for availability, but they have no device to run on here.)

Running DirectML on the physical GPU inside a container requires all of:

  1. Process isolation — Hyper-V-isolated containers get no GPU. (The default isolation on this host is hyperv, so you must pass --isolation process.)

  2. The DirectX GPU device, attached with the exact device interface class GUID: --device class/5B45201D-F2F2-4F3B-85BB-30FF1F953599. A wrong variant is silently accepted by docker run but matches no device, so the container falls back to the WARP software renderer with no error.

  3. A base-image OS build that matches the host build. Basic process isolation tolerates skew (a 26100 image runs on a 26200 host), but GPU driver-store injection does not — hcs::CreateComputeSystem fails with “The system cannot find the path specified” when the builds differ. This is the same client-host (26200 / 25H2) vs Server base image (servercore:ltsc2025 = 26100) skew that breaks docker build --isolation process layer commits.

Current status on this host: BLOCKED by the build skew. The GPUs are GPU-PV partitionable (Get-VMHostPartitionableGpu lists both AMD adapters), process isolation works, and the DirectML runtime is built correctly — but GPU device assignment fails at CreateComputeSystem because the ltsc2025 (26100) base does not match the host (26200), and no public client 26200 base image exists to rebuild against. A DXGI enumeration inside the (correctly-flagged) container therefore sees only Microsoft Basic Render Driver (WARP), zero hardware adapters.

To retire the block: rebuild the base on a servercore/nanoserver tag whose build equals the host, or run the image on a host whose build equals the image (26100, e.g. a Windows Server 2025 host). Until then, DirectML on the AMD GPU still works fine outside containers — run the source-built ORT / GenAI binaries directly on the bare host and the DmlExecutionProvider selects the RX 9070 XT.

Re-check after any Docker / containerd / hcsshim / Windows / base-image / GPU-driver upgrade with the self-contained probe under windows/scripts/diagnostics/:

.\windows\scripts\diagnostics\Test-GpuPassthrough.ps1

It prints host/image builds and partitionable GPUs, runs a process-isolation control, attaches the GPU device, compiles + runs a DXGI adapter enumerator inside the container, and gives a verdict: PASSTHROUGH WORKS (a HARDWARE adapter is visible), BLOCKED (build-skew CreateComputeSystem failure), or DEVICE-NOT-INJECTED (started but only WARP). Note the DML probes in Test-Container.ps1 validate that the provider is built and registered (GetAvailableProviders → dml=1 natively, get_available_providers() / onnxruntime_genai.is_dml_available() in the base interpreter and the app venv, plus the x64 D3D12Core.dll PE-machine check); they do not create a device, so they pass under either isolation regardless of whether a hardware adapter is present.

Media fan-out and memory budgeting#

Media scheduling is sequential (one log per solve — out\windows-build-logs\bk-<runid>-<stage>.log, one per media-core library, one per aux branch, one for the merge). Sequential gives media-core the whole host RAM budget — and since its parallelism is memory-bound, more RAM = more ONNX jobs, which matters more than overlapping the small aux branches (a former -ConcurrentMedia overlap mode was removed for exactly that reason).

-MediaMemoryGb auto-detects from host RAM (default 0 = auto). It resolves to usable_physical_GB − HostReserveGb. -HostReserveGb (default 22 — see the learned-the-hard-way note below) is the RAM left for Windows + dockerd + Defender; lower it to push closer to the metal (riskier — under memory pressure the hcsshim ttrpc wedge is more likely). Pass an explicit -MediaMemoryGb N to override auto-detection. The budget is published to the sccache WebDAV endpoint (preseed/memory-limit-gb.txt; as an ARG or ENV it would be a cache key, #51), and Get-BuildJobCount scales each build’s job count to it (a MEMORY_LIMIT_GB in the environment wins, host RAM is the fallback, and BUILD_JOBS overrides the heuristic outright).

Worked example (this 64 GB host, Windows reports 61.4 GB usable → floor 61, default -HostReserveGb 22): auto -MediaMemoryGb = 61 − 22 = 39 g → ONNX runs ~j10 (mem/4, cores=32).

media-core, toolchain and the merge/GStreamer stage all solve process-isolated with every host CPU (see windows-build-lanes.md § Build isolation and CPU parallelism); the run+commit path and its -MediaCoreCpus flag went with build.ps1 on 2026-08-31. The litert/tvm aux branches get the whole budget once media-core is done — halved per child under -ConcurrentAux, which overlaps only those two.

Building a project inside the image, as AGENTS.md carried it#

Moved out of AGENTS.md on 2026-09-15 (owner decision D10), unedited except for this heading and the relative links. The RULES stayed there; this is the reference behind them.

Consumers building large projects in this image should read docs/windows-container-build-performance.md before hand-rolling anything. The number that motivates it, measured on a ~690-object C++23 modules project: 9.6 s ninja / 44 s wall for a no-change incremental build against a reused container, versus 352-484 s when every build got a fresh one.

The rule that follows: reuse ONE container, recreating it only when the image ID changes; stream sources in and executables/logs out, never the intermediate build tree. Transport choice, the Dev Drive filter setup, and the four traps that cost measurable time (bind mount slower behind a filesystem filter, sccache unsafe on clang-cl C++20/23 modules until mozilla/sccache#2876 ships, a named volume unusable as a CMake build dir, deep paths aborting tar transfers) are all in that page.

Maximum resource envelope (verified 2026-07-12)#

The defaults ARE the maximum for this 64 GB / 32-thread host — there is no faster configuration to unlock, and the full-chain rebuild of 2026-07-12 (base → sdk → toolchain → media → final, phase-tagged resource CSV) is the proof:

Phase

Minutes

AvgCpuPct

MaxCpuPct

MinFreeGB

media-core

111

37

100

0.2

media-litert

18

38

100

24.9

media-tvm

~25

42

100

41.8

  • CPUs: 32/32 on every heavy stage. That chain took them from the classic lane’s docker run --cpu-count 32 + commit, the only >2-CPU path there (docker build was pinned at 2 CPUs by the host defect — hence its cheap COPY/clone-only layers); the BK lane gets all CPUs from process isolation.

  • RAM: 39 GB is the measured optimum, not a conservative default. During media-core the host bottomed out at 0.2 GB free — the 22 GB reserve was consumed almost exactly. Raising -MediaMemoryGb (or cutting -HostReserveGb) does not add jobs fast enough to beat the starvation cliff: the 53 GB experiment deadlocked media-core at 0 % CPU (see the hard-way note below).

  • Average CPU of ~35–45 % during compiles is CORRECT and expected — it is the memory-bound signature (jobs = min(32, 39 GB / ~4 GB-per-ONNX-job) ≈ 10), not a tuning failure. Do not chase 100 % average CPU on this host.

  • The only real “go faster” levers are infrastructural: ~128 GB RAM (true j32 on ONNX), or a populated sccache remote (-SccacheEndpoint / SCCACHE_WEBDAV_ENDPOINT) to make rebuilds warm — cold full-chain is ~5–6 h with ~2.5 h of that in the media fan-out.

Per-run resource log. Every Build-Buildkit.ps1 run samples host CPU / free RAM / commit charge / container-VM (vmmem) size every 20 s into out\windows-build-logs\resources-<timestamp>.csv, tagged with the current build phase — the BK stage label (Dockerfile.media-builder:media-core-built-onnx), plus init and done — and prints a per-phase exhaustion summary at the end, including on failure. The sampler starts only after every preflight gate has passed, so a rejected launch leaves nothing orphaned, and -ConcurrentAux children run without one (the parent’s already covers the machine). Re-analyze any run later with pwsh -File windows/scripts/build/Build-ResourceSampler.ps1 -Summarize -CsvPath <csv>; MinFreeGB per phase shows which step pushed the host hardest, and an AvgCpuPct far below 100 during a compile phase means the step was memory-bound (jobs = min(cores, MEMORY_LIMIT_GB/perJob)), not CPU-bound. Disable with -NoResourceLog. Since 2026-10-12 each row also carries diskFreeGB for the build drive, and the summary its per-phase minimum (MinDiskFreeGB): export and unpack slow down as it nears a stage’s disk floor.

Per-run record. Beside the stage logs, every run writes bk-<run>-driver.log: the run, each stage’s start, end and failure, every retry verdict with the line that made it transient, and every disk-gate verdict, none of which a stage log holds. bk-<run>-manifest.txt carries status=ok, or status=failed:<stage> with every stage that finished, so a stopped run keeps its stage seconds too (hub #184).

Why the reserve is 22 GB, not ~8 (learned the hard way). An earlier default of -HostReserveGb 8 auto-sized media-core to 53 GB, which hung the build: during a GPU build dockerd + containerd juggling the ~50 GB CUDA image layers, plus svchost/Defender, hold ~16–18 GB steady — so 53 GB container + ~17 GB host exceeded the 61 GB physical, the Hyper-V VM starved at ~43 GB, and media-core deadlocked at 0 % CPU with the host at 0.3 GB free (log frozen mid-ONNX for 2 h). The --memory cap is real RAM committed to the utility VM, so container_cap + host_footprint must fit physical RAM with margin. 22 GB reserve (→ ~39 GB container, ~56 GB peak) is the verified-safe budget here. The heavy CUDA TUs (FlashAttention, MoE kernels) use more than the ~4 GB/job estimate, so do not shrink the reserve without watching docker stats + host free RAM.

The Windows cache, tier by tier#

Moved out of AGENTS.md on 2026-09-15 with only this heading and the list marker changed; corrected since where the code moved (2026-09-18, 2026-09-25). The rule – assume nothing from the Linux chain, preserve the layer ordering and the per-file module closures, check the reserve before blaming a cache key – stays there; the tiers, the numbers and the two incidents are here.

The WINDOWS chain caches differently — do not assume rules 1-4 apply. It relies on (a) deliberate layer ORDERING — Install-Vs.ps1 sits ABOVE the versions.env COPY in Dockerfile.base so a pin bump cannot re-pay VS Build Tools (confirmed live 2026-08-08: 4 of 16 base steps CACHED through a PYTHON_VERSION bump, and they were the expensive ones), (b) TIERED, PER-FILE in-container module closures so a host-only module edit re-keys only the RUNs that import it, (c) sccache, and (d) buildkitd’s GC reserve. Preserve (a) and (b) in any Dockerfile edit — moving a COPY above the VS layer, or widening a module stage, costs hours per bump.

(b) has only actually been true since 2026-08-31. Dockerfile.toolchain-builder’s patched-llvm RUN bind-mounted the WHOLE windows/scripts/modules directory, and patched-llvm is the DEFAULT toolchain target (-StockLlvm is the opt-out), so any .psm1 edit re-keyed the LLVM 23.1.0 compile and every media lane derived from that image. It is a six-file mount now, and BuildKit.ModuleClosure.Tests.ps1 fails a whole-dir modules mount in any windows Dockerfile except Dockerfile.probe (exempt by design — PROBE_NONCE busts its layer anyway).

(d) is the one that fails SILENTLY and looks like a Dockerfile problem. reservedSpace in windows/buildkitd.toml is the only floor GC will not prune below, and it must exceed the fresh chain spine (~120–150 GB: base incl. VS + sdk + toolchain + branch images). Set below that, the ~37 GB VS-class layer is evicted between driver runs and every run re-solves the prefix — #9 RUN Install-Vs.ps1 re-executing for 4–7 min while #8, the COPY of that very script, reports CACHED. It has happened twice (2026-08-11, 2026-08-26). Before blaming a cache key, check the reserve against buildctl du’s Total and against the store size: Reclaimable: 0B is not “nothing to clean”, it is what a store already pruned to its floor looks like. maxUsedSpace below the working-set size has the same effect, because it forces eviction regardless of what the reserve says.

The module tiers (#134, 2026-08-26; toolchain narrowed 2026-08-31) — check which one a module is in before editing it, because the cost differs by hours:

  1. Dockerfile.media-builder’s buildmods six (SourceBuild.Common + Shared, Patches, Cuda, Native.Common, TargetArch.Common). They ARE the import closure — SourceBuild.Common imports the other five and every mounted build script imports it — so the set cannot be shrunk and every media/merge RUN keys on all six. A one-line edit costs a full media rebuild on both lanes.

  2. tvmmods (FROM buildmods AS tvmmods + WindowsTvm.Common.psm1), mounted by media-tvm-built alone. That branch runs parallel to media-core, so an edit costs nothing on the long pole. Write-AssembledWheelDistInfo and Get-PyprojectDependencies moved off the tier-1 facade into this leaf on 2026-08-31 — Build-TvmFromSource.ps1 is their only caller.

  3. The merge leaves in Dockerfile.media-merge-builder’s buildmods: WindowsGstPlugins.Common, WindowsMeson.Common, WindowsRustToolchain.Common, WindowsInstaller.Common. An edit costs the GStreamer layer.

  4. Dockerfile.toolchain-builder’s patched-llvm RUN mounts the same six as tier 1, per-file. It is the DEFAULT toolchain target, so an edit re-pays the patched-LLVM compile AND every media lane below it — the most expensive tier in the chain.

  5. WindowsOrtProvenance.Build.psm1 (the ORT gate G2, 2026-09-23) is a per-file mount at C:\bkmnt\ortmods\ in the five ORT-consumer RUNs: FFmpeg, OpenCV and the GenAI tail in Dockerfile.media-builder, GStreamer in the merge, and the AMD GPU EP in Dockerfile.rocm-migraphx. An edit re-runs media-core from FFmpeg on (each stage builds FROM the one before), the merge and the AMD GPU EP, never ONNX. SourceBuild.OrtChainOnly.Tests.ps1 holds the mount shape.

Do NOT move a helper into WindowsSourceBuild.Common because “that is where helpers go” — if one branch is its only consumer, it belongs in a leaf. BuildKit.ModuleClosure.Tests.ps1 enforces both directions (a mounted script’s transitive closure must be mounted; leaves must stay out of buildmods, and tvmmods must keep exactly one consumer). It is mutation-proven — trust it over reading the Dockerfile.

Wired, with the rules an agent must not break. Full rationale, measurements and decision history: docs/windows-build-resources.md § Persistent compile cache (sccache).

  • sccache runs WebDAV-remote-only since 2026-08-16. SCCACHE_MULTILEVEL_CHAIN is an ARG with no default in every compiling stage, never an ENV (§ What the published image carries); unset means WebDAV only. Restore disk,webdav per run with -BuildArg SCCACHE_MULTILEVEL_CHAIN=disk,webdav, and only after re-verifying against a newer buildkit. SCCACHE_DIR alone does nothing without the chain variable.

  • sccache is the released 0.18.0 zip, installed into CARGO_BIN (since 2026-09-18; the SCCACHE_GIT_REV source build is retired). 0.18.0 carries #2722 + #2811 + #2816 — everything the pin carried. History and the CUDA canary bar: § Persistent compile cache (sccache).

  • CMAKE_CUDA_COMPILER_LAUNCHER is ON BY DEFAULT since 2026-08-18. Never flip that default off silently, and never export the launcher onto a new sccache without all THREE canaries — the miscompile it once caused is invisible until the DLL link.

  • uv/pip wheel cache in Dockerfile.torch, set INSIDE the RUN (an ENV would bake a build-only mount path into the shipped image).

Still NOT wired, with a measured reason: source-fetch mounts. The clones are shallow (Invoke-GitClone passes --depth), so they cost minutes against compiles that cost hours. If you do it, cache the ARCHIVES/CLONES only, never the working tree — directory RENAMES fail on cache mounts and Build-GstreamerFromSource.ps1 moves the extracted tree. Also raise the tier-0 type==exec.cachemount cap in windows/buildkitd.toml — it is shared by every cache mount plus local sources and git checkouts, and the sccache L0 (15G) and uv cache (10G) already claim 25 GB of its 30 GB reserve (the cap is 60 GB). Cache sizes and that cap are ONE decision, not two. (Since 2026-08-16 the L0 mount is attached but DORMANT — the chain defaults to WebDAV-only — so its 15G is reserved rather than consumed. Do not repurpose that headroom: the tier is meant to return, see #99.)

Persistent compile cache (sccache)#

Without BuildKit cache mounts a container-local sccache cache dies with the layer, so the WebDAV remote is the only compile cache that survives a container. sccache is therefore REQUIRED by default for the media stages: Build-Buildkit.ps1 fails fast when a media stage is requested and no reachable endpoint is configured (-NoSccache opts into a deliberate cache-less build). The gate is media-only (Assert-SccacheEndpoint, $compileStages = @('media') in WindowsBuildDriver.Common.psm1): the toolchain’s CPython build (MSBuild/ClangCL) has no sccache wiring, and its default patched-llvm target uses the endpoint when one is passed (#164) but compiles LLVM cold without it, so toolchain-only builds are never blocked on an endpoint. One-time host setup:

# one-time host setup (any WebDAV-capable server works; dufs is a single binary)
scoop install dufs
mkdir C:\sccache-cache
dufs C:\sccache-cache -A -p 5000

# then build with the endpoint (use an IP reachable from inside containers,
# e.g. the host's LAN IP — not localhost)
.\windows\Build-Buildkit.ps1 -Gpu -SccacheEndpoint http://192.168.1.10:5000

CMake-based builds (every configure through Invoke-CmakeConfigure: ONNX, GenAI, OpenCV, LiteRT, LiteRT-LM, TVM, IREE, HailoRT, and on the rocm lane MIGraphX and the AMD GPU EP) then route clang-cl through sccache. GStreamer (Meson) is cached since 2026-10-12 (#182): meson drops an sccache prefix in front of clang-cl, so the 'sccache clang-cl' of 2026-08-04 never reached a compile (16 requests for 5,143 ninja steps). Build-GstreamerFromSource.ps1 now builds a clang-cl.exe shim that runs sccache on the patched clang-cl, hands it to meson as CC/CXX, and fails the stage when sccache counted fewer than 100 compile requests. FFmpeg (clang-cl + make) is cached since 2026-08-20 (#100): the launcher goes in at make time (make CC='sccache clang-cl'), never into configure’s --cc, and FFMPEG_SCCACHE=0 opts out. gst-plugins-rs (cargo, phase 8b of the GStreamer build) goes through the same server since 2026-10-15: RUSTC_WRAPPER plus CARGO_INCREMENTAL=0, whenever meson’s compiles do. The crates’ rlibs hit; the two plugin cdylibs cannot. The step took about 2 min uncached in every variant on 2026-10-14. The first build populates the cache; subsequent --no-cache rebuilds and version bumps reuse unchanged object files.

sccache is the OFFICIAL RELEASED 0.18.0 zip since 2026-09-18 — the source build at SCCACHE_GIT_REV is retired. Install-RustToolchain.ps1 downloads sccache-v0.18.0-x86_64-pc-windows-msvc.zip, verifies it against SCCACHE_WINDOWS_ZIP_SHA256, and installs sccache.exe into CARGO_BIN, which precedes the scoop shims on PATH. Test-Toolchain.ps1 asserts the version, which works now because 0.18.0 reports 0.18.0 — unlike main-at-git-rev, which still said 0.17.0 and forced a path-based assert.

Why the source build existed, and what would bring a bare-nvcc exception back:

  • CUDA 13.3’s --simt-only dryrun mis-parse. Released sccache decomposes nvcc by parsing nvcc --dryrun; 13.3.33 emits --simt-only AFTER the input file, so the positional parser took the flag as the input, mis-grouped the cicc/ptxas device steps, and the per-arch .cubin files were never produced. The build died at the combine step with fatbinary fatal : Could not open input file '<tu>.compute_80.cubin' (measured here 2026-08-08 on ONNX’s CUDA provider). Fixed by mozilla/sccache#2722, merged 2026-08-04 — five days AFTER v0.17.0 shipped.

  • #2811 (dryrun quote collapse). \" escapes flattened before tokenization packed ~30 -D pairs into one 493-char token, so the cpp4 preprocess lost USE_CUDA & friends; that was the 2026-08-10 miscompile (dropped instantiations, lld-link: undefined symbol). Merged 2026-08-19.

  • #2816 (--diag-suppress separated form). The OpenCV #115 blocker; merged 2026-08-26.

  • All three ship in 0.18.0 (released 2026-09-14), which is why the git-rev pin, the windows/upstream/sccache-nvcc-quote-fix/ series and the cargo install --git --rev path are gone.

  • The cuda_llm scope (patch 006). While the fix was in flight, ONNX’s onnxruntime_providers_cuda_llm target was scoped to BARE nvcc, because sccache’s server died deterministically on the cutlass-generated fused_moe GEMM launchers (~4910 s in, clients got os error 10054; #2808). Patch 006 was retired 2026-08-18 with the #114 series: fused_moe compiles through the launcher and links green. If undefined fused_moe/QkvToContext symbols return, check the series still applies before resurrecting a bare-nvcc exception.

  • The canary bar. Never point CMAKE_CUDA_COMPILER_LAUNCHER at a new sccache on the strength of a green compile: the miscompile class is invisible until the DLL link. A candidate must pass windows/scripts/diagnostics/Test-CudaCache.ps1 (its in-container payload verify-cuda-cache/Test-Cache.ps1 compiles the same .cu twice and asserts a cache hit AND a backend write) plus an ONNX canary through the fused_moe launchers before cuda_llm is re-wrapped.

The CUDA launcher (CMAKE_CUDA_COMPILER_LAUNCHER) is ON BY DEFAULT since 2026-08-18 (SCCACHE_CUDA_LAUNCHER="1" in the media-core-built-onnx stage), after a decision history worth keeping:

  • The 2026-08-10 miscompile (dropped instantiations, lld-link: undefined symbol) was root-caused to sccache’s Windows dryrun quote-collapse — \" escapes flattened before tokenization packed ~30 -D pairs into one 493-char token, so the cpp4 preprocess lost USE_CUDA & friends. Fixed upstream (mozilla/sccache#2811, MERGED 2026-08-19). The --diag-suppress separated form (mozilla/sccache#2816) also merged upstream 2026-08-26; both ship in the released 0.18.0 the lane now installs.

  • The three-canary bar passed on the evening of 2026-08-18: fused_moe compile green, providers_cuda link green COLD (153 CUDA device writes), link green on the HIT run at 100.00% CUDA/PTX/CUBIN hit rate (207/816 hits) — onnx’s CUDA portion drops from ~60 to ~33 min warm. The canaries are Test-CudaCache.ps1 + a fused_moe compile + a full providers_cuda LINK — the miscompile class is invisible until link, which is why all three are required before trusting any new sccache with the launcher.

  • The #2808 DEADLOCK separately proved to be #99 collateral (gone under a healthy backend). Patch 006 (bare fused_moe) was RETIRED 2026-08-18 (moe compiles through the launcher, link green).

  • Opt out per run with -BuildArg SCCACHE_CUDA_LAUNCHER=; never flip the default off silently.

  • Except ORT’s PTX-only targets when Blackwell is in the arch list (2026-09-26, narrowed 2026-10-15). sccache 0.18.0 aborts an nvcc compile whose output for one arch is PTX only (Missing "cubin" file output, mozilla/sccache#2862, open). The first chain with 120 died at [2323/2383] on fpA_intB_gemm.

    • Which targets. ORT turns 86;87;89;120 into 86-real;87-real;89-real;120a-real (cmake/external/cuda_configuration.cmake). Only onnxruntime_providers_cuda_llm (and the CUDA plugin’s _llm twin) gets 120-virtual instead, on MSVC alone (REPLACE_SM120_REAL_WITH_VIRTUAL: native sm_120a pulls tcgen05 headers MSVC cannot host). Every other target compiles cubins only, which sccache caches.

    • Until 2026-10-15 Disable-OrtCudaLauncherForPtx kept nvcc bare for all of ORT, and the nvidia ORT stage took 83 min (4956 s in the chain of 2026-10-14, the longest step of any chain).

    • Now Get-OrtPtxLauncherExemptArg keeps the launcher and passes -DCMAKE_PROJECT_onnxruntime_INCLUDE=…/patches/onnxruntime/sccache-ptx-exempt.cmake. That file, deferred to the end of ORT’s configure, clears CUDA_COMPILER_LAUNCHER on every target with a -virtual arch and prints them (sccache-ptx-exempt: nvcc stays bare for onnxruntime_providers_cuda_llm; …).

    • Tests. SourceBuild.OrtCudaLauncher.Tests.ps1 runs CMake on a stand-in project: a -virtual target loses the launcher, in a subdirectory too, and a real-only one keeps it.

    • First chain, 2026-10-16 (nvidia, hub 8a4ef0fa, the first with the launcher on ORT 1.31’s CUDA, so cold): configure printed nvcc stays bare for onnxruntime_providers_cuda_llm, nvcc went through sccache (292 CUDA hits, 277 misses, where it had made none), and the stage took 4746 s against 4956 s with nvcc bare. The warm rerun and the three canaries above are still to come.

    • Drop both when a released sccache fixes #2862. OpenCV passes -real archs only and is unaffected.

The disk level is off: Windows BuildKit cache mounts lose writes into a directory an earlier RUN populated, so no chain names disk, and the host’s WebDAV is level 0 (§ Two cache levels below). Re-tested 2026-10-15 with Invoke-SccacheWriteProbe.ps1, run twice: the first run wrote without an error, the second lost 165 of 250 writes on the mount and none in a plain directory. Re-enable it with -BuildArg SCCACHE_MULTILEVEL_CHAIN=disk,webdav only after Test-SccacheWrite.ps1 and a media build with zero genai write errors say so. The C:\sccache mounts use sharing=shared because locked made the concurrent litert and tvm solves take turns; with L0 on, that means two sccache servers on one directory, which has never been measured here. A damaged L0 tree fails path-dependently with os error 3: point SCCACHE_DIR at a fresh subdirectory (C:\sccache\v3) instead of debugging it.

Two cache levels#

Since 2026-10-15 a build can use two levels: the host’s own WebDAV (level 0) and an S3 bucket that every host shares (level 1). A second machine then gets hits for what this one compiled, without each compile waiting on the slower link.

SCCACHE_MULTILEVEL_CHAIN = webdav,s3
   level 0: this host's dufs, http://<host-lan-ip>:5000   (~1 ms, one per host)
   level 1: the shared bucket, http://192.168.188.216:3900 (Garage on a VPS, WireGuard only)

Why these two backends. sccache allows each backend type once per chain, so two WebDAV servers cannot be chained. A disk level would be the natural level 0, but it loses writes on the cache mounts (above). The host’s dufs is already set up by Install-NewHost.ps1 on every build host, so the shared level speaks S3 instead.

How sccache 0.18 runs a chain (src/cache/multilevel.rs, checked 2026-10-15):

  • Reads try level 0 first. A hit on level 1 is copied back to level 0 in a background task.

  • Writes go to level 0 inside the compile (SCCACHE_MULTILEVEL_WRITE_ERROR_POLICY default l0). Every slower level gets its write in a background task, so no compile waits on the upload.

  • A stopping server drops the background tasks that have not finished. Complete-SccacheServerSession therefore calls Wait-SccacheUploadDrain before --stop-server. It polls --show-stats --stats-format json until level 1’s writes plus write failures reach level 0’s writes, and gives up after 600 s with a warning. Each level-0 write starts exactly one write per slower level; backfills count apart (backfills_to), and preprocessor entries count in neither. A cold stage over a slow upload therefore ends with a wait, not with lost objects; whatever is left after 600 s, a later build uploads again.

Measured end to end on 2026-10-15 (BuildKit probe on bk-windows-toolchain, sccache 0.18.0, clang-cl, 20 fresh one-function C files, the real key as env secrets):

Phase

Result

webdav,s3, cold

20 misses; level 0 and level 1 each 20 writes, 0 write failures

Wait-SccacheUploadDrain

every upload done after 0 s (L1 (s3): 20 written, 0 failed of 20)

A fresh server on s3 alone, same files

20 hits, 0 misses: the objects came from the bucket

The next RUN

neither AWS_* variable set

The JSON counters the drain reads look like this, one object per level: {"name":"L1 (s3)","writes":20,"write_failures":0,"hits":0,"misses":20,"backfills_from":0,"backfills_to":0,…}. A wrong secret does not fail quietly: the server does not start at all (Server startup failed: … S3Error { code: "AccessDenied", message: "Forbidden: Invalid signature", resource: "/sccache/.sccache_check" }), so the compiles of that RUN run uncached. That is how the first probe found a command line stored as the secret.

The key never enters an argument, a layer or the image history. Build-Buildkit.ps1 passes it as two env-sourced BuildKit secrets, --secret id=sccache_s3_key_id,env=… and id=sccache_s3_secret,env=…. Every sccache RUN mounts them as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (--mount=type=secret,…,env=…). Probed on this host with a dummy secret (2026-10-15):

  • The variable is set inside the RUN, and is gone in the next RUN.

  • A build without the secret runs, with the variable unset.

  • A file-target secret also works, but needs a forward-slash target (target=C:/…), and it leaves a 0-byte placeholder in the layer. The env form leaves nothing.

The bucket’s endpoint, name, region and SCCACHE_S3_USE_SSL are ordinary build args, so they appear in the image history next to the WebDAV endpoint. They are reachable only over the VPN. The publish gate fails an image whose environment carries AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY or AWS_SESSION_TOKEN (WindowsImageEnv.Common, and verify_image_env.py with the shared fixture).

Setting up a client host (once per host, after Install-NewHost.ps1):

# asks for the secret without echo, tries it against the bucket, writes the five KATAGLYPHIS_SCCACHE_S3_* user variables
pwsh -File windows\scripts\host\Set-SccacheS3Level.ps1 -Endpoint http://192.168.188.216:3900 -AccessKeyId <key id>
pwsh -File windows\scripts\host\Set-SccacheS3Level.ps1 -ReportOnly       # what is set (the secret as set/missing)
pwsh -File windows\scripts\host\Test-HostSetup.ps1                        # "sccache S3 level reachable"
pwsh -File windows\scripts\host\Copy-SccacheCacheToS3.ps1 -DryRun         # optional: seed the bucket from this host

  • Variables. Build-Buildkit.ps1 reads them from the process, the user and the machine environment, in that order, so a shell that started before them still finds them. The parameters -SccacheS3Endpoint, -SccacheS3Bucket and -SccacheS3Region override them. The run then prints sccache levels: webdav … -> s3 ….

  • Defaults. Bucket sccache, region garage.

  • The secret is checked before it is stored. One with whitespace is refused (a pasted command line was stored once, 2026-10-15). With s5cmd on PATH the script lists the bucket with the key first: AccessDenied or Invalid signature stores nothing, an unreachable server only warns.

  • Copy the command before the key, or take it from the history. Copying the command after the key replaces the key in the clipboard, and the paste at the prompt is the command.

  • Server side. linux/sccache-s3/README.md has the VPS: WireGuard through the FritzBox, Garage, the firewall rule and the size cap.

  • Seeding. Copy-SccacheCacheToS3.ps1 copies the dufs tree into the bucket 1:1. Both backends store an object under the same key, a/b/c/<hash> (normalize_key), with no key prefix. preseed\ and .sccache_check stay out.

The app builds in a reused container get the level too (2026-10-15). Invoke-ContainerBuild, which the Windows lanes of OmniAccelerANT, AccelerANTgine and Ballett call through Invoke-RepoContainerBuild.ps1, hands it to the build’s docker exec:

  • By name only. Get-HostSccacheS3ExecEnv (WindowsContainerBuild.Reuse) reads the five KATAGLYPHIS_SCCACHE_S3_* settings and returns SCCACHE_ENDPOINT, SCCACHE_BUCKET, SCCACHE_REGION, SCCACHE_S3_USE_SSL, the chain and both AWS_* key halves. Invoke-WithProcessEnvironment sets them for the docker CLI alone, and the exec gets a value-less -e NAME for each, which docker resolves from its own environment. The key is on no command line, and the reused container’s config never holds it.

  • Per exec, never on the docker run. A reused container keeps whatever its run was given, so a setting added later would never reach it.

  • Left out when incomplete or unreachable, with a warning. A sccache server whose S3 level fails its startup check (403, or no answer) caches nothing at all, which the image chain saw once as Server startup failed … AccessDenied.

  • The bind-mount transport does not get it. That container is kept after a failure for inspection, and its config would then hold the key. CI has no route to the bucket anyway.

  • A server outlives the exec in a reused container. Background uploads therefore finish without a drain. A server started before the level was set keeps its old configuration until its idle timeout (600 s) or sccache --stop-server in the container.

  • Not yet run against a live Stevedore engine (dockerd was stopped on 2026-10-15); the suite is ContainerBuild.SccacheS3.Tests.ps1.

The Linux lanes have no remote sccache tier at all yet, only the cache inside the container, so the bucket does not reach them.

What happens when a level is missing, so neither level can stop a build:

Situation

Effect

No KATAGLYPHIS_SCCACHE_S3_* set

WebDAV alone, as before 2026-10-15

Half of them set

The driver warns, names what is missing and leaves the S3 level out

The bucket’s port does not answer the host

The driver warns and builds on WebDAV alone

Inside a RUN: no key, or the endpoint unreachable

Clear-UnreachableSccacheEndpoint drops the S3 variables and takes s3 out of the chain, which then runs on WebDAV alone; an unreachable WebDAV level goes the same way and leaves the bucket

A wrong key

The server refuses to start (AccessDenied, Invalid signature); Set-SccacheS3Level.ps1 refuses such a key when s5cmd is on PATH, and the row below catches one that slips through

The bucket stalls the server’s startup check

One level that fails its check stops the whole sccache server (multilevel.rs), and the client gives up after 10 s. Start-SccacheServerSession then drops the S3 level (Remove-SccacheS3Level) and starts the server again on WebDAV. The app builds’ exec passes SCCACHE_SKIP_CACHE_CHECK=true instead, because their own build starts the server

The bucket dies in the middle of a RUN

Writes fail and are counted; reads wait for the TCP connect on each level-0 miss. The RUN finishes slower, it does not fail

Measured on 2026-10-16: a flaky bucket stopped a chain, which the startup fallback above now prevents. The VPS stopped answering through the tunnel at about 00:49 (no TCP connect, no ping). The nvidia ORT stage, which was still compiling, finished: 608 of 2364 level-1 writes failed, and level 0 lost none. The next stage, FFmpeg, passed the 2 s TCP probe. Its server still timed out in the startup check (Timed out waiting for server startup. Maybe the remote service is unreachable?), and every compile failed. Before the fallback, the bucket was a hard dependency of every RUN that named it. The cause was on the network side: the FritzBox’s IPv6 prefix changed, and wg had resolved the endpoint’s name once, so the VPS kept sending to the old address for about 8 hours. wg-reresolve-dns@wg0.timer now follows such a change in about 20 s (the runbook’s WireGuard section).

Adding the secret mounts and ARGs changed the RUN instructions, so the chain after 2026-10-15 re-keys every sccache stage once. Those stages compile warm from level 0.

Tests: BuildDriver.SccacheLevels.Tests.ps1 (the config, the secrets, and every one of the 16 sccache RUNs carrying both mounts and the ARGs), Build.SccacheEndpointProbe.Tests.ps1 (§ the S3 level), SourceBuild.SccacheUploadDrain.Tests.ps1, and the publish gate’s credential cases.

Bazel’s cache#

The two LiteRT bazel builds keep their actions on the host’s dufs too (2026-10-15). sccache never sees Bazel’s compiles, and each build starts from an empty output base. Until now the LiteRT-LM build (5244 actions, ~400 s) and the ai-edge-litert wheel (5655 actions, ~226 s) therefore ran in full in every variant: cpu, nvidia and rocm alike in the chain of 2026-10-14.

  • The option. Get-BazelRemoteCacheArg (WindowsBazel.Common) adds --remote_cache=<SCCACHE_WEBDAV_ENDPOINT>/bazel and --remote_download_outputs=all to the option list each build shares with its aquery. With no endpoint in the RUN it adds nothing. With one that does not answer a HEAD within 3 s it adds nothing and warns. The build then runs as before.

  • dufs as the cache. dufs speaks Bazel’s HTTP cache protocol as it is (checked 2026-10-15): a PUT to bazel/cas/<sha256> creates the directories (201), a GET returns the object, HEAD answers 200, and a missing key is a 404.

  • Every output is downloaded, not Bazel 7’s toplevel default, because the install steps copy files from bazel-bin that are not top-level outputs.

  • The variants share it. cpu, nvidia and rocm run the same actions, so the second and third variant of a cold chain hit on the first.

  • Where it lives. Bazel’s objects sit under bazel\ in the dufs directory, beside sccache’s. Nothing evicts them; delete bazel\ to reset. Copy-SccacheCacheToS3.ps1 leaves the directory out of the bucket, which only sccache reads. Bazel cannot speak S3, so this cache stays on each host.

  • Tests. Bazel.RemoteCache.Tests.ps1: no endpoint, a dead one, a live HTTP stub that answers 404, and both scripts and the seed wired. Not yet measured in a chain.

What the published image carries#

The build host’s sccache settings are ARGs, never ENV (2026-09-23). Until then four Dockerfiles ENV’d them: the patched-llvm stage (#164), media-builder’s common stage, the merge stage and Dockerfile.rocm-migraphx. :winamd64 (digest 3137eebe…, built 2026-09-22 from hub 0d85b8c1) shipped SCCACHE_WEBDAV_ENDPOINT=http://192.168.188.116:5000, the owner’s LAN WebDAV. On a GitHub runner that address is unreachable. sccache 0.18.0 checks its storage when the server starts and exits when the check fails (tcp connect error). Every client then times out after 10-12 s with exit 2, CMake reports clang-cl as “not able to compile a simple test program”, and the AccelerANTgine and OmniAccelerANT Windows lanes went red (runs 35921977157, 35912798986).

Variable

Where it lives

Why

SCCACHE_WEBDAV_ENDPOINT

ARG, no default, redeclared in every compiling stage

a LAN address; nothing outside the build host reaches it

SCCACHE_MULTILEVEL_CHAIN

ARG, no default

the build host’s cache layout (#99); it names a remote only the build host has

SCCACHE_FORCE_LOCAL

ARG, no default

the hub’s own diagnostic switch, read only by Test-SccacheRemoteConfigured

SCCACHE_DIR, SCCACHE_CACHE_SIZE, SCCACHE_ERROR_LOG, SCCACHE_LOG, SCCACHE_IDLE_TIMEOUT

ENV, the runtime defaults

container-local: a path, a size, a log file and level, a timeout. None names a host, so they behave the same on the build host, a runner and a laptop, and the Linux image ships the same set (build-cache-tiers.md)

The runtime defaults keep their values. Initialize-BuildCacheEnvironment and Get-SccacheContainerEnv already override SCCACHE_DIR for consumer builds. SCCACHE_LOG=warn stays beside SCCACHE_ERROR_LOG because without a level the error log is never written (#90). What a consumer build on the build host does lose is the remote tier, which it only ever reached through the leaked ENV; Invoke-ContainerBuild now forwards it at run time: § The build host’s remote tier, at run time.

Why every compiling stage redeclares the ARG. An ARG is in the environment of the RUNs of the stage that declares it, and only that stage. The media-core chain crosses solves: media-core-built-ffmpeg is FROM ${MEDIA_CORE_ONNX_IMAGE}, an image, so nothing declared in common reaches it. Each of the seven compiling stages in Dockerfile.media-builder therefore declares SCCACHE_WEBDAV_ENDPOINT, SCCACHE_MULTILEVEL_CHAIN and SCCACHE_FORCE_LOCAL right above its RUN, as do patched-llvm, the merge built stage and rocm-migraphx. With no default an ARG is simply absent when the driver does not pass it, which is what the old empty ENV meant too (sccache 0.18 ignores an empty value, and Test-SccacheRemoteConfigured treats one as unset). A missed stage would compile uncached without a word, so the static lint refuses it: a RUN that mounts both C:\sccache and C:\sccache-logs must see ARG SCCACHE_WEBDAV_ENDPOINT in its own stage.

Three gates hold it:

  • Static, in preflight: linux/scripts/verify_image_env.py --dockerfile, run by lint-dockerfiles.sh over both lanes’ Dockerfiles. It refuses an ENV of a build-host sccache name, an ENV that expands one, a LAN literal in an ENV or ARG default, and the missing per-stage ARG above.

  • Publish, Windows: Build-Buildkit.ps1 solves windows/Dockerfile.publish-gate (Invoke-BkPublishGate) at three points: on every BASE_IMAGE a stage inherits that this run did not build, before that stage solves (§ An image this run did not build); on the toolchain right after its solve; and on the final tag after the smoke gate and before -FinalTar/-PushRef. -SkipSmokeGate skips none of them. Its one RUN mounts WindowsImageEnv.Common.psm1 and runs Assert-ImageEnvPublishable over the Process, Machine and User scopes. The Process scope is the config’s ENV; a RUN that set a Machine variable publishes that too. No ARG follows its FROM, because an ARG would join the environment under test, and it does not run the entrypoint, because VsDevCmd adds variables the image does not carry.

  • Publish, Linux: _manifest_image_env_gate, the first thing create_manifest in build-runtime-manifest.sh does. It reads each wrapper’s config ENV (nerdctl image inspect --platform linux/<arch>) and runs the same Python matcher. No switch waives it: not RUNTIME_IMAGE_SMOKE=0, not --force, and --manifest-only/--repair run it too; see build-cache-tiers.md.

What counts as a leak, identically on both lanes (one case file, linux/scripts/tests/image-env-cases.json, grades the PowerShell and the Python matcher): any sccache remote-backend name or the two layout names above, whatever the value; and an RFC1918 (10/8, 172.16/12, 192.168/16) or link-local (169.254/16, fe80::/10) address in a host position — after scheme://, @ or \\, as a whole value or list item, or followed by :port or /. A dotted quad inside a path (C:\TensorRT-10.13.3.9\lib) is not a host, and a whole value under a *VERSION* name is a version: the base bakes every versions.env key into the Machine environment, CUDA_WINDOWS_ARM64_CURAND_VERSION=10.4.4.72 among them.

What the gates do not cover. Hostnames and loopback (http://buildhost:5000, 127.0.0.1), files inside the image, and the image history: BuildKit records a RUN’s build args in its created_by, so the final image’s history still names the endpoint for every compiling RUN. That is metadata, not configuration, and nothing reads it at run time. Moving the endpoint into a secret mount would remove it; that is open as backlog #177.

An image this run did not build#

A stage inherits its parent’s config ENV: FROM an image or an earlier stage copies it, and only ARGs stop at the boundary. So the Dockerfile fix protects only images built from the fixed Dockerfiles. A bk-windows-toolchain (or -media, -torch) left by a run from before 2026-09-23 still carries the endpoint, and every stage built on it inherits it. The merge built stage used to overwrite the variable with its own ENV; now it passes the parent’s value straight through to the published image. Two ways a stale parent gets reused: a run whose -Stages leaves out the stage that produces it, and a driver process that was already running when the checkout moved (PowerShell parsed the old script, so that process has no gate at all).

Invoke-BkStage therefore grades every BASE_IMAGE this run did not build and has not graded yet, before it solves the stage. A stale one stops the run in seconds, not hours later at the final gate:

[bk:publish-gate:bk-windows-toolchain] buildctl failed (exit 1) — full log: ...
[bk:publish-gate:bk-windows-toolchain] docker.io/local/kataglyphis:bk-windows-toolchain was not built
by this run and failed the publish gate: a build-host setting in its ENV (built before 2026-09-23?)
or no such image. Rebuild it: put the stage that produces it in -Stages, from a fresh driver process.

The fresh toolchain is graded right after its solve as well: every later stage inherits it, and a RUN can write the Machine or User scope, which no Dockerfile lint sees. A parent this run built is not graded again; the final image always is. Each gate solve takes seconds, and a repeat on an unchanged image is a cache hit. Tests: ImageEnv.PublishGate.Tests.ps1 lifts Invoke-BkStage out of the driver and runs it against a fake buildctl.

Restart a chain across this fix; never resume one. Start a fresh driver process with toolchain in -Stages, so that no toolchain built from the old Dockerfile is reused.

Cache impact of the change: the patched-llvm stage and every media stage re-key (and, on the rocm lane, rocm-migraphx); the toolchain’s built (CPython) stage, base and the sdk slot do not. The full re-key set is in CHANGELOG.md, 2026-09-23. The 2026-09-24 follow-up (the parent gate, the probe, the forward) re-keys nothing beyond that: its only image input is two modules that just the final windows/Dockerfile copies.

The consumer-side probe#

Images published before 2026-09-23 still carry the endpoint, so the consumer side defends itself. Enable-SccacheCompilerWrapper (WindowsBuild.Common.psm1) is the one place both consumer wiring sites go through — Initialize-BuildCacheEnvironment and Invoke-CmakeConfigureAndBuild — and before it sets a launcher it calls Clear-UnreachableSccacheEndpoint. That function resolves the endpoint’s host and connects to every address it resolves to at once, within one 2 s budget (-TimeoutMs) that includes the resolution; the first connection that succeeds counts. One address at a time would lose a reachable host: Windows takes about 2 s to refuse a closed IPv6 address, which localhost and a dual-stack name resolve to first, so an IPv4-only listener behind http://localhost:<port> was removed after 2049 ms and is now kept in 37 ms (measured 2026-09-24). Reachable: nothing changes, so the build host keeps its cache. Unreachable or not a URL: it removes SCCACHE_WEBDAV_ENDPOINT from the process environment (and SCCACHE_MULTILEVEL_CHAIN when that names webdav), writes one WARN naming the endpoint, and sccache caches on local disk instead of failing every compile:

WARNING: sccache: SCCACHE_WEBDAV_ENDPOINT=http://192.168.188.116:5000 is unreachable
(TCP 192.168.188.116:5000, no connection within 2000 ms) - removed for this process, ...

The image build does NOT use this path. Its compile stages wire sccache through WindowsSourceBuild.Common, gated on Test-SccacheRemoteConfigured, where an unreachable endpoint must stay a loud failure rather than quietly become an uncached multi-hour build. Tests: windows/scripts/tests/Build.SccacheEndpointProbe.Tests.ps1 (a loopback port bound but never listening, a listening one, localhost against an IPv4-only listener, a 200 ms bound against Windows’ ~2 s refusal, an unresolvable name, and in-suite mutants for the removal, the bound and the localhost case, each showing that case can fail on the host it runs on).

The build host’s remote tier, at run time#

Until 2026-09-23 a consumer build on the build host reached the WebDAV tier only through the leaked ENV. Now Invoke-ContainerBuild (WindowsContainerBuild.Reuse) forwards this host’s SCCACHE_WEBDAV_ENDPOINT and SCCACHE_MULTILEVEL_CHAIN into the container as run-time -e entries (Add-HostSccacheRemoteEnv), on both transports and for every caller: AccelerANTgine passes -CacheEnv (Get-SccacheContainerEnv), and OmniAccelerANT’s agentic-loop driver passes no -CacheEnv at all. A host without the variables, such as a GitHub runner, forwards nothing. A key the caller sets in -CacheEnv wins, and SCCACHE_WEBDAV_ENDPOINT = '' opts out. The container’s probe (above) still drops an endpoint it cannot reach, so a bad forward costs one WARN, not a build.

Two limits. A reusable container keeps the environment it was created with, so it picks the forward up when it is recreated: a new image does that, -FreshContainer forces it. And a docker run by hand forwards nothing (OmniAccelerANT’s documented parity run is one); add -e SCCACHE_WEBDAV_ENDPOINT=$env:SCCACHE_WEBDAV_ENDPOINT yourself. Tests: WindowsContainerBuild.Reuse.Tests.ps1 (Add-HostSccacheRemoteEnv) and Modules.Orchestrators.Tests.ps1 (the build run carries the -e).

Note (.dockerignore): The repo .dockerignore must NOT contain a windows/ exclusion — the Windows Dockerfiles COPY from the windows/scripts/ directory within the build context. If windows/ is added to .dockerignore, the COPY steps will fail with “file not found in build context”. This exclusion is safe for Linux builds (they read only linux/ from the same root context) but breaks Windows builds.