Fresh Windows Host Bring-Up (BK lane + repo gates)

Fresh Windows Host Bring-Up (BK lane + repo gates)#

Goal: take a fresh Windows 11 machine to (1) a green .\windows\build-buildkit.ps1 -Gpu on the preferred BuildKit/containerd lane and (2) working repo gates (lint / tests / preflight) — using only this checklist and the sections it links. Every step names its shell ([admin] / [non-admin]) and ends with a verify command. Deep rationale lives in Windows Build Image; this page is the ordered path through it, not a replacement.

Check yourself against the machine, not against this page. windows/scripts/verify-host-setup.ps1 asserts every claim below and prints a fix for each failure. Run it first (to see what a fresh box still needs), last (to confirm bring-up), and after any host change:
pwsh -File windows\scripts\verify-host-setup.ps1 -SccacheEndpoint http://<lan-ip>:5000
It needs no admin (Defender exclusions are reported UNKNOWN rather than skipped, so their absence cannot look like success) and exits 1 on any failure.

pwsh, never powershell, and that includes the ADMIN window. Every script here carries #requires -Version 7.0; under Windows PowerShell 5.1 they refuse with a #requires message and change nothing. That refusal is easy to miss: wrapped in a command that collects only the pipeline stream it looks exactly like “the script ran and did nothing” — it cost two round trips on 2026-08-08 against apply-buildkitd-gcpolicy.ps1, whose effect (reservedSpace in the deployed toml) then silently stayed at the old value. Verify the effect, not the exit.

Do not hardcode a pwsh path. On this host pwsh 7 is the MSIX/Store build under C:\Program Files\WindowsApps\Microsoft.PowerShell_<version>_x64__…\, not the MSI at C:\Program Files\PowerShell\7\ — the path carries the version and moves on every update. Resolve it with (Get-Command pwsh).Source (app-execution aliases occasionally fail to resolve under elevation; fall back to a recursive search of WindowsApps for Microsoft.PowerShell_*). This page and that script are two views of one contract — change them together. The reason it exists: until 2026-08-07 the CNI section here handed fresh hosts a .conf template that silently broke the entire nerdctl lane, and it read as authoritative for days. Prose cannot be executed; that is the whole failure mode this guards.

Phases:

A — one-time host provisioning [admin]
B — repo checkout + gate tooling [non-admin]
C — build-service tuning [admin] (debug flags, log limit, GC policy, Defender, sccache/dufs)
D — per-boot / per-run checks
E — first build + verification

Fast path for Phase A5 + C: setup-new-host.ps1. Once the interactive steps are done (A1 Stevedore+reboot, A2 docker-users + a new shell, A3 services, B0 Git/B1 repo), a single elevated run of windows\scripts\setup-new-host.ps1 does the entire scriptable half — CNI .conflist authored from the live vEthernet (nat) subnet (magic constants removed: it derives network/prefix + gateway at runtime), then apply-containerd-config.ps1 (debug flags, teardown env var, Defender exclusions, .conf derive), apply-buildkitd-gcpolicy.ps1 + the step-log env var, the patched runhcs shim (built from hcsshim source if no -ShimPath is given — Go installed via scoop as needed — then deployed), and dufs (scooped if missing, started serving the cache dir, ONLOGON task registered, machine SCCACHE_WEBDAV_ENDPOINT set to the host’s LAN IP).
pwsh -File windows\scripts\setup-new-host.ps1 -ReportOnly   # plan first (safe, non-admin)
pwsh -File windows\scripts\setup-new-host.ps1               # admin - bring the host to green
pwsh -File windows\scripts\setup-new-host.ps1 -ShimPath C:\src\hcsshim\containerd-shim-runhcs-v1.exe
It is idempotent and refuses to run while a build is live (unless -Force). The rest of A5/C below is what the script does by hand — read it to understand, run the script to execute.

⚠️ FIRST CHECK on any Windows host doing container builds. The build-COPY-commit failure hcsshim::ActivateLayer 0x20 (buildkit) / mkdir \\?\Volume{<GUID>}\C:. — “Der Verzeichnisname ist ungültig” (docker legacy) hits BOTH engines, deterministically, and survives -NoCache, service restarts, Defender exclusions, a full store reset and a reboot. Root cause (RESOLVED 2026-08-10, superseding the 2026-08-09 “faulty Adrenaline install” verdict): an ENABLED AMD RDNA4 dGPU + Adrenalin driver locks freshly-written container layers (upstream docker/for-win#14977; A/B-proven on the RX 9070 XT host — dGPU off → green, on → red, same boot). Probe and repair order: (1) pwsh -File windows\scripts\probe-build-copy.ps1 -Heavy (the committed probe; only a -Heavy-green verdict counts — light lanes can be green while RUN-layer finalize is broken), then (2) on RDNA4 hosts: elevated toggle-rdna4-gpu.ps1 -Disable → re-probe -Heavy → build → re-enable (the Assert-NoActiveRdna4Gpu preflight enforces this), and (3) after ANY red finalize: REBOOT before further experiments — failed finalizes wedge hcs state and falsify every later A/B. NOT an ISO/OS-corruption problem: sfc/DISM report 0 components corrupt, but the severity DOES move with Windows updates (post-KB5101684 even tiny RUN layers trip). The Linux cross lane and all repo gates are unaffected. Remaining-valid diagnostics: while build-COPY fails in both engines (ApplyDiff), docker run + docker commit still works (CommitLayer OK) — so the classic lane’s run+commit stages stay viable once a FROM image exists; full bootstrap still needs a working BK lane (every Dockerfile has a COPY).

[2026-08-09 end-of-session narrative — SUPERSEDED by the 2026-08-10 update below; kept as history. Its closing “repair order” is obsolete: the modern order is in the FIRST paragraph above.] With Adrenaline fixed + pristine Stevedore the buildkit lane on the discovered host STILL refused multi-level commits (any layer writing into an existing parent dir: ActivateLayer 0x20 at snapshotter reimport, identical on buildkit 0.32.0 and 0.32.2, on windowcon/native/windowssvm snapshotter names). It was only cleared by a Windows in-place repair upgrade (official ISO, same build 26200/25H2, “keep files and apps”) — after it every layer commits, including writes into existing dirs. Only the FINAL export (reimport of the committed snapshot) still trips 0x20 on that host, where Defender’s engine (MsMpEng) is unkillable by design and the identical Stevedore+OS stack builds the BK lane fine on the working machine. Order on a host with this symptom: probe → reinstall AMD Adrenaline → probe → if commits still fail, in-place-repair Windows → probe → BK lane commits; residual export 0x20 = host-residual (classic lane or the healthy host).

Update (2026-08-10): the 2026-08-09 export-residual verdict was polluted by two bugs in the probe itself (unquoted dest=$outDir sent buildctl a literal $outDir; the -Docker lane crashed on a $docker/[switch]$Docker collision — AGENTS.md § Windows Build Invariants, ArgQuoting traps). With the fixed probe (now exporting type=image,...,unpack=true, the same output path build-buildkit.ps1 uses, and exiting non-zero per failing lane), the discovered host’s LIGHT probe lanes are green on buildkit (commit + export + unpack) — but a light-green probe is NOT chain-green: the real chain’s first COPY after the heavy pwsh-install RUN still died deterministically (ActivateLayer 0x20 at child-snapshot finalize/reimport, FRESH snapshot IDs under -NoCache, so not poisoned cache). Minimal repro, now committed as the probe’s -Heavy lane (Dockerfile.heavy): RUN writing 2×100 MB, then a one-file COPY — the COPY’s finalize fails while the fresh heavy parent layer is still held. RESOLVED the same day by a same-boot A/B: the holder is the ENABLED RDNA4 dGPU (RX 9070 XT + Adrenalin) — disable it and both tiny and heavy RUN-layer finalize go green on the first try (upstream: docker/for-win#14977, RDNA3.5/4, open). Severity tracks the Windows patch level: pre-KB5101684 only heavyweight RUN layers tripped; after it, even 10-byte RUN layers. COPY-only layers are safe either way. Falsified on the way (all still-red at the time): Defender exclusions/toggles, WSearch/ SysMain, daemon bounces, a vmcompute restart, minifilter detaches (no third-party filters exist), --no-cache, settle delays, reboots, a nanoserver base and split solves. Failed finalizes additionally WEDGE hcs state — after one, even tiny RUN layers fail until a reboot, which is why earlier A/B rounds contradicted each other. Build workflow on RDNA4 hosts: elevated toggle-rdna4-gpu.ps1 -Disable → build (display falls back to the iGPU; DirectML-on-host is unavailable during the window) → re-enable. build-buildkit.ps1 refuses to start while the dGPU is enabled (Assert-NoActiveRdna4Gpu; -SkipHostChecks overrides). The 2026-08-09 Adrenaline-reinstall / in-place-repair root-cause claims are SUPERSEDED — they coincided with patch/reboot changes that moved the trigger threshold. Also: never judge a host by a type=local export of a Windows image — the local exporter itself dies mid-receive (error from receiver: ... file already closed) even on a healthy host.

Phase A — One-time host provisioning [admin]#

A1. Install Stevedore, reboot#

winget install stevedore     # or: choco install stevedore; custom dir: --custom="INSTALLDIR=D:\Stevedore"

Reboot afterwards — this enables the Windows Containers feature. Details and the custom-INSTALLDIR path substitution: Windows Build Image § Prerequisites.

Path quirk: Stevedore puts docker.exe, buildctl.exe, nerdctl.exe and containerd.exe under <INSTALLDIR>\bin\, but dockerd.exe and buildkitd.exe at the install root. When in doubt, trust the service ImagePath in the registry, not a hand-typed path.

Verify:

Get-Service stevedore, containerd, buildkitd          # all three must exist
& "$env:ProgramFiles\Stevedore\bin\docker.exe" version

A2. docker-users group membership#

The installer normally adds the installing user; the group grant is what makes non-admin builds possible (dockerd’s and buildkitd’s named pipes are ACL’d to docker-users; containerd’s pipe stays admin-only by upstream design).

net localgroup docker-users $env:USERNAME /add        # only if missing

Log out/in after a group change. Verify (non-admin shell):

whoami /groups | Select-String docker-users           # must match
& "$env:ProgramFiles\Stevedore\bin\buildctl.exe" --addr npipe:////./pipe/buildkitd debug workers

A3. Services running, reboot-safe#

Get-Service stevedore, containerd, buildkitd | Set-Service -StartupType AutomaticDelayedStart
Start-Service stevedore, containerd, buildkitd

Verify: Get-Service stevedore, containerd, buildkitd → all Running.

A4. Conditional dockerd fixes (only on the matching symptom)#

stevedore service won’t start / 1053 timeout → a stale Docker Desktop C:\ProgramData\docker\config\daemon.json conflicts with the service’s --host flags. Remove/rename it: Windows Build Image § Stevedore Setup Fixes, Fix 1.
docker build fails runtime "com.docker.hcsshim.v1" binary not installed → apply Fix 2 (re-register with --default-runtime=io.containerd.runhcs.v1). Note: current Stevedore releases may not need this — the reference host runs WITHOUT the flag; apply it only when the symptom appears.

Verify: & "$env:ProgramFiles\Stevedore\bin\docker.exe" info succeeds.

A5. CNI nat conf (required — RUN steps have NO network without it)#

nat.exe already ships in C:\Program Files\containerd\cni\bin; only the conf is missing on a fresh host. Install it as a .conflist (plugin-list form), NOT a bare .conf — see the format note below — using the subnet from Windows Build Image § Getting it going, step 2. The ipam.subnet/GW values MUST match the live vEthernet (nat) adapter (ipconfig), and dockerd restarts can silently re-create that network on a new subnet (the driver’s preflight fail-fasts on drift with the exact fix).

// C:\Program Files\containerd\cni\conf\0-containerd-nat.conflist
{
    "cniVersion": "0.3.0",
    "name": "nat",
    "plugins": [
        {
            "type": "nat",
            "master": "Ethernet",
            "ipam": {
                "subnet": "<subnet of the vEthernet (nat) adapter>",   // DERIVE, don't copy: see below
                "routes": [ { "GW": "<the adapter's own IP>" } ]
            },
            "capabilities": { "portMappings": true, "dns": true }
        }
    ]
}

No magic subnets — derive them. Every example number shipped in these docs (172.31.32.0/20, etc.) was a snapshot of ONE host and went stale; the only correct values are the live adapter’s. setup-new-host.ps1 derives them automatically; to do it by hand:

$n = Get-NetIPAddress -AddressFamily IPv4 | Where-Object { $_.InterfaceAlias -eq 'vEthernet (nat)' }
$n.IPAddress, $n.PrefixLength     # adapter IP + prefix -> GW + subnet (e.g. 172.21.32.1 / 20 -> subnet 172.21.32.0/20)

The ipam.subnet/GW MUST match that live vEthernet (nat) adapter (ipconfig), and dockerd restarts can silently re-create that network on a new subnet (the driver’s preflight fail-fasts on drift with the exact fix).

Install BOTH forms — conf AND conflist (corrected 2026-08-07, same day, after the conflist-only state cost a launched chain). Same content, two filenames:

0-containerd-nat.conflist is required by nerdctl, which cannot parse a bare .conf — it indexes plugins[0] with no length check and PANICS with index out of range [0] with length 0, both in network create (netutil_windows.go:40) and in run (container_network_manager.go:857).
0-containerd-nat.conf is required by buildkitd. With only the conflist present, BuildKit RUN steps get no network adapter at all: a probe container showed an empty ipconfig, DNS failed, and a raw TCP connect to a literal GitHub IP returned “unreachable network”. The containerd debug log showed the HcsCreateComputeSystem spec for buildkitsandbox with no networking block. Restoring the .conf and Restart-Service buildkitd -Force fixed it on the spot: IPv4 172.31.44.107, gateway 172.31.32.1, DNS 192.168.188.1, github.com resolved.

The earlier claim here that “containerd and BuildKit read either form” was wrong. The 2026-08-07 conversion fixed nerdctl and silently killed the buildctl lane; it went unnoticed because no chain build ran in between. Keep both files, and when you edit one, edit both — build-buildkit.ps1 fail-fasts on a missing .conf (Get-CniConfFormIssue), but nothing detects the two drifting apart in content.

Verify:

Test-Path 'C:\Program Files\containerd\cni\conf\0-containerd-nat.conflist'  # True (nerdctl)
Test-Path 'C:\Program Files\containerd\cni\conf\0-containerd-nat.conf'      # True (buildkitd)
ipconfig | Select-String -Context 0,4 'vEthernet \(nat\)'                   # subnet matches the conf

The drift guard reads either name (Get-CniNatSubnetDrift checks .conflist then .conf) — note its contract is “file absent = nothing to judge”, so a conf under any OTHER name turns the guard into a silent no-op.

Verify nerdctl works too (ADMIN shell — nerdctl opens containerd’s pipe, which is Administrator-only; buildctl stays non-admin because buildkitd has --group docker-users and containerd has no equivalent). This is the fastest confirmation that the conflist is correct, because nerdctl is the component that is picky about it:

nerdctl --namespace buildkit run --rm --network nat `
    docker.io/local/kataglyphis:bk-windows-base cmd /c ipconfig

Expect an IPv4 address inside the conf’s subnet with the nat gateway. Two warnings are normal and harmless (default network named "nat" does not have an internal nerdctl ID, and a failed to remove hosts file on exit). If instead you get panic: runtime error: index out of range [0] with length 0, the config is still in bare-.conf form.

If nerdctl is “not recognized”: C:\Program Files\Stevedore\bin is on the MACHINE path, so only shells opened AFTER the Stevedore install see it — open a new window rather than editing $env:Path.

Full recipe set (interactive shell into an image, nerdctl build, the ENTRYPOINT trap, zombie cleanup): Windows Build Image § nerdctl lane.

Phase B — Repo checkout + gate tooling [non-admin]#

B1. Git for Windows + clone#

Install Git for Windows (winget install Git.Git). The gates run under Git Bash (C:\Program Files\Git\bin\bash.exe) — do NOT rely on a bare bash on PATH, which on many hosts resolves to System32\bash.exe (WSL).

git clone --recurse-submodules git@github.com:Kataglyphis/Kataglyphis-ContainerHub.git
cd Kataglyphis-ContainerHub
git config core.hooksPath .githooks      # pre-commit runs the same checks CI enforces
git config core.longpaths true           # deep vendored trees; host LongPathsEnabled=1 recommended too

The external/ submodule (DocumANTation Sphinx theme) is optional for building images — it is only needed for cd docs && make html. A plain clone builds the Windows chain fine.
Line endings: .gitattributes pins the load-bearing files, so core.autocrlf=true (the Windows default, and what the reference host uses) is tolerated. The preflight crlf-guard check catches any *.sh that went CRLF in the working tree; after editing media .psm1/.ps1 files confirm git diff shows only your change, not a whole-file EOL flip (AGENTS.md § Windows Build Invariants).

Verify: git config core.hooksPath prints .githooks; git status is clean.

B2. PowerShell 7 (pwsh)#

Everything on this lane requires pwsh 7 (#Requires -Version 7.0 in every script; owner policy — see AGENTS.md).

winget install Microsoft.PowerShell

Verify: pwsh -NoProfile -c '$PSVersionTable.PSVersion' → 7.x.

B3. PSScriptAnalyzer + Pester (the PowerShell gates)#

CI parity pins (.github/workflows/windows-scripts.yml): PSScriptAnalyzer 1.25.0, Pester >= 5.7. Invoke-Tests.ps1 FAILS (never silently skips) when Pester >= 5 is missing.

Install-Module PSScriptAnalyzer -RequiredVersion 1.25.0 -Scope CurrentUser -Force
Install-Module Pester -MinimumVersion 5.7 -Scope CurrentUser -Force -SkipPublisherCheck

Verify:

Get-Module -ListAvailable PSScriptAnalyzer, Pester | Select-Object Name, Version

B4. Python for preflight/sync — dodge the Windows Store stub#

On a fresh Win11, python3 on PATH is the Microsoft Store stub (opens the Store instead of running). preflight.sh provides the escape hatch PREFLIGHT_PYTHON; the simplest working setup is uv:

scoop install uv          # or: winget install astral-sh.uv (install scoop first: https://scoop.sh)
uv python install 3.14    # puts python3.14.exe into %USERPROFILE%\.local\bin

Verify:

uv run --no-project python -V                              # Python 3.x
& "$env:USERPROFILE\.local\bin\python3.14.exe" -V          # if installed via uv python install

B5. shellcheck / hadolint / actionlint — nothing to install#

The preflight lint gates auto-bootstrap these: a PATH copy is used when present, otherwise the pinned release from versions.env is downloaded once into a version-keyed cache dir and SHA256-verified (lint-shell.sh, lint-dockerfiles.sh, lint-workflows.sh). First run needs network; a failed bootstrap fails the gate loudly (no silent skip).

B6. Run all three gates#

pwsh -File windows/scripts/Invoke-Lint.ps1                 # parse gate + PSScriptAnalyzer
pwsh -File windows/scripts/tests/Invoke-Tests.ps1          # harness + Pester suites

# Git Bash (not WSL bash):
PREFLIGHT_PYTHON="uv run --no-project python" bash linux/scripts/preflight.sh

Verify: all three exit 0. Preflight’s KNOWN_SLUGS array is the single source of the no-build gate list (don’t enumerate it in docs — enumerations drift; tests/test-preflight-slugs.sh keeps the array honest) — see AGENTS.md § Validation.

Phase C — Build-service tuning [admin]#

Do C1–C3 in this order: C1 and C2 are plain registry edits, C3 (apply-buildkitd-gcpolicy.ps1) preserves the existing flags, adds --config, and performs the one buildkitd restart. Never restart buildkitd while a build is solving.

C1. Permanent debug flags on containerd + buildkitd (owner policy)#

Debug logging stays PERMANENTLY ON on build hosts, so the next snapshotter incident carries its evidence immediately (owner decision 2026-08-04; if the log grows huge, truncate it — never disable the flags). Recipe and rationale: Windows Build Image § BuildKit/containerd lane (“How to capture the debug evidence again”).

Use the script — it is the source of truth for the containerd side:

pwsh -File windows\scripts\apply-containerd-config.ps1 -ReportOnly   # inspect, no admin needed
pwsh -File windows\scripts\apply-containerd-config.ps1               # admin; restarts containerd

containerd runs with no config.toml here — every setting lives in the service’s ImagePath/Environment registry values, which is why it needs a script to be reproducible at all (buildkitd has buildkitd.toml + apply-buildkitd-gcpolicy.ps1; this is the missing counterpart, added 2026-08-07). It owns three things a fresh host must have: the debug flags below, CONTAINERD_SHIM_RUNHCS_V1_TEARDOWN_TIMEOUT (the runhcs shim inherits the SERVICE environment — a shim built from the upstream patch keeps its 30 s defaults and silently reverts to the ExportLayer 0x3 defect without it), and the load-bearing Defender exclusions. Never run it while a build is solving.

The manual equivalent, if you want to see what it does — set via registry, because sc.exe quoting mangles these in PowerShell:

Set-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\containerd' -Name ImagePath `
  -Value '"C:\Program Files\Stevedore\bin\containerd.exe" --run-service --service-name containerd --log-level debug --log-file C:\ProgramData\containerd\containerd-debug.log'
# buildkitd: insert --debug before --run-service in ITS ImagePath (exe path may be
# the install ROOT, not bin\ — read the current value first and keep every flag):
Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\buildkitd' | Select-Object -Expand ImagePath

Restart containerd (Restart-Service containerd -Force, then Start-Service buildkitd, stevedore — both depend on it). Verify (reference host state):

(Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\containerd').ImagePath   # contains --log-level debug --log-file ...
(Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\buildkitd').ImagePath    # contains --debug

C2. Disable buildkitd’s per-step log limit (REQUIRED for compile stages)#

Without this, heavy steps deadlock silently at the 2 MiB clip (Windows Build Image § Getting it going, step 4). Since 2026-08-10 this is ENFORCED: build-buildkit.ps1’s Assert-BuildkitdStepLogEnv preflight refuses to launch while the value is missing (a Stevedore repair once wiped it silently); -SkipStepLogGate is the documented one-launch escape when no admin is at hand:

Set-ItemProperty -Path HKLM:\SYSTEM\CurrentControlSet\Services\buildkitd `
  -Name Environment -Type MultiString `
  -Value @('BUILDKIT_STEP_LOG_MAX_SIZE=-1','BUILDKIT_STEP_LOG_MAX_SPEED=-1')

(The restart comes with C3.) Verify after C3:

(Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\buildkitd').Environment

C3. Deploy the GC policy + build-history cap (REQUIRED — not optional tuning)#

Skipping this cost this project twice: buildkitd’s computed defaults evicted the multi-hour VS Build Tools layer between two runs, and unlimited build history pinned a 414 GB store at Reclaimable: 0B. The repo policy (windows/buildkitd.toml: three GC tiers, reservedSpace = 200GB, plus the [history] maxAge/maxEntries cap) is deployed by:

pwsh -File windows\scripts\apply-buildkitd-gcpolicy.ps1    # admin; refuses while a build runs

Sizing on a different disk: the toml’s literals assume a ~930 GB C:. Reproduce the INVARIANTS, not the numbers — reservedSpace must exceed the fresh chain spine (~120–150 GB; rule of thumb 20–25 % of the disk, floor 150 GB), maxUsedSpace ≈ 1.5× reservedSpace, minFreeSpace ≥ 25–30 GB always, [history] unchanged everywhere. The sizing rationale lives as a comment block in windows/buildkitd.toml itself. Re-run the apply script after every repo-side toml change — deploy is a copy, nothing syncs automatically (this host ran a stale copy for hours after the [history] section landed).

Full story: Windows Build Image § BuildKit/containerd lane, “Store GC” bullet. Verify:

& "$env:ProgramFiles\Stevedore\bin\buildctl.exe" debug workers -v | Select-String 'reservedSpace|maxUsedSpace|minFreeSpace'
# must show reservedSpace=200GB tiers, NOT the computed defaults (maxUsedSpace 100GB / minFreeSpace 187GB)

C4. Windows Defender exclusions — LOAD-BEARING, not hygiene#

These exclusions are load-bearing: without them the realtime scanner races container churn and finalize/export operations flake constantly (the hcs-temp sharing-violation family). They also tame — but do NOT cure — the ExportLayer 0x3 heavy-churn finalize defect (TVM-class finalizes became reliable with them; OpenCV-class still trips it, which is why the warm/materialize pattern stays — full story: windows-builds.md § roadmap). Skipping this step on a new machine makes builds flaky across the board. The full set (the § Getting it going step-3 list plus the process exclusions added 2026-08-05 after the hcs-temp sharing-violation flake family):

Add-MpPreference -ExclusionPath "C:\ProgramData\containerd"
Add-MpPreference -ExclusionPath "C:\ProgramData\buildkitd"
Add-MpPreference -ExclusionPath "C:\ProgramData\nerdctl"
Add-MpPreference -ExclusionPath "C:\temp"
Add-MpPreference -ExclusionProcess "buildkitd.exe"
Add-MpPreference -ExclusionProcess "containerd.exe"
Add-MpPreference -ExclusionProcess "dockerd.exe"

Also exclude your repo checkout directory (antivirus file locks during FetchContent/cargo are a documented failure family — see the note at the top of Windows Build Image). Verify (admin — non-admin reads print “N/A”):

Get-MpPreference | Select-Object -Expand ExclusionPath
Get-MpPreference | Select-Object -Expand ExclusionProcess

C5. sccache / dufs WebDAV server (REQUIRED for the media stages)#

This server is load-bearing twice: it is the compile cache (sccache WebDAV backend) AND the transport for the warm/materialize handoff tars (the bkhandoff/ subdir) that neutralize the ExportLayer 0x3 snapshotter defect — without it the BK media solves fail fast. setup-new-host.ps1 automates all of it (scoop install if missing, cache dir, start, ONLOGON task, machine-level endpoint env with the host’s LAN IP — never localhost). By hand (non-admin, except the machine-env line):

scoop install dufs
mkdir C:\sccache-cache
dufs C:\sccache-cache -A -p 5000                 # keep it running

# endpoint = the host's LAN IP (ipconfig), NEVER localhost — containers must reach it:
[Environment]::SetEnvironmentVariable('SCCACHE_WEBDAV_ENDPOINT','http://<host-LAN-IP>:5000','Machine')  # admin; or pass -SccacheEndpoint per run

dufs does NOT survive reboots. Either make it logon-persistent once —

schtasks /Create /TN dufs-sccache /TR "\"%USERPROFILE%\scoop\shims\dufs.exe\" C:\sccache-cache -A -p 5000" /SC ONLOGON

— or accept restarting it manually after every reboot (the reference host does the latter; then the Phase-D check below is what saves you). Verify:

(Invoke-WebRequest http://<host-LAN-IP>:5000 -Method Head -UseBasicParsing).StatusCode   # 200
[Environment]::GetEnvironmentVariable('SCCACHE_WEBDAV_ENDPOINT','Machine')               # set (new shells inherit)

Phase D — Per-boot / per-run checks#

Run these before every chain launch (30 seconds; each one has cost a real run):

dufs up + endpoint reachable? (Invoke-WebRequest $env:SCCACHE_WEBDAV_ENDPOINT -Method Head -UseBasicParsing).StatusCode → 200. A reboot kills a manually-started dufs (cost a run on 2026-08-04).
Services running? Get-Service stevedore, containerd, buildkitd → all Running.
Disk headroom ≥ 40 GB free — now gated automatically. Both drivers refuse to start below the floor (Assert-DiskHeadroom; override with -SkipHostChecks, raise/lower with -MinFreeGb), and the BuildKit lane additionally verifies the patched runhcs shim is still installed (Assert-ShimPatch — a Stevedore update silently restores the stock binary, and the first heavy media finalize then dies with ExportLayer 0x3 hours into the run). Both were manual checks here until 2026-08-07; on 2026-08-06 a chain ran 2.5 h and died of a disk shortage disguised as a missing ninja. The disk gate checks every drive the build uses — C: (the layer stores) plus the repo checkout’s drive, which on a VHDX-backed checkout has its own exhaustion mode that a C:-only check cannot see (§ VHDX-backed checkouts in windows-builds.md). The shim gate compares the live binary’s SHA256 against the hash deploy-shim-patch.ps1 recorded when it installed the patch (state file: C:\ProgramData\kataglyphis\shim-patch.json); a Stevedore update overwriting the binary is then an unambiguous hard failure. Hosts that have not yet re-run the deploy script fall back to the older file-size heuristic and get a warning telling them so — run deploy-shim-patch.ps1 once to record the hash. -ReportOnly prints the recorded hash and whether it still matches. (Get-PSDrive C).Free / 1GB — below ~25 GB free, hcsshim gets “weird” before an honest disk-full error (ExportLayer 0x3/0x70, spawn flakes). Reclaim levers, non-admin first: buildctl prune --free-storage <MB> (see the target trap below), then buildctl prune-histories, buildctl prune --free-storage <MB>, docker image prune -f; the full playbook is Windows Build Image § Store GC. Two traps that make the levers look broken: --free-storage is a minimum-free target, so it deletes nothing once the disk is already above it (ask for more free space than the disk has to drain everything unpinned), and a superseded lineage of stage tags can pin whole duplicate copies of the base spine — the biggest single reclaim measured on this host (266 GB). If the checkout or the store lives on a dynamically-expanding VHDX, the store levers cannot see the biggest pool: dead blocks in the VHDX itself (270 GB physical for 16 GB of data on the reference host). Check it — the report costs nothing and stops nothing:
```
pwsh -File windows\scripts\compact-host-vhdx.ps1 -VhdxPath <your.vhdx> -ReportOnly   # admin
```
Without -ReportOnly it stops the build services, compacts and restores the disk — admin, and never while a build solves. Read the ReFS caveat in § Store GC first: on ReFS guests compaction reclaims ~nothing, and the reclaim that does work is rebuild-host-vhdx.ps1, which rebuilds the disk around its live data. Run its -CopyOnly phase whenever you like — it touches nothing live — but the swap detaches the volume, so nothing may hold a handle on it: no shell sitting in the checkout, no editor, no agent session. Losing that volume mid-session is how a working session died on 2026-08-06.
CNI subnet drift — if dockerd/the host restarted since the last run, expect it; build-buildkit.ps1’s preflight fail-fasts with the exact fix (see Phase A5).
Debug log size — C:\ProgramData\containerd\containerd-debug.log grows unbounded; if it is huge, truncate (admin: Clear-Content C:\ProgramData\containerd\containerd-debug.log), never disable the flags.

Phase E — First build + verification#

E1. Optional inputs, expectations#

TensorRT (GPU lane, optional): drop the NVIDIA EULA zip into windows\downloads\ if you have one; without it the build skips TensorRT gracefully — the zip-less state is the normal state of the reference host’s GPU lane (AGENTS.md § TensorRT Setup). Do not wait on it.
Cost: cold full chain ≈ 5–6 h (≈ 2.5 h in the media fan-out); hot rebuild of the whole BK chain ≈ 44 min. Parallelism is memory-bound — ~35–45 % average CPU during compiles is the expected signature, not a fault (Windows Build Image § Maximum resource envelope; 32 CPU / 39 GB is the verified max on the 64 GB reference host).
Logs: per-stage under out\windows-build-logs\.
Warm/materialize is normal: heavy media libraries build in “warm” solves (no exported image) and materialize in a second calm solve — that two-step pattern in the log is the designed workaround for the host snapshotter defect, not a failure.
Transient retries are automatic: the driver retries the known flake families (ActivateLayer 0x20, hcs-temp finalize/export). On any OTHER weird hcsshim failure: check free disk first (AGENTS.md § Common Failure Modes).

E2. Launch (non-admin)#

# endpoint via machine env (C5) or explicitly:
.\windows\build-buildkit.ps1 -Gpu                                    # full chain
.\windows\build-buildkit.ps1 -Gpu -SccacheEndpoint http://<LAN-IP>:5000
.\windows\build-buildkit.ps1 -Gpu -FinalTar out\bk-winamd64.tar      # + docker-loadable tar

The driver’s preflight verifies buildkitd reachability and CNI subnet before solving. Single-stage iteration: -Stages toolchain etc.

E3. Verify the images exist (admin — containerd’s pipe)#

& "$env:ProgramFiles\Stevedore\bin\nerdctl.exe" --namespace buildkit images
# expect docker.io/local/kataglyphis:bk-winamd64 and the bk-windows-* stage tags

E4. Smoke test — on the GPU lane ALWAYS pass -ExpectGpu#

Without -ExpectGpu, a broken CUDA env is silently SKIPPED instead of failed (Windows Build Image § Smoke Testing). Two routes:

# (a) via docker after a -FinalTar export (loads as local/kataglyphis:winamd64):
& "$env:ProgramFiles\Stevedore\bin\docker.exe" load -i out\bk-winamd64.tar
& "$env:ProgramFiles\Stevedore\bin\docker.exe" run --memory 48g --rm --isolation process `
  local/kataglyphis:winamd64 pwsh -File C:\temp\scripts\smoke-test-container.ps1 -ExpectGpu

# (b) directly from the containerd store (admin shell):
& "$env:ProgramFiles\Stevedore\bin\nerdctl.exe" --namespace buildkit run --rm `
  docker.io/local/kataglyphis:bk-winamd64 pwsh -File C:\temp\scripts\smoke-test-container.ps1 -ExpectGpu

Expected: the § Smoke Testing baseline (167 passed / 0 failed / 1 skipped on the GPU lane; the single skip is GPU device passthrough, blocked by host/base OS-build skew).

E5. Publish (optional, non-admin)#

& "$env:ProgramFiles\Stevedore\bin\docker.exe" login ghcr.io      # once, same shell
.\windows\build-buildkit.ps1 -Gpu -PushRef ghcr.io/kataglyphis/kataglyphis_beschleuniger:winamd64

Classic-lane fallback#

If the BK lane is unavailable, windows\build.ps1 -Gpu (docker-classic, Hyper-V run+commit) works with only Phases A, B and C4/C5 — see Windows Build Image § Build Commands and § Stevedore Setup Fixes.

Fresh Windows Host Bring-Up (BK lane + repo gates)

Contents

Fresh Windows Host Bring-Up (BK lane + repo gates)#

Phase A — One-time host provisioning [admin]#

A1. Install Stevedore, reboot#

A2. docker-users group membership#

A3. Services running, reboot-safe#

A4. Conditional dockerd fixes (only on the matching symptom)#

A5. CNI nat conf (required — RUN steps have NO network without it)#

Phase B — Repo checkout + gate tooling [non-admin]#

B1. Git for Windows + clone#

B2. PowerShell 7 (pwsh)#

B3. PSScriptAnalyzer + Pester (the PowerShell gates)#

B4. Python for preflight/sync — dodge the Windows Store stub#

B5. shellcheck / hadolint / actionlint — nothing to install#

B6. Run all three gates#

Phase C — Build-service tuning [admin]#

C1. Permanent debug flags on containerd + buildkitd (owner policy)#

C2. Disable buildkitd’s per-step log limit (REQUIRED for compile stages)#

C3. Deploy the GC policy + build-history cap (REQUIRED — not optional tuning)#

C4. Windows Defender exclusions — LOAD-BEARING, not hygiene#

C5. sccache / dufs WebDAV server (REQUIRED for the media stages)#

Phase D — Per-boot / per-run checks#

Phase E — First build + verification#

E1. Optional inputs, expectations#

E2. Launch (non-admin)#

E3. Verify the images exist (admin — containerd’s pipe)#

E4. Smoke test — on the GPU lane ALWAYS pass -ExpectGpu#

E5. Publish (optional, non-admin)#

Classic-lane fallback#