# Linux Host Setup (GPU, drivers, runtime, boot)

Goal: take a **fresh Ubuntu/Debian host** to a state where it can build and run
this repo's Linux images with GPU acceleration. This is the Linux counterpart to
[Fresh Windows Host Bring-Up](windows-host-setup.md).

Scope is the **host**, not the images. What goes inside a container is owned by
[Linux Build Basics](linux-build-basics.md) and
[Linux Accelerator Images](linux-accelerator-images.md); running Linux
containers from a Windows host is
[Rancher Desktop](rancher-desktop-linux-containers.md).

Every phase ends with a verify command. Run them — a driver that installed
without error but did not load is the common failure here, and it looks
identical to success until the first build.

---

## Phase A — GPU drivers

### A1. NVIDIA driver

```bash
sudo apt update
sudo apt upgrade -y
sudo apt install -y build-essential linux-headers-$(uname -r)

sudo ubuntu-drivers devices        # list what this GPU supports
sudo ubuntu-drivers install        # let the tool pick
# ...or pin a specific branch:
# sudo apt install -y nvidia-driver-590
```

Reboot afterwards. The kernel module is only loaded on a fresh boot.

Verify:

```bash
nvidia-smi
```

### A2. CUDA toolkit

`ubuntu-drivers` does not install the toolkit — `nvcc` comes separately.

```bash
sudo add-apt-repository -y multiverse
sudo apt update
sudo apt install -y cuda-toolkit
```

CUDA does not put itself on `PATH`. Add a profile drop-in so every login shell
and every build script sees it:

```bash
sudo tee /etc/profile.d/cuda-path.sh >/dev/null <<'PROFILE'
export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
PROFILE
```

Verify (in a **new** shell):

```bash
nvcc --version
```

### A3. Purge and recover a broken NVIDIA install

A half-upgraded driver stack is the usual cause of `nvidia-smi` failing after an
`apt upgrade`. Do not fight it incrementally — purge and reinstall.

First see what was pulled in manually, so you know what to put back:

```bash
sudo apt-mark showmanual | grep -E 'nvidia|cuda'
```

Then remove everything, including the repo definitions:

```bash
sudo apt-get remove --purge '^nvidia-.*' 'libnvidia-*' 'linux-modules-nvidia-*' -y
sudo apt purge -y 'cuda-*' 'nsight-*'
sudo apt autoremove --purge -y
sudo apt autoclean

sudo rm -f /etc/apt/sources.list.d/cuda*.list
sudo rm -f /etc/apt/sources.list.d/nvidia*.list
sudo rm -f /usr/share/keyrings/cuda-archive-keyring.gpg
sudo dpkg -P cuda-keyring 2>/dev/null || true

sudo dpkg --configure -a
sudo apt --fix-broken install -y
sudo apt update
sudo reboot
```

Then start again at [A1](#a1-nvidia-driver).

### A4. AMD

AMD GPUs need no proprietary driver package for the ROCm lane — the kernel
driver is in-tree. What is sometimes missing is firmware:

```bash
sudo apt install -y linux-firmware
```

Verify:

```bash
rocm-smi
```

---

## The prerequisites list AGENTS.md carried

Moved out of `AGENTS.md` on 2026-09-15 (owner decision D10), unedited except for this heading and the relative links. The RULES stayed there; this is the reference behind them.

- **nerdctl** with BuildKit backend
- **QEMU/binfmt** — and `--install-service` is NOT enough on its own. The unit
  it writes used `After=`/`Wants=` only, which order STARTUP and do not
  propagate a restart; combined with `Type=oneshot` + `RemainAfterExit=yes`,
  systemd holds the unit permanently satisfied while its effect — a
  registration inside the rootlesskit namespace — dies with every containerd
  restart. Measured 2026-08-27 on the dev host: unit last ran 2026-08-09,
  containerd restarted 2026-08-26, and the runtime stage then failed BOTH
  foreign arches with an empty BuildKit error. The template now sets
  `PartOf=containerd.service` (re-runs the unit on containerd restart) **plus**
  `wait_for_namespace()` + `Restart=on-failure` (afefdfc) to win the cold-boot
  race: `After=` orders only the unit start, so on a fresh boot the binfmt unit
  fired before containerd-rootless had unshared and written its `child_pid`,
  dying with `cat: …/child_pid: No such file`. `wait_for_namespace()` polls for
  the pid file and a joinable namespace before registering; `Restart=on-failure`
  is belt-and-braces. A daemon restart and a cold boot no longer silently strip
  foreign-arch emulation. On this rootless host the privileged
  `tonistiigi/binfmt --install` container does NOT work (wrong namespace); use:
  ```bash
  linux/scripts/setup-rootless-binfmt.sh --arches arm64,riscv64 --install-service
  ```
  `--install-service` installs a systemd --user unit so re-registration is
  automatic. Without registration, foreign-arch execs fail with `exec format
  error` — including wrong-arch NATIVE tool sub-builds inside "no-emulation"
  cross stages (the IREE tblgen failure mode).
- **Registry access** (GHCR) for pushing intermediate and final images
- **Disk space**: far more than the ~50GB+ this line used to say. The chain's disk
  preflight (`_chain_disk_preflight` in `build-cross-chain.sh`) budgets 60 GB per
  arch for a run from `base` (40 GB from `media` on), plus ~120 GB for the runtime
  lane (`CROSS_RUNTIME_LANE_GB`), and warns when free space is below the sum; the
  runtime lane refuses to start without its share
- **Python 3** for digest resolution (`registry-digest.py`)

## Phase B — Container runtime host config

### B1. Docker without sudo

```bash
sudo usermod -aG docker $USER
```

Log out and back in — group membership is only picked up on a new session.

### B2. Make the NVIDIA runtime the default

Without this, every GPU container needs an explicit `--runtime nvidia`, and
Compose files that omit it silently run on CPU.

```bash
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json >/dev/null <<'DAEMON'
{
  "default-runtime": "nvidia",
  "runtimes": {
    "nvidia": {
      "path": "/usr/bin/nvidia-container-runtime",
      "runtimeArgs": []
    }
  }
}
DAEMON

sudo systemctl daemon-reload
sudo systemctl restart docker
```

Verify:

```bash
docker info | grep -i 'default runtime'
```

### B2b. A GPU container on a Jetson, with rootless nerdctl

Measured on a Jetson AGX Orin (L4T R39, `nvidia-container-toolkit` 1.20.1,
2026-09-22): the arm64 GPU wrapper ran PyTorch, the ONNX Runtime CUDA EP,
OpenCV CUDA and an `nvcc -arch=sm_87` kernel on the GPU with this call:

```bash
nerdctl --cdi-spec-dirs "$HOME/.config/cdi" run --rm \
  --runtime "$HOME/.local/bin/crun" \
  --annotation run.oci.keep_original_groups=1 \
  --device nvidia.com/gpu=all <image> \
  python -c "import torch; print(torch.cuda.get_device_name(0))"
```

Each flag answers one failure, in the order they appear:

| Symptom | Cause | Fix |
|---|---|---|
| `unresolvable CDI devices nvidia.com/gpu=all` | The toolkit writes its spec to `/var/run/cdi`, which rootlesskit's own `/run` hides from the daemon. | Copy it: `mkdir -p ~/.config/cdi && cp /var/run/cdi/nvidia.yaml ~/.config/cdi/`, and pass `--cdi-spec-dirs`. Copy again after a driver update, which regenerates the spec. |
| `NvRmMemInitNvmap failed: error Permission denied`, then `No CUDA GPUs are available` | `/dev/nvmap` and `/dev/nvhost-*` belong to the host group `video`. A rootless container process cannot carry a host group, and `runc` cannot keep it. | `crun` with the `run.oci.keep_original_groups=1` annotation keeps the caller's groups. |
| `OCI runtime create failed: unknown version specified` | Ubuntu's `crun` (1.14) predates the OCI spec containerd 2.x writes. | Install the static release binary into `~/.local/bin` and check its SHA256 against the release's `digest` field (1.29.1 worked). |

[`linux/jetson-webcam/`](../linux/jetson-webcam/README.md) wraps this call
around a USB-camera detection demo.

The official PyTorch `cu130` wheels warn that they do not support
compute capability 8.7 (Orin). A matmul and the elementwise kernels it needed
still ran correctly, but a kernel that exists only as SASS for other
architectures will fail. The libraries this repo builds (ORT, OpenCV, TVM)
carry native `sm_87` code.

### B2c. A GPU container on an x86 desktop, with rootless nerdctl

Set up 2026-10-01 on the build host: an RTX 2080, driver 595.58.03 (`nvidia-driver-open`,
DKMS), Secure Boot on, `nvidia-container-toolkit` 1.20.1, nerdctl 2.4.0. The DeepStream GPU
gate ran this way ([`linux-accelerator-images.md` § The GPU run](linux-accelerator-images.md#the-gpu-run-2026-10-01)).

1. **Secure Boot: enroll the DKMS key before anything else.** Until then the module never
   loads: the kernel logs `Loading of module with unavailable key is rejected` at every boot
   and `nvidia-smi` finds no driver. As root, `mokutil --import
   /var/lib/shim-signed/mok/MOK.der`, reboot, and enroll it in MOK Manager.
2. **The NVIDIA Container Toolkit**, from NVIDIA's apt repository; Ubuntu does not package it.
3. **Generate the CDI spec**: `sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml`.
4. **Hand it to rootless nerdctl.** `/etc/cdi` is not visible to rootless nerdctl, so copy the
   spec into the user's own directory and name that directory in nerdctl's config:

   ```bash
   mkdir -p ~/.config/cdi && cp /etc/cdi/nvidia.yaml ~/.config/cdi/
   # ~/.config/nerdctl/nerdctl.toml
   cdi_spec_dirs = ["/home/<user>/.config/cdi", "/etc/cdi", "/var/run/cdi"]
   ```

5. **Regenerate and copy again after every driver update.** The spec lists the driver's
   library files by version, so an old spec mounts files that no longer exist.

Verify:

```bash
nerdctl run --rm --device nvidia.com/gpu=all \
  ghcr.io/kataglyphis/kataglyphis_beschleuniger:latest-amd64 nvidia-smi
```

No `--runtime` or group annotation is needed here, unlike on the Jetson (B2b): the desktop
device nodes are world-accessible, and the image's runtime user (uid 1001) ran a DeepStream
pipeline on the GPU as well.

### B3. Cap container log growth

A long build or a chatty service will fill the disk with JSON logs. Merge these
keys into the same `daemon.json`:

```json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}
```

Restart Docker afterwards. Disk exhaustion mid-build surfaces as
`no space left on device` deep inside a compile step — see
[Build resource monitoring](build-resource-monitoring.md).

### B3b. Install or upgrade `nerdctl-full`

`nerdctl-full` ships nerdctl together with the whole stack it drives —
containerd, BuildKit (`buildkitd` + `buildctl`), runc, the CNI plugins, the
snapshotters and the rootless helpers. They are version-matched, so on this
host **upgrading buildkitd means upgrading the bundle**; there is no separate
buildkit package to bump.

```bash
bash linux/host-config/install-nerdctl-full.sh                    # dry run: version delta + plan
NERDCTL_INSTALL_CONFIRM=1 bash linux/host-config/install-nerdctl-full.sh
bash linux/host-config/install-nerdctl-full.sh --rollback         # restore the backup
```

Knobs: `NERDCTL_VERSION=x.y.z` pins a release (default: latest),
`NERDCTL_PREFIX` (default `/usr/local`, or `$HOME/.local` in rootless mode),
`NERDCTL_ROOTLESS=1|0` forces the prefix mode instead of auto-detecting it — see
[B3c](#b3c-install-rootless-into-homelocal-no-sudo), `NERDCTL_BACKUP_DIR`,
`NERDCTL_FORCE=1` (re-install the version already present),
`NERDCTL_SKIP_CACHE_CENSUS=1` (proceed without a cache baseline when buildkitd
is down).

**This host runs rootless AND rootful side by side, and that changes the
procedure.** The `systemd --user` units are the rootless stack the chain builds
with; but a rootful `containerd.service` *and* `buildkit.service` also run, and
they execute the same `/usr/local/bin` binaries plus unit files the bundle
ships in `/usr/local/lib/systemd/system`. Extracting the bundle replaces all of
that underneath them. Measured on this host: GNU tar 1.35 and `cp -a` both
unlink-and-recreate rather than failing with `ETXTBSY`, so there is **no error
at all** — the root daemons simply keep executing the now-deleted old inode,
and because both units are `Restart=always` the real version jump then happens
unattended at some later moment. The script therefore refuses unless you choose:

```bash
NERDCTL_INCLUDE_ROOTFUL=1 ...   # stop, upgrade and restart them too — no skew
NERDCTL_IGNORE_ROOTFUL=1  ...   # rootless only; accept the documented skew
```

A dry run always shows the plan and the choice — only the actual install is
blocked.

What the script guarantees, and why each guard exists:

- **It refuses while a build runs.** Extracting over live binaries mid-chain
  kills the run; chains here regularly last 10+ hours.
- **It verifies the release SHA256** against the published `SHA256SUMS` before
  touching anything, and refuses if the checksum is missing.
- **It backs up exactly the `bin/` binaries the bundle ships, plus the
  `/usr/local/lib/systemd/system` unit files it rewrites**, so `--rollback`
  puts nerdctl, buildkitd/buildctl, containerd and runc back *and* restores the
  rootful unit definitions — it stops the root services first, copies the units
  back and runs a system `daemon-reload`, so a rollback does not strand them on
  the new definitions. Know its scope: the bundle also ships `libexec/` (CNI)
  and `share/`, and those stay at the newly installed version. That is harmless
  for the build path, but a rollback is *not* a full downgrade — for that,
  re-run with `NERDCTL_VERSION=<previous>`.
- **It counts BuildKit cache-mount records before and after.** Compile caches
  (ccache/sccache/uv/cargo/cerbero) live in `~/.local/share/buildkit`, not in
  `/usr/local`, so an upgrade must not change that number — see
  [`prune-safe.sh`](../linux/host-config/prune-safe.sh) for why those records
  are the thing worth protecting.
- **It proves the stack came back up** (`nerdctl images`, `buildctl du`,
  service active) instead of assuming, and tells you the rollback command if
  it did not.

**Why this was done, and what it bought** — BuildKit `v0.31.0` introduced a
daemon crash, *"concurrent map iteration and map write"*
([moby/buildkit#6915][bk6915]), that reproduces under **concurrent** builds,
which is exactly how this chain drives it with three arch lanes at once. The
fix ships in BuildKit `v0.31.2`, bundled by nerdctl-full `2.3.5`. Two caveats,
so nobody upgrades expecting the wrong thing: it does **not** cure
[BKD1](failure-modes.md) — the buildkitd session rot (export hangs, `no active
session`, lost layer blobs) has no upstream fix and is still cured by stopping
the chain and restarting the service — and the parallel-build cache-miss fix
([moby/buildkit#6954][bk6954]) landed in `v0.32.0`, which no nerdctl-full
release ships yet.

**Performed on this host, 2026-08-26** (the reference run, with what was
actually proven afterwards rather than assumed):

| | before | after |
|---|---|---|
| `nerdctl` on disk | 2.3.4 | **2.3.5** |
| `buildctl` on disk | v0.31.1 | **v0.31.2** |
| buildkitd **daemon** reports | v0.31.1 | **v0.31.2** |
| containerd **Server Version** | v2.3.2 | **v2.3.3** |
| cache-mount records | 51 | **51** (unchanged) |
| buildkitd workers | — | 1 |

Run as `NERDCTL_INCLUDE_ROOTFUL=1 NERDCTL_INSTALL_CONFIRM=1`, so the rootful
pair was stopped and restarted with the rootless one. Verified afterwards: the
rootful daemons came back on NEW pids (1982/1983 → 30591/30592, i.e. they are
not stranded on the deleted old inodes), no daemon's `/proc/<pid>/exe` reads
`(deleted)`, and both root units report `NeedDaemonReload=no` — the system
`daemon-reload` took, so nothing flips over unattended later.

Two gotchas worth repeating, both hit during that run:

- The env vars must be on **one line** with the `bash` call. Split across two
  lines they are set in the parent shell, never exported, and the script runs
  as a plain dry run that changes nothing — which is exactly what happened on
  the first attempt.
- Installing into `/usr/local` needs `sudo` (that tree is root-owned), so that
  mode cannot be driven from a non-interactive session. This used to read as an
  unconditional rule — it is not. A `$HOME/.local` install needs no sudo at all
  and is the way to update a host whose sudo prompts for a password; see
  [B3c](#b3c-install-rootless-into-homelocal-no-sudo).

[bk6915]: https://github.com/moby/buildkit/issues/6915
[bk6954]: https://github.com/moby/buildkit/issues/6954

Afterwards run `linux/host-config/verify-host-config.sh` and
`linux/scripts/preflight.sh` before starting the next chain — the daemon
restart is also the moment the staged `buildkitd.toml` gcpolicy takes effect.

### B3c. Install rootless, into $HOME/.local (no sudo)

On a host whose stack is rootless-only, install the bundle **into the user's own
prefix**. Nothing about this needs root, so it is the only mode that can update
a machine where `sudo` prompts for a password. The distinction B3b's last gotcha
used to blur: rootless describes both how the daemons **run** and where the
bundle **lives**.

```bash
NERDCTL_INSTALL_CONFIRM=1 bash linux/host-config/install-nerdctl-full.sh
```

The mode is auto-detected — no knob needed. The script reads the `ExecStart` of
the live `systemd --user` units and installs into whatever prefix they already
name, because those units are the only authority on what the host actually
runs. `NERDCTL_ROOTLESS=1` forces it on a host being converted for the first
time; `NERDCTL_ROOTLESS=0` forces the `/usr/local` path back.

**The units are rewritten for you, and that is the load-bearing half.**
Extracting the bundle is not a prefix change on its own: the units keep the
absolute `ExecStart` they were generated with, so without the rewrite the
daemons keep launching the old prefix and the install moves nothing. The script
repoints `~/.config/systemd/user/{containerd,buildkit}.service` and prepends
`${PREFIX}/bin` to their `Environment=PATH` (`containerd-rootless.sh` resolves
`rootlesskit`, `slirp4netns` and `containerd` through `PATH`), then reloads and
restarts. The pre-image goes into `${NERDCTL_BACKUP_DIR}/systemd-user`, so
`--rollback` restores the units as well as the binaries.

**The CNI plugins are the half nobody verifies.** Rootless nerdctl resolves them
under **its own** default — `$HOME/.local/libexec/cni` — not under `/usr/local`.
A `/usr/local` install therefore leaves a rootless host with *no* plugins at
all, and the symptom is a container-networking error that never mentions the
install. Same class as [B6](#b6-ufw-silently-breaks-container-networking): the
build looks correctly configured and the networking simply is not there. The
script now counts them and fails the run at zero.

**Performed on this host, 2026-09-08** (aarch64, Snapdragon X, WSL2):

| | before | after |
|---|---|---|
| prefix | `/usr/local` | `$HOME/.local` |
| nerdctl / containerd / buildkit | 2.3.5 / v2.3.3 / v0.31.2 | unchanged |
| CNI plugins where nerdctl looks | **0** | **18** |
| sudo needed | yes | **no** |

The binaries were **byte-identical** — sha256 matched across `nerdctl`,
`containerd`, `buildkitd`, `rootlesskit`, `runc` and `containerd-rootless.sh`,
because `/usr/local` already held the same bundle. This was a relocation, not a
version change, which is why the script proves the daemons now *execute* from
`${PREFIX}/bin` instead of asserting that their reported version moved.

**A drop-in `ExecStart` beats the unit file's.**
`linux/host-config/buildkit.service-override.conf` therefore carries
`@NERDCTL_PREFIX@` rather than a literal path; `apply-host-config.sh` and
`verify-host-config.sh` substitute the live prefix before installing or
diffing. Without that, applying host config after a rootless install silently
reverted `buildkitd` to `/usr/local` — invisible until a build failed.

`NERDCTL_INCLUDE_ROOTFUL` / `NERDCTL_IGNORE_ROOTFUL` are moot here: a
`$HOME/.local` extraction cannot reach root daemons, so the script skips the
rootful-coexistence fork entirely in this mode. Afterwards, run the same
`verify-host-config.sh` + `preflight.sh` hand-off B3b ends with.

### B4. Verify a `nerdctl-full` install

The `nerdctl-full` tarball bundles containerd, BuildKit, runc, snapshotters and
the rootless helpers. A partial extraction leaves some of them off `PATH`, and
the symptom is a confusing failure much later — a build that cannot find
`buildctl`, or rootless mode silently unavailable. Check all of them at once:

```bash
#!/usr/bin/env bash
binaries=(
    nerdctl containerd runc buildkitd buildctl
    containerd-stargz-grpc ctd-decoder
    slirp4netns bypass4netns
    fuse-overlayfs containerd-fuse-overlayfs-grpc
    tini buildg rootlesskit gomodjail
)

for cmd in "${binaries[@]}"; do
    if command -v "$cmd" >/dev/null 2>&1; then
        printf '\nOK   %s: ' "$cmd"
        "$cmd" --version 2>/dev/null || "$cmd" version 2>/dev/null \
            || echo "(no standard version flag)"
    else
        printf '\nMISS %s (not on PATH)\n' "$cmd"
    fi
done
```

`command -v` alone passes on a **stale copy in the other prefix** while the
daemons run from the intended one, and the list above contains no CNI plugin at
all — the exact gap that stayed invisible until 2026-09-08. Assert both:

```bash
prefix="$(sed -n 's/^ExecStart="\{0,1\}\([^"[:space:]]*\)\/bin\/.*/\1/p' \
           ~/.config/systemd/user/containerd.service | head -1)"
echo "units name: ${prefix}"
echo "PATH gives: $(command -v nerdctl)"        # must be ${prefix}/bin/nerdctl
cni="$(nerdctl info --format '{{.CNIPath}}')"
echo "CNI: $(find "${cni}" -maxdepth 1 -type f ! -name 'LICENSE' ! -name 'README.md' | wc -l) plugins in ${cni}"
for p in $(pgrep -f 'containerd$|buildkitd'); do readlink -f "/proc/$p/exe"; done
```

Expected on a healthy rootless host: `PATH` and the units agree, 18 CNI
plugins, and every daemon `exe` under `${prefix}/bin`.

### B5. TLS handshake failures inside containers

A container that cannot resolve names — `TLS handshake timeout` on an image
pull, or `apt` failing to reach a mirror — is usually the host handing it an
unreachable resolver.

```bash
# temporary: overwrite the resolver and restart the runtime
echo -e "nameserver 1.1.1.1\nnameserver 8.8.8.8" | sudo tee /etc/resolv.conf
sudo systemctl restart containerd
```

On a systemd-resolved host, `/etc/resolv.conf` is a managed symlink and the edit
above will be reverted. Make it stick in `/etc/systemd/resolved.conf` instead:

```ini
[Resolve]
DNS=8.8.8.8 8.8.4.4
FallbackDNS=1.1.1.1
```

```bash
sudo systemctl restart systemd-resolved
```

### B6. UFW silently breaks container networking

If the firewall is enabled, containers can lose outbound networking entirely —
image pulls hang, `apt` inside a build reaches nothing. UFW defaults to dropping
forwarded traffic, which is exactly what a container bridge relies on.

```bash
sudo nano /etc/default/ufw
# DEFAULT_FORWARD_POLICY="ACCEPT"
sudo ufw reload
```

This is the Linux counterpart to the Windows lane's CNI `.conf` requirement —
in both cases the build looks correctly configured and the RUN steps simply have
no network. Check this before debugging a mirror or a DNS resolver.

---

### B7. Reclaiming disk without losing the compile caches

Moved here from `AGENTS.md` on 2026-09-15: the rule lives there, the
measurements live here.

**`linux/host-config/prune-safe.sh`, never `nerdctl builder prune -f`.** The
`-f` prune deletes `type==exec.cachemount` records — ccache, sccache, uv, cargo,
the LLVM source tree — together with the cheap-to-regenerate layer cache.
Measured on this host: **207 GB of cachemount against 4.9 GB of layer cache**,
so the destructive half is 97% of what a `-f` prune takes, and one run paid
about **1.5–2 h of cold LLVM rebuilds** for a few GB.

`prune-safe.sh` prunes `type==regular` only, through `buildctl --filter`
(nerdctl has no such flag), takes `PRUNE_KEEP_GB` and `DRY_RUN`, and proves
cachemount survival by counting the records before and after. `PRUNE_KEEP_GB`
is GB of *layer cache*: the script adds the cache mounts and local sources to
the `--keep-storage` it passes, because that flag bounds the whole store
([`build-cache-tiers.md` § 3.2.1](build-cache-tiers.md#321-the-buildkit-store-fallback-disk1)).

**A cache id with more than one record** is listed after the cache mounts. With
`sharing=shared`, BuildKit looks a cache id up without waiting: when the record
it would reuse is locked, for instance mid-release by a parallel stage, it
creates a second one. From then on it reuses the lowest record id it can lock
(its bolt index order), and the other is touched only in the next such race.
On 2026-10-01 six ids carried 52 GB of surplus that way (`/uv-cache-riscv64`
alone 25.9 GB). The Dockerfiles' mount options were uniform; the race is
BuildKit's. `PRUNE_DUP_CACHEMOUNTS=1` removes the surplus records, and refuses
while any build holds a record. Through wave 4:
**12 invocations, ~1 TB reclaimed, 0 cachemount losses.**

**Mid-run, use `PRUNE_KEEP_GB>=100`.** Smaller budgets evict the in-flight
lanes' fresh layers and buy recompile churn; the kata-buildcache media slugs,
rewritten every round, are the better lever when the store runs lean.

**The lever ORDER mid-run:**

1. `prune-safe.sh`.
2. `nerdctl rmi` of specific already-PUSHED tags — it refuses in-use ones, so
   this is safe.
3. The regenerable cache-export directory `~/.cache/kata-buildcache`. This is a
   DIFFERENT store, which `prune-safe.sh` cannot reach: it grew 62 → 110 GB
   inside one session and caused a disk emergency.

`nerdctl system prune` and `nerdctl builder prune` are not on this list at all.
`system prune` is worse again mid-run, because "unused" means
not-container-referenced, so it deletes TAGGED cross-stage locals too — on
2026-08-18 the `cross-media-*` tags vanished mid-run and only the
registry-digest-pinned handoffs survived, via a re-pull that cost ~25 minutes.

### B8. ghcr registry hygiene

Also moved from `AGENTS.md` on 2026-09-15.

Two tools over `ghcr-common.sh`: **`ghcr-prune-package.sh`** deletes UNTAGGED
versions, **`ghcr-delete-tags.sh`** deletes NAMED tags from an explicit list and
never guesses what is legacy. Both dry-run by default
(`GHCR_PRUNE_CONFIRM=1` / `GHCR_DELETE_TAGS_CONFIRM=1` perform), and both take
the PAT from a docker login or `GHCR_TOKEN`.

**The shared Accept header is a safety property, not tidiness.** Listing the
index media types makes a multi-arch tag resolve to its INDEX, so its children
are visible; omitting them collapses the tag to a single platform manifest, and
an incomplete keep-set is exactly how a prune tool deletes something it should
not.

**Never "delete all untagged" by hand.** The per-arch entries of a multi-arch
index are themselves untagged manifests, and a chain that is pushing creates
untagged manifests seconds before it tags them. The keep-set is: every tag,
plus every index CHILD resolved live (abort on any unresolvable tag), plus
everything younger than `KEEP_DAYS`. The sanity gate checks digests, not tag
names: every tag's live digest must be readable and in the keep-set. Two tags
on one image are fine. The tag-count gate it replaced refused one on 2026-09-27,
during the `:latest-cross` rename.

Runs: 604 versions deleted on 2026-08-24; then on 2026-08-27, 23 untagged plus
47 tags with 0 failures — **81 → 34 tags, 204 → 134 versions** — with
`:latest-cross` verified 3/3 children HTTP 200 afterwards and a running build
untouched. Six long-dangling legacy tags (`android`, `compiler`, `latest`,
`media`, `sdk`, `torch`, whose children had 404'd since before either tool
existed) were deleted the same day by operator decision. `:latest` is therefore
gone and will not come back by itself: the cross lane's orchestrators tag
`cross-*` and `latest-cross`, and `build-runtime-manifest.sh` publishes only the
cross manifest, so the native lane has no build path any more. (Superseded
2026-09-22: the cross manifest was renamed from `:latest-cross` to `:latest`,
so `:latest` is live again as the default multi-arch index.)

## Phase C — Performance mode

Build hosts ship in a power-saving governor by default, which costs real
wall-clock on long compiles.

### C1. CPU frequency governor

Check what is active:

```bash
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
```

Expect one of `powersave`, `schedutil`, or `performance`.

Set it for the current boot — either form works:

```bash
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
# ...or, if cpupower is installed:
sudo cpupower frequency-set -g performance
```

Make it survive a reboot:

```bash
sudo tee /etc/systemd/system/cpu-performance.service >/dev/null <<'UNIT'
[Unit]
Description=Set CPU to performance mode
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/bin/bash -c 'echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor'
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
UNIT

sudo systemctl daemon-reload
sudo systemctl enable --now cpu-performance.service
```

### C2. AMD GPU performance level

```bash
sudo rocm-smi --setperflevel high            # all devices
sudo rocm-smi --setperflevel high --device 0 # one device
```

Inspect:

```bash
rocm-smi --showclkfrq      # current clock frequencies
rocm-smi --showperflevel   # current performance level
rocm-smi --showprofile     # available performance profiles
```


### C3. Stop the host suspending mid-build

A machine that suspends during a multi-hour cross chain kills the run. Desktop
installs enable idle suspend by default, so this is a required step on any host
that builds unattended — not an optimization.

In `/etc/systemd/logind.conf`:

```ini
HandleSuspendKey=ignore
HandleHibernateKey=ignore
HandleLidSwitch=ignore
HandleLidSwitchExternalPower=ignore
HandleLidSwitchDocked=ignore
```

```bash
sudo systemctl restart systemd-logind
```

`logind` only covers keys and the lid. Block the sleep targets themselves as
well, or something else can still trigger a suspend:

```bash
sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target
```

And belt-and-braces in `/etc/systemd/sleep.conf`:

```ini
[Sleep]
AllowSuspend=no
AllowHibernation=no
AllowSuspendThenHibernate=no
AllowHybridSleep=no
```

Verify:

```bash
systemctl status sleep.target      # should read: masked
```

On laptop-class hardware also check that TLP is not overriding the governor set
in [C1](#c1-cpu-frequency-governor): `systemctl status tlp`.
---

## Phase D — Host toolchain

### D1. A newer GCC than the distro ships

```bash
# only if add-apt-repository is missing:
sudo apt install --reinstall -y software-properties-common \
  python3-software-properties python3-launchpadlib

v=16   # the major you need; it must be newer than the distro's own
sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
sudo apt update
sudo apt install -y "gcc-${v}" "g++-${v}"
```

Register the alternatives and pick one:

```bash
sudo update-alternatives --install /usr/bin/gcc gcc "/usr/bin/gcc-${v}" 60
sudo update-alternatives --install /usr/bin/g++ g++ "/usr/bin/g++-${v}" 60
sudo update-alternatives --config gcc
sudo update-alternatives --config g++
```

Verify:

```bash
gcc --version && g++ --version
```

> The **image** toolchain is pinned separately in
> `linux/scripts/01-core/versions.env` and built by `linux/Dockerfile.toolchain`.
> This section is only about the host compiler — for building a kernel module or
> a driver out-of-tree.

### D2. Flutter on ARM hosts

After installing the SDK on an ARM host, **delete the `cache` folder inside the
SDK directory**. It ships x86_64 artifacts that are not replaced automatically,
and the failure surfaces much later as an unrelated-looking tool crash.

### D3. Multiple clang versions side by side

Same mechanism as GCC above, for clang on the host (the images carry their own,
source-built at `LLVM_RELEASE`):

```bash
v=20   # an installed clang-<major>
sudo update-alternatives --install /usr/bin/clang   clang   "/usr/bin/clang-${v}"   100
sudo update-alternatives --install /usr/bin/clang++ clang++ "/usr/bin/clang++-${v}" 100
sudo update-alternatives --config clang
sudo update-alternatives --config clang++
```

Set both, not just `clang` — a mismatched `clang`/`clang++` pair produces link
errors that read like a missing standard library.

### D4. Python CLI tools that build from source on riscv64

On amd64 and arm64, `uv tool install <something>` is one command because PyPI
has wheels. On **riscv64 there are no wheels** for the packages these CLIs
depend on, so uv silently switches to building them from source — and then needs
a Rust toolchain, a C toolchain, OpenSSL headers and the CPython headers that
nothing installed for you. The errors name a compiler or a linker, never the
missing `-dev` package, which is why this section exists.

Ready-made for the Mistral Vibe CLI, including both workarounds below:

```bash
bash linux/host-config/install-mistral-vibe-riscv64.sh
```

The four things that make it work, in case you are installing something else:

**1. The native dependencies.** `cryptography` links against OpenSSL and
`libffi`; a `python*-dev` is needed by anything with a C extension. Install the
`-dev` packages, not just the `openssl` binary — `pkg-config --modversion
openssl` is the check that actually matters:

```bash
sudo apt install -y build-essential pkg-config libffi-dev libssl-dev openssl \
  "python$(python3 -c 'import sys; print("%d.%d" % sys.version_info[:2])')-dev"
pkg-config --cflags --libs openssl   # empty output => libssl-dev is missing
```

**2. The Rust target.** This host's Rust is set up for the native RVA23 profile,
so cargo must be pointed at that triple rather than the generic
`riscv64gc-unknown-linux-gnu`:

```bash
export CARGO_BUILD_TARGET=riscv64a23-unknown-linux-gnu
export RUSTFLAGS="-C target-cpu=native"
rustc -vV | grep '^host:'
```

**3. Drop the dependencies that do not know RVA23 yet.** `textual-speedups`
pins a `target-lexicon` old enough that it rejects
`riscv64a23-unknown-linux-gnu` before compiling anything. It is a pure optional
accelerator for Textual's rendering, so deleting the dependency line costs TUI
performance and nothing else:

```bash
sed -i '/textual-speedups/d' pyproject.toml
```

That is also why the install goes through a **git clone plus `uv tool install
.`** rather than `uv tool install mistral-vibe` — you need the source tree in
order to edit `pyproject.toml` before the resolver reaches it.

**4. Clear the cache between attempts.** A source build that failed leaves a
half-built entry behind, and the retry then fails identically for a cause you
already fixed:

```bash
uv cache clean cryptography
uv cache clean textual-speedups
```

Finally, `uv` and the tools it installs live in `~/.local/bin`, which is not on
a login PATH by default — `source "$HOME/.local/bin/env"` (or `uv tool
update-shell`) before deciding an install failed.

> When a **new** dependency starts failing this way, the triage order is:
> is it a wheel or a source build (`uv` prints `Building …`), does it need a
> C library (`-dev` package) or Rust (`target-lexicon` / unknown-target errors),
> and is it optional? An optional accelerator gets deleted from
> `pyproject.toml`; a required one needs the target support upstream.


---

### D5. The host tools preflight needs and never asked for

`preflight.sh` fails on a correctly-set-up Linux host without these. `uv`,
`shellcheck` and `uidmap` were undeclared until 2026-09-15. All but `uidmap` are
user-scope and need no sudo. Two tools this list used to lead with, `pwsh` and
`pytest`, are no longer preflight prerequisites.

**PowerShell is no longer needed by preflight.** The `shared-config` slug used to
shell out to `shared/config/Sync-SharedConfig.ps1`, and without `pwsh` it failed
with `pwsh: command not found`, which read exactly like config drift. It now runs
the bash twin, `shared/config/sync-shared-config.sh`, because no hub Linux image
ships pwsh. Install pwsh on a Linux host only for the Windows-side Pester suites
and linter:

```bash
# NOTE the arch in the asset name: linux-x64 on amd64, linux-arm64 on an ARM
# host. This snippet said linux-x64 unconditionally until 2026-09-15, which on
# a Jetson installs an x86-64 pwsh that cannot exec.
curl -fsSL -o /tmp/ps.tar.gz \
  "https://github.com/PowerShell/PowerShell/releases/download/v7.6.5/powershell-7.6.5-linux-$(case "$(uname -m)" in aarch64|arm64) echo arm64 ;; *) echo x64 ;; esac).tar.gz"
mkdir -p ~/.local/powershell && tar -xzf /tmp/ps.tar.gz -C ~/.local/powershell
chmod +x ~/.local/powershell/pwsh && ln -sfn ~/.local/powershell/pwsh ~/.local/bin/pwsh
pwsh -NoProfile -Command '$PSVersionTable.PSVersion'
```

**pytest is no longer needed by preflight either.** The `mutations` slug needed it
while about a quarter of the manifest drove `python3 -m pytest` over the
`linux/llm-stack/` suites; without it each of those reported `baseline test
already fails unmutated (vacuous bite)`. Those entries left with the benchmark lab
(2026-09-12) and the NAS census (2026-09-15), and no manifest entry runs pytest
today. The serving-shape tests in `linux/llm-stack/tests/` still use it
(`llm-stack-serving.yml` installs it in a venv). To run them on the host, whose
python is PEP 668 externally-managed, the flag is not optional:

```bash
pip3 install --user --break-system-packages pytest
```

`--user` writes to `~/.local/lib/python3.*/site-packages` and touches no system
package; `--break-system-packages` only waives the PEP 668 guard.

**uv**, for three gate suites at once. `test-cmake-format`, `test-python-lint-gate`
and the `cmake-format` / `ruff` gates all bootstrap their tool through `uv`/`uvx`.
Without it all three fail with `Required tool not found: uv`, which names the
missing tool but is easy to read as a broken gate rather than a missing
prerequisite:

```bash
# installs uv + uvx to ~/.local/bin; the installer is downloaded to a file and
# checked against UV_INSTALL_SH_SHA256 (versions.env), never piped into sh
bash -c 'source linux/scripts/01-core/python_uv.sh && uv_ensure_installed'
```

**shellcheck on PATH**, for `test-shellcheck-warnings`. Note the asymmetry: the
*gate* (`lint-shell.sh`) bootstraps its own SHA-pinned binary and needs nothing
on PATH, but that *suite* asserts `command -v shellcheck` and fails without it.
The bootstrap's cache lives under `$TMPDIR`, so copy it somewhere durable:

```bash
cp "$(bash linux/scripts/lint-shell.sh --print-bin)" ~/.local/bin/shellcheck
```

**uidmap** — the one that does need sudo, and the only hard blocker for a
rootless stack. `newuidmap`/`newgidmap` must be setuid-root from the distro;
the nerdctl-full bundle ships everything else (slirp4netns, fuse-overlayfs) but
cannot ship these. Without them `containerd-rootless-setuptool.sh install`
fails and [B3c](#b3c-install-rootless-into-homelocal-no-sudo) cannot proceed:

```bash
sudo apt install -y uidmap
grep "^$(whoami):" /etc/subuid /etc/subgid   # ranges must exist (usually preallocated)
```

## Phase E — Package sources and automatic updates

### E1. A slow `apt update` is usually the mirror, not the link

Measured on this host: `apt update` took **107 s** on a 254 Mbit/s connection.
The cause was `security.ubuntu.com` round-robin DNS resolving to an unhealthy
node — nothing to do with local bandwidth. Swapping to the regional mirror took
it to **under 3 s**.

This matters beyond the host: every image build starts with `apt update`, so a
bad mirror is paid on every uncached layer.

Confirm it is the mirror and not the link or IPv6:

```bash
speedtest-cli                                  # is the link actually slow?
time sudo apt -o Acquire::ForceIPv4=true update # is it IPv6?

# where exactly does it stall?
time sudo apt update 2>&1 | while IFS= read -r line; do
  echo "$(date +%H:%M:%S) $line"
done
```

Then swap the mirror:

```bash
sudo cp /etc/apt/sources.list.d/ubuntu.sources ~/ubuntu.sources.bak
sudo sed -i 's|http://security.ubuntu.com/ubuntu|http://de.archive.ubuntu.com/ubuntu|g' \
  /etc/apt/sources.list.d/ubuntu.sources
sudo apt update
```

> Keep the backup **outside** `/etc/apt/sources.list.d/`. A `.bak` file left in
> that directory is parsed as a second source and apt warns on every run.

Revert with `sudo cp ~/ubuntu.sources.bak /etc/apt/sources.list.d/ubuntu.sources`.

### E2. Unattended upgrades

```bash
sudo apt install -y unattended-upgrades apt-listchanges
sudo dpkg-reconfigure --priority=low unattended-upgrades
```

Non-interactively, `/etc/apt/apt.conf.d/20auto-upgrades`:

```text
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
```

Which origins are allowed is set in `/etc/apt/apt.conf.d/50unattended-upgrades`.
Security-only is the conservative default:

```text
Unattended-Upgrade::Allowed-Origins {
        "${distro_id}:${distro_codename}-security";
        // "${distro_id}:${distro_codename}-updates";
};
```

**Excluding a package is the important part on a build host.** An unattended
Docker upgrade is how a working container runtime breaks overnight — see the
Jetson case in
[Linux Accelerator Images § Edge accelerators](linux-accelerator-images.md#edge-accelerators):

```text
Unattended-Upgrade::Package-Blacklist {
    "docker-ce";
    "docker-ce-cli";
    "containerd.io";
};
```

Other options worth setting deliberately rather than by default:

```text
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "02:00";
Unattended-Upgrade::Remove-Unused-Dependencies "true";
Unattended-Upgrade::Mail "root";
Unattended-Upgrade::MailOnlyOnError "true";
```

`Unattended-Upgrade::Mail` needs a working MTA on the host, or the notification
is silently dropped. `msmtp` is the least-effort option — it relays to an
existing mailbox rather than running a mail server:

```bash
sudo apt install -y mailutils msmtp msmtp-mta
```

Configure it per-user in `~/.msmtprc` — **create this without sudo**, and give
it mode 600 or msmtp refuses to use it:

```
defaults
auth           on
tls            on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile        ~/.msmtp.log

account        mymail
host           smtp.example.com
port           587
from           builder@example.com
user           builder@example.com
password       <APP_PASSWORD>

account default : mymail
```

```bash
chmod 600 ~/.msmtprc
echo "test" | mail -s "Test from $(hostname)" you@example.com
```

For a provider with 2FA, this needs an **app password**, not the account
password. The file holds that credential in plaintext, so it belongs in the
build user's home on the host, never in a repo.

Automatic reboots interrupt long builds — leave it off on a machine that runs
the chain unattended.

If the host uses kernel livepatching (Canonical Livepatch) or a vendor kernel,
read that vendor's guidance first — unattended-upgrades may still install kernel
packages without rebooting into them, so `uname -r` and the installed package
drift apart silently.

Dry-run and check it is actually firing:

```bash
sudo unattended-upgrades --dry-run --debug
systemctl list-timers --all | grep apt      # apt-daily / apt-daily-upgrade
sudo journalctl -u unattended-upgrades --no-pager
```

Logs also land in `/var/log/unattended-upgrades/unattended-upgrades.log`.

### E3. Turning automatic updates off entirely

On a build host that must not change underneath a running chain:

```bash
sudo systemctl stop    unattended-upgrades.service
sudo systemctl disable unattended-upgrades.service
sudo systemctl mask    unattended-upgrades.service

sudo systemctl stop    apt-daily.timer apt-daily-upgrade.timer
sudo systemctl disable apt-daily.timer apt-daily-upgrade.timer
sudo systemctl mask    apt-daily.timer apt-daily-upgrade.timer
```

Masking, not just disabling — a dependency can re-enable a merely disabled unit.


### E4. Pinning a package to a specific source or version

`Package-Blacklist` in [E2](#e2-unattended-upgrades) stops a package being
upgraded. Pinning is the other half: it forces which source a package comes
from, which is what you want when the distro's version is wrong for the host
(an older driver branch, a vendor repo, a PPA).

```bash
sudo tee /etc/apt/preferences.d/my-pin >/dev/null <<'PIN'
Package: somepackage*
Pin: release o=LP-PPA-somevendor
Pin-Priority: 1001
PIN
```

A priority above 1000 permits a *downgrade* to that source, which is the point —
anything at or below 1000 will not pull a package backwards. To exclude a source
entirely, pin it negative:

```bash
sudo tee /etc/apt/preferences.d/no-distro-version >/dev/null <<'PIN'
Package: somepackage*
Pin: release o=Ubuntu
Pin-Priority: -1
PIN
```

Always verify which candidate actually won — pinning fails quietly:

```bash
sudo apt update
apt-cache policy somepackage
```

> **The trap:** APT preferences require **each field on its own line**. A
> single-line `echo "Package: x Pin: ... Pin-Priority: ..."` writes a file APT
> parses as garbage and ignores, with no error. The symptom is a pin that simply
> does nothing, which reads as "pinning doesn't work here".

If the pinned package comes from a PPA and you want unattended-upgrades to keep
it current, that origin has to be allowed explicitly:

```bash
echo 'Unattended-Upgrade::Allowed-Origins:: "LP-PPA-somevendor:${distro_codename}";' \
  | sudo tee /etc/apt/apt.conf.d/51unattended-upgrades-somevendor
```
---

## Troubleshooting

### Reading the journal

```bash
# errors in a date window
journalctl -p err --since "2025-04-21" --until "2025-04-22"

# search the last week of messages by text
journalctl --since "1 week ago" --grep "Error message"

# kernel ring buffer from the PREVIOUS boot — for anything that hard-locked
journalctl -k -b -1 | grep -i thermal
```

The `-b -1` form is the one that matters after a machine wedges: the current
boot's log will not contain the cause.

### `nvidia-smi` reports a device it cannot address

```
Unable to determine the device handle for GPU1: 0000:28:00.0: Unknown Error
```

Seen on a dual-GPU host: the PSU could not supply both cards. It is a **power**
fault, not a driver fault — reinstalling the driver will not help. Run one GPU,
or move to a PSU that covers both.

### Boot drops to a GRUB prompt

A `grub>` shell means GRUB loaded its modules but could not find `grub.cfg`;
`grub rescue>` means it could not even load `normal.mod`. Usual causes are
partition renumbering or UUID changes after a resize or clone, an overwritten
bootloader after installing another OS, or deleted `/boot/grub/*`.

**Boot once from the prompt:**

```
grub> ls                                  # find the partition holding /boot/grub
grub> set root=(hd0,msdos1)
grub> set prefix=(hd0,msdos1)/boot/grub
grub> insmod normal
grub> normal
```

**Then fix it permanently.** Identify the *disk*, not the partition:

```bash
lsblk -d -o NAME,SIZE,MODEL
sudo grub-install /dev/sda        # or /dev/nvme0n1
sudo update-grub
```

Check that `/etc/default/grub` has `GRUB_TIMEOUT` greater than zero, and verify
the UEFI boot order lists the Linux entry before Windows:

```bash
sudo efibootmgr -v
```

Fast Boot can skip keypress detection and Secure Boot can block unsigned
modules — disable both while diagnosing.

If the manual steps do not get you booting, Boot-Repair automates most of them
and is worth trying before a reinstall:

```bash
sudo add-apt-repository -y ppa:yannubuntu/boot-repair
sudo apt update
sudo apt install -y boot-repair
boot-repair          # choose "Recommended repair"
```

### Choosing the default boot entry

```bash
# what entries exist
grep -E "^menuentry|^submenu" /boot/grub/grub.cfg
```

Set a permanent default in `/etc/default/grub` (`GRUB_DEFAULT=0`, or a name such
as `GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 6.8.0-45-generic"`),
then run `sudo update-grub`.

Remember the last selection instead:

```bash
sudo sed -i 's/^GRUB_DEFAULT=.*/GRUB_DEFAULT=saved/' /etc/default/grub
sudo sed -i 's/^#\?GRUB_SAVEDEFAULT=.*/GRUB_SAVEDEFAULT=true/' /etc/default/grub
sudo update-grub
```

Boot into one entry exactly once:

```bash
sudo grub-reboot "Windows Boot Manager (on /dev/sda2)"
sudo reboot
```

Or set the next boot via UEFI, which does not need a working GRUB config:

```bash
sudo efibootmgr -v
sudo efibootmgr -n 0003
sudo reboot
```

From a rescue system with the root filesystem mounted, `grubenv` can be written
directly:

```bash
sudo grub-editenv /mnt/nvme0root/boot/grub/grubenv set next_entry=9
sudo grub-editenv /mnt/nvme0root/boot/grub/grubenv list
```

### Finding what filled the disk

`no space left on device` from a build step is a host problem, not a build
problem. Locate the consumer before pruning anything:

```bash
du -sh .                                        # this directory
sudo du -ah --max-depth 2 . | sort -h           # what is under it
sudo du -ah / --max-depth=2 | sort -hr | head -20   # worst offenders, whole disk
```

On a WSL host, exclude the Windows mounts or the scan walks the entire C: drive
and reports it as Linux usage:

```bash
sudo du -ah / --max-depth=2 --exclude='/mnt/*' | sort -hr | head -20
```

`ncdu` is the interactive form and usually faster to act on:

```bash
sudo apt install -y ncdu
ncdu /
```

Container images and the build cache are the usual answer. Reclaim those through
`linux/host-config/prune-safe.sh`, which is written to spare the compile-cache
mounts — a plain `docker system prune` throws away the ccache/sccache tiers and
the next build pays for it.

### `/tmp` is full during a build

`/tmp` is often a tmpfs sized to a fraction of RAM, and a large build can
exhaust it. Resize it for the current boot:

```bash
free -h                                  # check there is RAM to give
sudo mount -o remount,size=8G /tmp
```

To clear it without removing the directory or losing its permissions:

```bash
sudo find /tmp -mindepth 1 -maxdepth 1 -exec rm -rf {} +
sudo chmod 1777 /tmp                     # sticky bit must be restored
```

Never `rm -rf /tmp` itself — recreating it without mode `1777` breaks every
service that writes there.