# Python CI: the drivers, and the three traps in `uv` The Python consumers (OrchestrANT, WebDavClient) share their whole CI surface with this repository: | Layer | Where | |---|---| | Linux lane | [`../.github/workflows/python-ci-linux.yml`](../.github/workflows/python-ci-linux.yml) (`workflow_call`) | | Windows lane | [`../.github/workflows/python-ci-windows.yml`](../.github/workflows/python-ci-windows.yml) (`workflow_call`) | | Step drivers | `linux/scripts/02-toolchain/python/ci_{tests,static_analysis,build_docs,packaging}.sh` | | uv primitives | `linux/scripts/01-core/python_uv.sh` | | Chain ORT reconcile | `uv_reconcile_chain_ort` in `python_uv.sh`, `Sync-UvChainOnnxRuntime` in `windows/scripts/modules/WindowsUv.Common.psm1`; both run the ORT census `linux/scripts/03-media/runtime/ort-venv-census.py` (the Windows image's module copy runs the one `windows/Dockerfile` puts in `C:\temp\scripts\`) | | Chain OpenCV (Windows) | `Sync-UvChainOpenCv` in `WindowsUv.Common.psm1`, over `Copy-ChainOpenCvPackage` in `WindowsPythonApp.Common.psm1` | | App bundles (zip/MSI/deb/… inputs) | [`python-app-bundles.md`](python-app-bundles.md) | | Free-threaded wheel (declared, proved) | `linux/scripts/02-toolchain/python/free-threaded-wheel.py`, run by both packaging drivers (§ Two wheels: GIL and free-threaded) | A consumer's workflow is configuration, not steps. Its `scripts/linux/ci_*.sh` are wrappers that `antfrastructure_exec` into the drivers above — see [`../shared/linux/templates/README.md`](../shared/linux/templates/README.md). ## Positional arguments: empty means default Every driver reads its positionals as `"${N:-default}"`, and that expansion treats an **empty** argument exactly like an absent one. So the reusable workflows pass every version input unconditionally and let an empty value fall through to the driver's own default — there is no conditional command building anywhere, and a caller names a version only to genuinely override it. This is also why two consumers that *looked* different were running identical commands: passing `'3.14'` to `ci_static_analysis.sh` is exactly its default. ## One arch per caller: the `arches` input `python-ci-linux.yml` takes `arches` (string, default `"x64 arm64"`): the rows to run, space-separated. The default is both rows, so every caller written before the input runs exactly what it ran before, with the same job names (`x64`, `arm64`) and the same artifact names. A caller that follows the fleet's one-file-per-arch convention ([`adopting-in-a-new-project.md` § Workflow file names and display names](adopting-in-a-new-project.md#workflow-file-names-and-display-names)) calls the lane once per file: ```yaml # .github/workflows/linux-arm64.yml name: Linux arm64 · build + test jobs: linux: uses: Kataglyphis/ANTfrastructure/.github/workflows/python-ci-linux.yml@develop with: package-name: orchestrant arches: arm64 secrets: GHCR_PAT: ${{ secrets.GHCR_PAT }} ``` The docs build and the FTP deploy run on the `x64` row only, so the `linux-x64.yml` caller is the one that passes the FTP secrets. A small `plan` job turns the list into the build matrix, and it **fails** on an unknown name (`amd64`, `x86_64`), on a name listed twice, and on an empty list. Without that, `arches: "x64 arn64"` would run one row and stay green. Separators are whitespace, newlines included, so a block-scalar input works; `x64,arm64` is one unknown name. **Why a plan job and not a filtered static matrix.** An `exclude:` cannot drop a row that `include:` defines: GitHub applies `exclude` first, and an `include` entry that no longer fits any combination is added back as a new one. So the rows are written once, in the plan step's `case` table, and the build job reads `fromJSON(needs.plan.outputs.matrix)`. The cost is that the runner labels sit inside a `run:` block, where the workflow-convention gate's `*-latest` ban cannot see them; [`tests/test-reusable-linux-lane.sh`](../linux/scripts/tests/test-reusable-linux-lane.sh) runs that step and holds the labels to the same rule. **Consumers follow hub `develop`.** The fleet calls this lane at `@develop` (since 2026-09-25), so the input reached the consumers at once. WebDavClient split its `ubuntu-26.04-amd64-arm64.yml` into `linux-x64.yml` and `linux-arm64.yml` the same day; OrchestrANT's split followed. ### riscv64: the image itself runs under QEMU `arches: riscv64` runs on an amd64 runner with the riscv64 image under QEMU, tests only. The row registers QEMU's binfmt handler through `setup-riscv64-cross` with `sysroot: false` (the sysroot serves cross builds, and a Python lane has nothing to cross-compile), runs every container step with `--platform linux/riscv64`, and carries its own `timeout: 360` in the plan's matrix - the first opt-in packaging run hit the old 180-minute budget (the measured 139-min test leg plus the emulated wheel build would not fit). The docs build stays `x64`'s, and static analysis stays off the row with it: both steps carry `if: matrix.arch != 'riscv64'`. Pass `test-extras` (comma list) so every leg syncs only those extras - a riscv64 leg wants `test`, because the full set builds wheels from source under emulation. See [`riscv64-cross-test-lanes.md`](riscv64-cross-test-lanes.md). Packaging the riscv64 row is **opt-in** through `package-emulated: true`: an arch-specific wheel (a Cython build) otherwise exists for every arch but this one. The wheel is **cross-built on the amd64 row** (`PACKAGING_CROSS_TARGET=riscv64`, `--platform linux/amd64`, the riscv64 sysroot and the staged target Python under `/opt/python-cross/riscv64`), under the five setuptools knobs [`linux-cross-builds.md` § Cross Python wheels](linux-cross-builds.md#cross-python-wheels-setuptools-knobs) pins: `CC`/`LDSHARED` are the riscv64 cross wrapper, `CFLAGS` carries the target include dir, `SETUPTOOLS_EXT_SUFFIX` the target SOABI suffix and `_PYTHON_HOST_PLATFORM` the target tag — the last one on the wheel's `python -m pip wheel` command alone, because uv reads it while inspecting the interpreter and refuses the riscv64 tag at `uv venv` and `uv build` alike (`Unknown operating system: linux_riscv64`); the sdist stays on `uv build --sdist`. The wheel link searches the sysroot's libraries first (`LDSHARED` carries `-L/lib/riscv64-linux-gnu -L/usr/lib/riscv64-linux-gnu`). setuptools appends the host interpreter's `LIBDIR` (`/usr/lib/x86_64-linux-gnu`) to every extension link, so the driver's `-lc` found the host `libc.so`. That linker script's `OUTPUT_FORMAT(elf64-x86-64)` overrides lld's emulation, and lld refused every riscv64 object "as incompatible with elf64-x86-64" (runs 37196119524 and 37329297990, reproduced 2026-10-07). It was not a dropped target, as first read: the `-Wl,-m,elf64lriscv` pin that answered that reading failed the same way, and the link succeeds without it. The emulated row only tests: its compile hung four hours on one Cython unit (run 37127505865) and could not fit the 360-minute job. The app packages of a `packaging/app.json` app do not follow - they need the AppImage tooling the image ships for amd64/arm64 only, so `ci_packaging.sh` warns and ships wheels only there. **The test leg starts from the image's uv cache seed.** `ci_tests.sh` copies `PYTHON_UV_CACHE_SEED` into uv's cache before its first sync, so the wheels the app lock's `test` extra would build under QEMU (numpy, matplotlib, contourpy, pillow, line-profiler, psutil, pyyaml: 108 min on 2026-10-07) are already built ([`consumer-image-contract.md` § The riscv64 uv cache seed](consumer-image-contract.md#the-riscv64-uv-cache-seed)). It covers the lock of the OrchestrANT commit the image was built from; a package the consumer's lock moved since builds as before. **The cross wheel is installed where it runs.** After packaging, the row runs `ci-wheel-smoke.sh` in the riscv64 image under QEMU (CON83). It takes the one `cp3XY` riscv64 wheel from `dist/`, installs it into a fresh venv over the lock's core dependencies (`uv sync --locked --no-dev --no-install-project`, from the same seed), imports the package outside the checkout and loads every compiled module the wheel owns without running its body, since a module may import an optional extra the core install lacks (`wheel-smoke.py`). A wheel built for the wrong arch, linked against a missing symbol or shadowed by the source tree fails the row. The step needs a hub pin that ships the script; a host step before it skips it with a warning when the pin is older, so a consumer's lane stays green until its own gitlink moves. The packaging leg still syncs `SYNC_EXTRAS` (the caller's `test-extras`) instead of all extras, and it shares `UV_CACHE_DIR=/workspace/.uv-cache` so its build tools come from the cache the other rows warm. Measured 2026-10-03, with the emulated predecessor: without both, the leg resolved 306 packages (opencv-python and torchvision from git among them) and was killed at the 300-minute budget. ## The static-analysis knobs, and the bandit trap between them Two knobs decide what the six analysers grade, and each has a Windows twin that carries the same default: | Knob | Windows twin | Default | |---|---|---| | `STATIC_ANALYSIS_EXTRA_PATHS` | `-ExtraPaths` | empty | | `BANDIT_EXCLUDES` | `-BanditExcludes` | `tests,.venv,.venv_static_analysis,ExternalLib,third_party,archive,docs/test_results` | `STATIC_ANALYSIS_EXTRA_PATHS` is a space-separated path **list**, word-split on purpose, so no element may contain a space. It exists because a consumer whose importable package is not the whole first-party tree had the rest graded by nothing: OrchestrANT's `benchmarks/`, `frontend/`, `bench/` and `examples/` were outside every analyser until it landed. **The trap.** Both drivers spelled the extras as one `-r` per path. bandit's `-r` is `store_true` against a **single** `nargs='*'` positional, so `bandit -r a -r b` is `unrecognized arguments` and exit 2 — measured against bandit 1.9.4 in the family image. The bandit gate therefore failed on the very knob the other five analysers handled, which is why OrchestrANT could not adopt it. The form bandit accepts is **one `-r` followed by the whole target list**, and `tests/test-python-static-analysis.sh` counts the flags so it cannot come back. `BANDIT_EXCLUDES` **replaces** the default list rather than adding to it: a consumer that names an exclude set means that set. Setting it is how a consumer stops hard-coding the whole `-x` string in its own driver. ## Turning the Windows PowerShell lint on `python-ci-windows.yml` takes `lint-powershell` (boolean, default `false`) and `lint-path` (string, default `scripts`). Together they add a second job on the same Windows runner that runs [`windows/scripts/Invoke-Lint.ps1`](../windows/scripts/Invoke-Lint.ps1) — the mandatory parse pass, the AST traps and PSScriptAnalyzer — over the **caller's** tree, with the hub's ruleset consumed by reference. ```yaml jobs: windows: uses: Kataglyphis/ANTfrastructure/.github/workflows/python-ci-windows.yml@develop with: lint-powershell: true lint-path: scripts/windows secrets: GHCR_PAT: ${{ secrets.GHCR_PAT }} ``` It is a separate job and deliberately does **not** `needs:` the build: a syntax error in the build scripts is exactly when the lint is worth having, and it needs no image pull. `-FailOnAnalyzer` is passed unconditionally and is not an input — without it the analyzer prints its findings and exits 0, which is a check that cannot fail. **The lint without the build.** `build-python-package` (boolean, default `true`) gates the container build job, so a repo with no Python package can call this lane for the gate alone: ```yaml jobs: powershell-lint: uses: Kataglyphis/ANTfrastructure/.github/workflows/python-ci-windows.yml@develop with: build-python-package: false lint-powershell: true ``` There is no `secrets:` block there and that is the point: `GHCR_PAT` is `required: false`, because a required secret is refused at call time and would have made the lint unreachable for exactly the callers this switch is for. The build job asserts the token in its own first step, so a caller that wanted the build and forgot the secret is told which input it missed instead of failing inside a `docker login` against ghcr. OxidANT — a Rust crate whose Windows container build is a different workflow — had measured the lint job as byte-for-byte its own and still could not call this lane until the gate existed. Every caller written before the switch is unchanged, because it defaults on. ## Trap 1 — `--all-extras` is fatal with declared conflicts `uv sync --all-extras` is not "install as much as possible". On a project that declares `[tool.uv] conflicts`, uv refuses outright: ``` error: Extras `ml-ai` and `ml-ai-webgpu` are incompatible with the declared conflicts: {`orchestrant[ml-ai]`, `orchestrant[ml-ai-webgpu]`} ``` There is no flag that means "pick a satisfiable subset". OrchestrANT declares 12 pairwise conflicts across two mutually-exclusive families (the `ml-ai-*` backends and the `pytorch-*` backends), so every one of its lanes died here regardless of what it had been asked to do. `uv_sync_project()` handles it, in this order: 1. **`UV_SYNC_EXTRAS`** — an explicit list (`"a,b"` or `"a b"`) becomes `--extra a --extra b` and `--all-extras` is dropped. The project knows best. 2. **Auto-detect** — otherwise the conflict groups are parsed out of `pyproject.toml` and enough extras are excluded via `--no-extra` to make `--all-extras` satisfiable. Greedy in **declaration order**: keep an extra unless it conflicts with one already kept. Declaration order is not arbitrary — it keeps the first-declared member of each family, which for OrchestrANT resolves to `ml-ai` and `pytorch-cpu`, the right pair for CI. The choice is logged, with a pointer to `UV_SYNC_EXTRAS`. A project with no conflicts is unaffected: the exclusion list comes back empty and the command is byte-identical to before. The group scanner reads TOML, not one indentation style. It used to gather the `extra = "…"` occurrences of a line only **after** the character walk had already closed the group on that line's `]`, so a group written inline — `conflicts = [ [ { extra = "a" }, { extra = "b" } ] ]`, which uv accepts — produced no group, excluded nothing, and left `--all-extras` to die on the very conflict it was meant to route around. The group's text is now accumulated during the same walk and read at the `]` that closes it, so the inline, multi-line and mixed layouts all give the same answer. OrchestrANT writes the multi-line form, so this was latent there; what settles it is a consuming repo's CI lane running `uv sync --all-extras`, because no ANTfrastructure cross stage calls `uv_sync_project` at all — it is reached only from `02-toolchain/python/ci_*.sh`. ## Trap 2 — `UV_PYTHON` beats the activated venv The CI images export `UV_PYTHON=/opt/venv/bin/python` (a root-owned system venv) and run as the non-root user `kataglyphis`. **uv honours `UV_PYTHON` over the activated virtualenv**, so `--active` alone is not enough — the sync targets `/opt/venv` and dies: ``` error: failed to remove file `/opt/venv/lib/python3.14/site-packages/...`: Permission denied (os error 13) ``` Both `uv_sync_project()` and `uv_pip_install_requirements()` therefore pin `--python /bin/python`. That pin is load-bearing, not tidiness. If you add another uv entry point, pin it too, and end it in `uv_reconcile_chain_ort` (Trap 3). The image after CON11 exports neither variable (in source since 2026-09-26): `Dockerfile.torch` empties both, since Docker cannot unset an inherited ENV, and the entrypoint unsets them, so an activated venv is what uv targets again while `/opt/venv/bin` stays first on `PATH`. Keep the pin: every image before it exports both, and the pin is correct on either. The two traps stack: the extras error hides the venv error, because the resolve never gets far enough to write anything. Fixing only the first one just moves the failure. ## Trap 3 — ONNX Runtime comes from the chain, not PyPI The owner rule of 2026-09-23 says every component that uses ONNX Runtime loads the ORT this repository builds ([`onnxruntime-single-source.md`](onnxruntime-single-source.md)). A consumer's test venv inside our images is such a component. Before the rule, `uv sync --all-extras` installed whatever the lock named, so OrchestrANT's CI tested on PyPI `onnxruntime`, `onnxruntime-gpu` and `onnxruntime-genai` (plus the `-directml` pair on Windows), all overlapping in one `site-packages/onnxruntime`. **Inside our images the sync installs no PyPI ORT at all** (since 2026-10-12). With a `uv.lock` and a chain store (step 1 below: `ORT_CHAIN_WHEEL_DIR` on Linux; on Windows also the image's `PYTHON_WHEELS`), `uv_sync_project` and `Sync-UvProjectDependencies` pass `--no-install-package` for every lock entry the census counts as ONNX Runtime (`onnxruntime` and `onnxruntime-*`), the unlocked retry too, and the reconcile installs the chain wheels in their place. The chain's ORT then serves a Python that PyPI's has no wheel for yet: 1.31.0 stopped at cp314 when 3.15.0 came out, and a locked sync on 3.15 failed on it. Every `uv_sync_project` (Linux) and `Sync-UvProjectDependencies` (Windows) now ends by reconciling the venv it synced: 1. **Inside our images?** The chain wheel store says so. Linux: `ORT_CHAIN_WHEEL_DIR`, which `linux/Dockerfile.torch` sets to `/opt/onnxruntime-wheels`. Windows: `ORT_CHAIN_WHEEL_DIR`, else the image's `PYTHON_WHEELS` (`C:\runtime\wheels`), else `C:\runtime\wheels` if it exists. 2. **The census names the ORT distributions.** The venv's own interpreter runs `ort-venv-census.py --purge-list` with `-I`: every distribution named `onnxruntime` or `onnxruntime-*`, and every owner of the `onnxruntime`, `onnxruntime_genai` or `onnxruntime_extensions` package. 3. **No ORT distribution:** inside our images the venv's interpreter still checks that none of those three packages imports. An ORT package no distribution owns (copied files, a `pip --target` leftover) fails the sync. **Outside our images:** a loud `NOTICE` that names the distributions, and the venv stays as uv resolved it. 4. **The ABI check comes first.** The chain wheels are built for the image interpreter (cp314 today), and uv installs a path wheel of another ABI tag without complaint. So before the venv is touched, each store ORT wheel's tag must fit the venv's own (`abi3` and `none` fit any), or the sync fails naming the misfits. A `cp3XYt` venv is given the image's twins instead, when `PYTHON_WHEELS_FREE_THREADED` names their store (both lanes, § Free-threaded and GIL legs). On Linux it takes only the twins of the flavours `ORT_CHAIN_WHEEL_DIR` holds, since a GPU image's twin store carries two ORT flavours and a venv given both has two cores. 5. **The replacement:** `uv pip uninstall` every listed distribution that reached the venv (a skipped one never did), then `uv pip install --no-index --no-deps --force-reinstall` every `onnxruntime[-_]*.whl` in the store. Then two proofs, and **either one failing fails the sync**: `--check --store` (each ORT distribution byte-identical to a store wheel, `onnxruntime` owned once), and `import onnxruntime` in that venv, which catches a load failure on a fitting tag. 6. **`UV_NO_SYNC=1` is exported on success**, only when it was unset, and released by the next sync of a venv without ORT. `uv run` re-syncs to the lock by default and would put PyPI ORT back; `uv sync` itself ignores the variable. A failed `uv sync` keeps its own exit status; the reconcile never runs after it. **Where the Linux store comes from.** `/opt/wheels` is only a bind mount inside the torch RUN, so `setup-torch-venv.sh stage_chain_ort_wheels` copies exactly the wheels the census matched `/opt/venv` against (the installed flavour only) into `ORT_CHAIN_WHEEL_DIR` and proves `/opt/venv` against the store. Any failure stops the image build; a venv without ORT leaves the store empty. **A leg on another interpreter.** Inside our images an ORT project's legs run on the image interpreter. Two remedies work on both lanes: drop the other legs (`test-python-versions`, or the consumer's `$PythonVersions`), or list them in `EXPERIMENTAL_PYTHON_VERSIONS`, so a failed sync warns instead of failing the matrix. The reusable workflow passes only `PYTEST_PATHS` and `FREE_THREADED_SYNC_EXTRAS` into the container (§ What the test leg runs), so the consumer's `scripts/linux/ci_tests.sh` wrapper must export it. `UV_SYNC_EXTRAS` is NOT a remedy: every sync of the run reads it, so leaving ORT out of one leg leaves it out of all of them. The hub defaults are the image interpreter (owner decision 2026-09-23; the images carry CPython 3.14 only): `ci_tests.sh` `PY_VERSIONS='3.14'` and `ci_build_docs.sh` `COVERAGE_VERSION=3.14`, like the static-analysis and packaging drivers. `tests/test-python-ci-defaults.sh` holds them to `versions.env`'s `PYTHON_VERSION`. OrchestrANT dropped its 3.13 legs. A project that needs another interpreter names it for every driver, docs included. WebDavClient (no ORT; atheris has no cp314 wheel) pins 3.13 for its tests, static analysis and packaging, and needs `docs-python-version: '3.13'` as well: nothing sets it for it, and without it the first hub bump past this default fails its docs job on atheris. | Message | Cause | Fix | |---|---|---| | `chain ORT: need the store … the interpreter … and the census …` | the store is declared but missing (an image regression), or the venv interpreter or the census is missing | the census comes from the hub checkout (`linux/scripts/03-media/runtime/`) or, for the Windows image's module copy, from `C:\temp\scripts\ort-venv-census.py`; an image built before `windows/Dockerfile` COPYed it there has none, so import the module from the hub checkout | | `the store … holds no onnxruntime wheel` | the store is empty, for example a venv built without ORT | build the store with ORT, or keep ORT out of that venv | | `… is a cp313 venv, and the chain wheels are built for the image interpreter: …` | the leg's interpreter is not the image's; the venv was not touched | drop the leg or list it in `EXPERIMENTAL_PYTHON_VERSIONS` | | `… imports ONNX Runtime with no distribution to purge (…)` plus census `FAIL` lines | an ORT package no distribution owns | remove the files the `FAIL` lines name | | `the chain onnxruntime does not import in …` | the tag fits but the load fails (a DLL or `.so` the venv cannot find) | the import error follows the message | | `still carries a non-chain ONNX Runtime` | the census found a foreign distribution | read the `FAIL` lines that follow | `onnxruntime-extensions` and prebuilt plugin-EP wheels have no chain counterpart. They are purged and not replaced, which is what the rule asks for. **Not covered:** tool venvs built from requirement files (`uv_pip_install_requirements` / `Install-UvRequirements`: cmake-format, docs), `uvx` / `uv tool`, pip calls outside these helpers, ORT vendored under another import name, and runners that are not our images (they get the notice only). The Windows chain wheel importing in a plain consumer venv, without the app venv's DLL-directory shim, has not been run inside `:winamd64` yet. **Tests:** `linux/scripts/tests/test-uv-chain-ort.sh` (a real venv plus the real census, with a uv stand-in), `windows/scripts/tests/Uv.ChainOrt.Tests.ps1` (including the image's module copy in the layout `windows/Dockerfile` builds), and the mutation family `uv-chain-ort` in `docs/scripts/mutations.json`. **On Windows, OpenCV comes from the image too.** PyPI's `cv2.pyd` imports Media Foundation, which Server Core lacks, so `import cv2` fails in every `:winamd64` venv. After the ORT reconcile, `Sync-UvProjectDependencies` calls `Sync-UvChainOpenCv`. It uninstalls every `opencv*` distribution and copies the image's cv2 into the venv. The copy's config names the image dirs its DLL closure lives in: OpenCV's `bin`, the chain ORT and FFmpeg. Then it holds `uv run` off the lock with `UV_NO_SYNC`, as the ORT reconcile does, because a re-sync would put PyPI's opencv back. It is a no-op outside the image and in a venv without OpenCV. Why the config must be rewritten, and the bundle's variant of the same copy: [`python-app-bundles.md` § Windows: the image's OpenCV](python-app-bundles.md#windows-the-images-opencv-not-pypis). Test: `windows/scripts/tests/Uv.ChainOpenCv.Tests.ps1`. ## What the test leg runs **The project's own `testpaths`.** `ci_tests.sh` passes no path to pytest unless `PYTEST_PATHS` (the `test-paths` input, comma-separated) names some, so pytest reads the consumer's `testpaths`. It used to hard-code `tests/unit`. When WebDavClient gained its arm64 and Windows lanes (2025-10-20) every lane narrowed to that directory, and its six WebDAV client tests and the integration test ran on no lane until 2026-10-01. Windows drivers live in the consumers (`scripts/windows/Build-Windows.ps1`) and must not hard-code a subdirectory either. **A free-threaded leg that gates.** `3.14t` sits in `EXPERIMENTAL_PYTHON_VERSIONS` by default, because `--all-extras` pulls in wheels that only exist for the GIL build (onnxruntime-genai-cuda, ai-edge-litert, atheris, bcrypt 4). Measured 2026-10-01, that leg synced nothing and ran no test on any lane of OrchestrANT or WebDavClient, while every job stayed green. The `free-threaded-extras` input (`FREE_THREADED_SYNC_EXTRAS`) fixes that: - **Set:** a leg whose version ends in `t` syncs only those extras (`test`, say) plus the core dependencies, and gates like any other leg. - **Empty:** the leg stays experimental. A failed venv, sync or test run then prints a `::warning title=Python not tested::` annotation, so the run's summary shows it. - **ORT:** a core set without ORT never meets the chain ORT's ABI check below (Trap 3). A test that needs an extra the leg does not install must skip, with `pytest.importorskip`. Tests: `linux/scripts/tests/test-python-ci-defaults.sh`, mutations `python.ci-tests-runs-the-configured-testpaths`, `python.ci-tests-free-threaded-leg-syncs-its-own-extras` and `python.ci-tests-experimental-sync-failure-is-annotated`. ## Windows arm64: the runner-native test job `python-ci-windows.yml` with `arm64-tests: true` adds a `windows-11-arm` job. No Windows arm64 image runs on a hosted runner, so it takes no container. It checks the caller out with its ANTfrastructure pin and runs `windows/scripts/python/Invoke-PythonTestLegs.ps1 -InstallUv`. - **uv** is the `UV_VERSION` release zip, checked against `UV_WINDOWS_ARM64_SHA256` in `versions.env`. `bump_versions.py` refreshes that pin with the Linux ones. - **Legs** come from `arm64-python-versions` (default: the pinned X.Y; `pinned` and `pinnedt` in a list stand for X.Y and X.Yt). Each leg gets its own venv through `New-UvProjectEnvironment`, so a bare `X.Y` asks for `X.Y+gil`. Without that, an `X.Yt` leg earlier in the same job would hand it the free-threaded build (§ Free-threaded and GIL legs in one container). - **Extras:** `arm64-extras` (empty means all extras), `free-threaded-extras` for a `t` leg, and `test-paths`, all as on Linux. Every leg gates. A failed leg does not stop the next one, and the error names every failed leg and its step: venv, sync or pytest. - **Wheels only, in practice.** The runner has MSVC but not the family's clang-cl setup, and a package without a `win_arm64` wheel would build with the default `cl`. Keep such packages off ARM64 with a marker, `"; platform_machine != 'ARM64'"`. Linux aarch64 reports `aarch64`, so the marker leaves Linux alone. WebDavClient does that for py-spy (no `win_arm64` wheel) and line_profiler (none for 3.14t). Check a lock before you turn the job on: `uv export --locked`, then `uv pip install --dry-run --no-deps --only-binary :all: --python-platform aarch64-pc-windows-msvc`. Run it once per leg's interpreter (`--python 3.14t` for the free-threaded leg). The consumer must pin an ANTfrastructure that ships the script. The job refuses an older pin by name instead of failing on a missing file. Test: `windows/scripts/tests/PythonTestLegs.Tests.ps1` (a uv stand-in that records each call). ## Free-threaded and GIL legs in one container Since CON66 the image ships the free-threaded interpreter itself, at `/usr/local/bin/python3.14t` and outside uv's store, so a `3.14t` leg downloads nothing. Since 2026-10-07 it is the toolchain stage's `--disable-gil` source build of `PYTHON_VERSION`, no longer uv's python-build-standalone download ([`consumer-image-contract.md` § The free-threaded Python](consumer-image-contract.md#the-free-threaded-python)). Since 2026-10-11 the leg also cannot download one. In the shipped image, whose ENV names `PYTHON_MAJOR_MINOR` and `PYTHON_CROSS_ACTIVE_ROOT`, `uv_venv_create` runs `uv venv` for `PYTHON_MAJOR_MINOR` and for its `t` twin with `UV_PYTHON_DOWNLOADS=never`, and logs the interpreter it took (`the image's own interpreter, UV_PYTHON_DOWNLOADS=never (/opt/python-freethreaded 3.14.8)`). When the image lacks that interpreter, the leg fails with `python3.14t is missing, though this image ships it` instead of downloading one. Other versions, and runs outside the image, still let uv download, and a caller's own `UV_PYTHON_DOWNLOADS` wins. Measured in the `:latest` amd64 child on 2026-10-11 with `python3.14t` hidden from `PATH`: the old code downloaded a free-threaded CPython and WebDavClient's `3.14t` leg passed, 23 tests green; the new code fails the leg. Test: `linux/scripts/tests/test-uv-python-request.sh`. An image older than 2026-10-07 has no `python3.14t`, so its `3.14t` leg now fails, and the rest of this section is history. uv 0.12 lets a plain `3.14` request take a free-threaded build: once a `3.14t` leg has downloaded `3.14.7+freethreaded` into uv's managed store, a later `uv venv --python 3.14` in the same container picks it over the image's GIL `3.14.4` (WebDavClient, 2026-09-29, BACKLOG CON40). `uv_ensure_python_available` also stripped the `t`, so `python3.14` counted as having `3.14t`. `uv_venv_create` therefore hands uv `uv_python_request`'s form: a bare `X.Y[.Z]` becomes `X.Y[.Z]+gil`, and `3.14t`, paths and explicit variants pass unchanged. Only discovery takes `+gil`; `uv python install 3.14+gil` fails with `No download found`, so the install step passes the bare version, which installs the GIL build. Measured in `:latest` on 2026-09-30: legs 3.14t, 3.14, 3.13, 3.15, 3.14t, 3.14 in one container each got the interpreter they named. Test: `linux/scripts/tests/test-uv-python-request.sh`. Windows has the same trap and the same cure. `New-UvProjectEnvironment` asks uv for `Get-UvPythonRequest`'s form. OrchestrANT's 3.14 leg took a free-threaded 3.14.7, which has no wheel for one of its locked packages (2026-09-30). Test: `windows/scripts/tests/Uv.PythonRequest.Tests.ps1`. A Windows `3.14t` leg gets the image's interpreter too. A `:winamd64` built after 2026-10-07 ships `C:\python-freethreaded\python3.14t.exe`, source-built beside the GIL build and last on `PATH` ([`windows-builds.md` § The free-threaded CPython](windows-builds.md#the-free-threaded-cpython)). `uv venv --python 3.14t` and `uv build --python 3.14t` resolve to it with `UV_PYTHON_DOWNLOADS=never`. An older image still has uv download one, and so does the arm64 runner-native job above, which runs without an image. **The chain's own twins.** The same `:winamd64` ships `cp3XY-cp3XYt` builds of the chain wheels whose code declares free-threading (ONNX Runtime, PyAV, apache-tvm-ffi and the two IREE packages) in `PYTHON_WHEELS_FREE_THREADED` (`C:\runtime\wheels-free-threaded`), apart from the GIL store ([`windows-builds.md` § The free-threaded wheels](windows-builds.md#the-free-threaded-wheels)). `Sync-UvChainOnnxRuntime` reads that store for a venv whose ABI tag is `cp3XYt`, so a Windows `3.14t` leg of an ORT project gets the chain ORT like its GIL legs. Without the store it fails on the ABI as before. The arm64 bundle carries the same store with `win_arm64` twins, minus the IREE compiler ([`windows-builds.md` § The arm64 cross twins](windows-builds.md#the-arm64-cross-twins)). Linux does the same since 2026-10-07. `:latest` names its twin store, `/opt/wheels-free-threaded`, in `PYTHON_WHEELS_FREE_THREADED`, a name with no version that Windows reads first as well, and `uv_reconcile_chain_ort` reads it for a `cp3XYt` venv ([`consumer-image-contract.md` § The free-threaded wheels](consumer-image-contract.md#the-free-threaded-wheels)). An image without the variable still refuses the leg on the ABI, and the runtime smoke's `FT-STORE` gate fails it. Test: `linux/scripts/tests/test-uv-chain-ort.sh`. **uv downloads nothing for a `+gil` request**, so a version the host lacks is installed first. OmniAccelerANT's 3.12 venv (its CMake format gate) stopped with `No interpreter found for Python 3.12+gil in managed installations, search path, or registry` (2026-10-01). Like Linux's `uv_ensure_python_available`, `New-UvProjectEnvironment` now asks `uv python find` first. When nothing is found it runs `uv python install X.Y`, which installs the GIL build, and then creates the `+gil` venv. Measured in `:winamd64`: 3.12 is installed and its venv is a GIL 3.12.14. 3.14 still takes the image's own `C:\temp\cpython`, with no download. `uv build` is a second door into the same trap. It discovers its own interpreter and ignores the venv, so `Invoke-CiPackaging.ps1` and `ci_packaging.sh` pass it the same `--python X.Y+gil`. Without that flag, OrchestrANT's only Windows binary wheel was `cp3XY-cp3XYt` (2026-10-01). The app bundle's GIL runtime could not install it. The GIL interpreter `uv build` then finds is the image's in-tree build (`C:\temp\cpython\PCbuild\amd64`). Its `python314.lib` sits beside `python.exe`, not in a `libs\` directory, which is the only place setuptools looks: the Cython link failed with `LNK1104: python314.lib`. The free-threaded download had hidden this, because it is a regular install. The binaries step therefore puts the interpreter's directory on `LIB` whenever a `python3*.lib` sits there. ## Two wheels: GIL and free-threaded A project that declares free-threading support ships two binary wheels: `cp3XY-cp3XY` for the GIL interpreter and `cp3XY-cp3XYt` for the free-threaded one. The sdist and the pure `py3-none-any` wheel are built once, because they serve both. **Opting in** is the official trove classifier, in `pyproject.toml`'s `[project]` table: ```toml classifiers = ["Programming Language :: Python :: Free Threading :: 2 - Beta"] ``` Any level counts (`1 - Unstable`, `2 - Beta`, `3 - Stable`, `4 - Resilient`), and so does the bare `Programming Language :: Python :: Free Threading`. `Programming Language :: Python :: 3.14t` is no classifier: PyPI refuses an upload that names it, and the drivers ignore it. The Cython build must declare the same, with `freethreading_compatible=True` in `cythonize(compiler_directives=...)` (Cython 3.1+). Without it, every compiled module re-enables the GIL when it loads, and the proof below fails. **What the drivers do.** `ci_packaging.sh` and `Invoke-CiPackaging.ps1` read the classifier with `linux/scripts/02-toolchain/python/free-threaded-wheel.py declares`. When it is there: 1. **The build.** After the GIL Cython wheel, the same `CYTHONIZE=True` build runs on the free-threaded twin of the packaging version (`3.15` gives `3.15t`). The interpreter is the image's, found by `uv python find` and never downloaded ([`consumer-image-contract.md` § The free-threaded Python](consumer-image-contract.md#the-free-threaded-python)). Without one the step fails and says so. 2. **The tag.** The build must leave one `cp3XYt` wheel. On Linux it is auditwheel-repaired like the GIL wheel. A pure result is logged and dropped. 3. **The proof.** A fresh venv of the same interpreter installs the shipped wheel without its dependencies, and `free-threaded-wheel.py prove` loads every compiled module the distribution owns. It creates each module without running its body: that is where CPython decides about the GIL, and a missing optional dependency then cannot fail or fake the verdict. A bare `.so` or `.pyd` counts as a module only when it exports `PyInit_`, so a library the wheel bundles beside its modules is not loaded as one. The step fails when `sys._is_gil_enabled()` is true afterwards, and names each module whose `RuntimeWarning` re-enabled it. The image's own wheels get the same twin and the same proof in the media stage, decided by a table rather than a classifier ([`consumer-image-contract.md` § The free-threaded wheels](consumer-image-contract.md#the-free-threaded-wheels)). Without the classifier the drivers build exactly what they built before, and log one line: `free-threaded wheel skipped: the project does not declare support (...)`. **The override** is `PYTHON_FREE_THREADED_WHEEL` (on Windows also `-FreeThreadedWheel`): `auto`, the default, follows the classifier, `on` builds without it, and `off` skips it. Any other value fails the run. A caller sets it with the `free-threaded-wheel` input of `python-ci-linux.yml` (passed to the packaging container) and of `python-ci-windows.yml` (passed to the build container, where `Invoke-CiPackaging.ps1` reads it). **The riscv64 cross build skips it** (`PACKAGING_CROSS_TARGET`). It has no free-threaded target interpreter and logs `free-threaded wheel skipped: the riscv64 cross build has no free-threaded target interpreter`. The Windows arm64 cross lane builds it, as below. ### The suite on the installed wheel The proof loads the wheel's modules; it runs none of their code. The `3.14t` test leg runs the code, but from the sources, not the Cython build. A project that wants its tests on the compiled modules runs `linux/scripts/02-toolchain/python/test-free-threaded-wheel.sh` after `ci_packaging.sh`, from its own packaging wrapper: ```bash bash "$(antfrastructure_path linux/scripts/02-toolchain/python/test-free-threaded-wheel.sh)" \ --python-version "${1:-3.14}" --extra test --copy tests/unit/test_thread_safety.py \ --env ORCHESTRANT_REQUIRE_COMPILED=1 --isolated -- -W error tests/unit/test_thread_safety.py ``` - **What it installs.** A fresh venv of the image's free-threaded interpreter, found and never downloaded, gets the lock's set (`uv export --locked --no-emit-project`, with `--extra` if given), then the shipped `cp3XYt` wheel of this arch with `--no-deps`. - **Where it runs.** Each `--copy` is staged at its relative path in a temporary directory, and pytest runs there, because from the checkout the sources would shadow the installed modules. Do not copy the package itself. `--isolated` adds `python -I`; leave it off when the tests import siblings by package path (`from tests.helpers import …`), which needs the run directory on `sys.path`. - **What it passes on.** `--env NAME=VALUE` reaches the pytest process only, and everything after `--` goes to pytest. The project's pytest config is not copied unless a `--copy` names it, so pass its warning filters there. - **When it skips.** In the three cases above in which the driver builds no free-threaded wheel: a cross build, `PYTHON_FREE_THREADED_WHEEL=off`, and `auto` without the classifier. A declared project with no `cp3XYt` wheel of this arch in `dist/` fails, and so does a call without `--copy`. OrchestrANT runs its thread-safety tests this way, WebDavClient its whole suite. ### The arm64 cross lane's free-threaded wheel Since 2026-10-12 (CON75) the Windows arm64 cross lane ships `cp314-cp314t-win_arm64`. The `:winarm64` image carries both halves. The host's x64 `C:\python-freethreaded` runs the build, and the target tree `C:\runtime\python-freethreaded` (CON74) supplies `python314t.lib`. `Invoke-CiPackaging.ps1` starts `New-CrossFreeThreadedWheel.ps1` as a child process, so the VS environment it enters ends with it: 1. **The plan.** The classifier decides, as natively. The step is skipped when the image has no target tree. 2. **The environment.** The script enters VS's x64_arm64 environment. A uv venv of the host's 3.14t holds `[build-system].requires` as declared. Its `sitecustomize` pins `EXT_SUFFIX` to `.cp314t-win_arm64.pyd` (`New-FreeThreadedBuildPython`). 3. **The target, for every `build_ext`.** `DIST_EXTRA_CONFIG` sets `plat_name = win-arm64` and the target's `libs\` as `library_dirs` for `build_ext`, and the platform for `bdist_wheel`. - A command-line `build_ext -L` reaches only the first instance. - OrchestrANT's `CleanBuildPy` wipes `build\lib`, so `bdist_wheel` relinks. With `-L` on the command line, that relink took the host's x64 `python314t.lib`. 4. **The build and the gates.** `uv build --wheel --no-build-isolation` runs with that venv. Then the static gates check the tags, the imports (`python314t.dll` only) and `Assert-WheelTargetArch`. 5. **The kit.** The wheel goes to `dist\windows-arm64\free-threaded\`, never `dist\`, because `Select-PythonAppWheel` refuses a binary wheel of another ABI. Beside it go the target tree as `python\`, `free-threaded-wheel.py`, `WindowsPythonWheel.Common.psm1` and `Test-FreeThreadedWheel.ps1`. **The device.** `container-ci-windows.yml`'s arm64 run job starts `Test-FreeThreadedWheel.ps1` from the downloaded product. It runs `Invoke-TwinDeviceProof`, the bundle gate's own proof: - the wheel installs offline into a venv of the packed `python3.14t.exe`; - then `free-threaded-wheel.py prove` runs on it. A product without the directory logs one line and passes. **The project's part.** setuptools' own MSVC compiler needs nothing: `plat_name = win-arm64` selects the ARM64 `cl.exe`. A project that swaps in `clang-cl` must also name the target, because `clang-cl` otherwise compiles for the host. - OrchestrANT adds `--target=aarch64-pc-windows-msvc` to its extensions' `extra_compile_args` when `VSCMD_ARG_TGT_ARCH` is `arm64`. VS's cross environment sets that variable. - In `:winarm64` (`ba6bfaae…`), OrchestrANT 0.0.30's wheel built this way and passed all three gates (2026-10-12). - On `windows-11-arm` it then loaded: OrchestrANT run 38221408508 imported all 64 compiled modules on free-threaded 3.14.8 with the GIL disabled (2026-10-12). - A project that sets the compiler's `cc` to `clang-cl` must `initialize()` the compiler first. The first `compile()` initializes lazily and puts `cl.exe` back, so OrchestrANT's and WebDavClient's first extension compiled with MSVC's `cl` until 2026-10-12. A `spawn` hook does not catch it, because setuptools 84 never calls one. **The app bundles pick by their runtime's ABI.** `python-app-bundle.sh` and `Select-PythonAppWheel` take the `cp314` wheel for their GIL runtime. A binary built for another ABI alone is an error, not a fallback to the pure wheel. **auditwheel on Linux** is PATH's, else the binary packaging venv's (OrchestrANT's `packaging` extra installs it there). Until 2026-10-07 only PATH counted, and it never has one, so every Cython wheel shipped as `linux_` under a log line calling it pure. With no auditwheel anywhere, a platform wheel ships unrepaired, with a warning. Tests: `linux/scripts/tests/test-python-free-threaded-wheel.sh` and `windows/scripts/tests/PythonWheel.FreeThreaded.Tests.ps1`. Inside `:latest` the Linux suite also runs `prove` for real: it compiles two C extensions against `/opt/python-freethreaded` (or `FT_PYTHON`), one declaring `Py_MOD_GIL_NOT_USED` and one not. A host without a free-threaded interpreter logs a SKIP line for that case. ## Upgrading Python One line moves every image, lane and default: `PYTHON_VERSION` in `linux/scripts/01-core/versions.env` (owner rule 2026-10-12: an upgrade must stay that easy). 1. Set `PYTHON_VERSION=X.Y.Z` and run `python3 docs/scripts/bump_versions.py --write --only PYTHON_VERSION`. It moves to the newest X.Y patch and writes `PYTHON_TGZ_SHA256` from python.org's tarball. 2. `python3 docs/scripts/sync_versions.py --write` copies the pin into the Dockerfiles' ARG defaults. 3. Rebuild the chain from `compiler`. Its GCC, LLVM and Rust layers stay cached, because the Python ARGs sit right above the CPython RUN; every stage after it builds again, and so does every wheel the chain builds per Python ABI (ONNX Runtime and its twins, GenAI, IREE, PyAV, tvm-ffi, LiteRT, OpenCV's cv2, riscv64's torch). **What follows without an edit:** - The images ship the source-built interpreter on every arch, GIL (`/usr/local/bin/pythonX.Y`) and free-threaded (`/opt/python-freethreaded`), never the distro's `python3`, so Ubuntu's own Python version does not matter. arm64 and riscv64 build what apt used to supply for it (numpy, PyGObject, pycairo, contourpy) from sdists. - `PYTHON_MAJOR_MINOR`, which the image advertises, and the wheel tags `PYTHON_ABI_TAG` and `PYTHON_FT_ABI_TAG` (`common.sh`). - The drivers' defaults. `ci_tests.sh` runs X.Y, plus X.Yt when `pyproject.toml` declares free-threading; the packaging, static-analysis, docs, wheel-smoke and wheel-test drivers take X.Y, and so do Windows' `Invoke-CiPackaging.ps1`, `Invoke-CiStaticAnalysis.ps1` and `Invoke-PythonTestLegs.ps1` (`Get-PinnedPythonVersion`). A consumer that passes no version follows the image, and one that picks its legs writes `pinned` (`pinned` alone for the GIL leg, `pinnedt` for its twin). `ci_build_docs.sh` also publishes the default leg's reports as `pytest-report.html`/`.md` and `pytest-report-free-threaded.html`, so a docs page links names without a version. - The free-threaded store's names carry no version: `/opt/wheels-free-threaded`, `PYTHON_WHEELS_FREE_THREADED`, and on Windows `C:\runtime\wheels-free-threaded` under the same variable. - Windows' build side: `Get-PinnedPythonVersion` feeds the free-threaded interpreter's name, the twin tag, the arm64 tvm-ffi wheel tags and the target's `pythonXY.lib`; OpenCV's `FindPythonInterp` stub asks the interpreter for its version; `Test-Container` grades the pin and fails without one. - The riscv64 uv cache seed, the app bundle's python-build-standalone, and the hub's own workflows, which read X.Y from `versions.env`. - Renovate reports every new release, a minor included, and its rule names no version. **What still needs a look:** a consumer's lock needs wheels (or buildable sdists) for the new tag, since only the image's own wheels come from the chain; PyPI packages whose `requires-python` stops below the new version (pyhailort says `<3.14`). On Windows, two pin sets stay version-bound: - the arm64 bundle's three native pytest wheels (`PYTEST_WINDOWS_ARM64_CHARSET_NORMALIZER/COVERAGE/MARKUPSAFE`), which `Build.Arm64WheelStack.Tests.ps1` names as soon as `PYTHON_VERSION` moves; - the arm64 torch stack (`TORCH_WINDOWS_ARM64_*`), cp313 until PyTorch ships newer `win_arm64` wheels. `ai-edge-litert` has no wheel for a new tag until the chain builds it from source (BACKLOG CON91); until then the Windows app venvs go without it. ## Which Linux image **`:latest`, on every architecture.** It is a multi-arch index — `linux/amd64`, `linux/arm64` and `linux/riscv64`, all children present as of 2026-08-12 — so docker resolves the right platform per runner and the tag does not have to be spelled per lane. The arch-suffixed tags (`:latest-amd64`, `-arm64`, `-riscv64`) exist, but treat them as an implementation detail of the image build. A consumer that names one is pinning itself to a single architecture. > Between 2026-08-04 and 2026-08-12 the index listed **amd64 only**, and lanes > had to name `:latest-cross-arm64` explicitly or `docker pull` on an arm runner > died with "no matching manifest for linux/arm64/v8 in the manifest list > entries". If you find such a workaround still in place, it is stale — remove > it rather than copying it. The same index was named `:latest-cross` until 2026-09-22; that name is retired and no release publishes it. Its registry tags stay until the conditions in [`AGENTS.md` § Image and tag naming](../AGENTS.md#image-and-tag-naming-published-tags) are met. Name `:latest`: [`rancher-desktop-linux-containers.md` § The image: always `:latest`](rancher-desktop-linux-containers.md#the-image-always-latest).