Decision record · ADR
ADR-0057: Make locally built image packages path-neutral at build time and verify them with release-pinned image tools
Status: accepted | Date: 2026-10-03
References: RFC-0007, RFC-0004, ADR-0005, ADR-0003
Context
Section titled “Context”Runtime images built by inferlab image build are published publicly. RFC-0007 kept machine-private facts out of generated build-context text and image metadata but exempted locally built binary payloads, which MAY embed paths of the sanitized build tree. An audit of a vLLM runtime image found every compiled extension carrying the builder’s home directory and workspace path in four places: __FILE__ strings in host code and in compressed device code (CUDA fatbins hide them from a plain byte scan), DWARF debug information (vLLM builds RelWithDebInfo by default, 1.3 GB of its 1.9 GB of extensions), and a DT_RPATH naming the workspace’s Pixi environment library directory. The DT_RPATH also precedes LD_LIBRARY_PATH, so a container that sees the builder’s filesystem could load libraries from it. The image additionally held every built wheel twice: installed, and as the /wheelhouse payload copied into a layer (about 650 MB for vLLM). Measured on CUDA 13.2: -ffile-prefix-map passed through NVCC_APPEND_FLAGS as -Xcompiler remaps both host and device __FILE__; only -lineinfo file tables written by ptxas escape it, and the current vLLM build does not use -lineinfo. DT_RPATH is written at link time and no compiler mapping reaches it.
Decision
Section titled “Decision”Package builds map paths at the source, rewrite what mapping cannot reach, and prove the result before a package is cached. Each build runs with prefix maps appended to the activation’s CFLAGS, CXXFLAGS, and NVCC_APPEND_FLAGS (as -Xcompiler options) and, as rustc --remap-path-prefix options, to the rustflags source Cargo honors: CARGO_ENCODED_RUSTFLAGS when the activation sets it, otherwise RUSTFLAGS when set, otherwise CARGO_BUILD_RUSTFLAGS, which merges with a package’s configured build.rustflags where RUSTFLAGS would replace them. The sanitized source tree maps to /opt/inferlab-src, the stack’s Pixi environment prefix maps to the image environment prefix, the effective Cargo home (the activation’s CARGO_HOME, otherwise .cargo under the home directory) maps to /opt/inferlab-cargo, and the workspace root maps to /opt/inferlab-workspace as a fallback, ordered so the most specific map wins. After the build, every ELF library search path naming the environment prefix is rewritten to the image environment prefix with its DT_RPATH or DT_RUNPATH kind preserved, remaining entries naming the workspace, a package-build directory, or the home directory are dropped, and the wheel is repacked with a regenerated RECORD. The finished wheel is then verified against the workspace root, the environment prefix, the home directory, the Cargo home, and every directory InferLab creates for the package build: each member as stored, each compressed ELF section decoded, and the device code extracted from each fatbin. A hit fails the package naming the member, and a compressed section or fatbin that cannot be decoded fails it too; content compressed in other formats inside a member is checked only as stored. The tools come from a new image environment in the InferLab toolchain, pinned by the release lock like the Eval and Bench runtimes: patchelf for search paths, cuda-cuobjdump for device code, and wheel for repacking. An image build, dry-run or not, whose selection includes a package requires that installed environment and fails before a record or builder effect otherwise. Package evidence records the runtime identity, the applied maps, the rewritten and removed search paths, and the verification outcome. Assembly installs wheels from a build-time bind mount of the context’s wheelhouse instead of copying it into a layer; it is part of this decision because the copied wheelhouse doubled every embedded path in the image as well as its size. Whether debug information ships remains the workspace’s build-type choice.
Consequences
Section titled “Consequences”Published images carry no builder home directory, workspace path, environment prefix, or package-build path in host code, device code, debug information, or library search paths, and the guarantee is enforced by verification rather than trusted to every build backend honoring the flags; a backend that ignores them fails loudly instead of leaking. Debug information stays usable and points at stable fixed paths. Every image shrinks by the size of its wheels. Embedded path strings no longer depend on the checkout location. Building image packages now requires inferlab toolchain install, and a toolchain installation gains a small image environment. Verification adds extraction and scan time proportional to package size and has no override, so a false positive blocks the image until the matcher or the build changes; home paths match only at a path-component boundary to keep that rare. A future dependency compiled with -lineinfo, a compiler outside the mapped C, C++, CUDA, and Rust set, a Rust package whose Cargo configuration sets a matching target rustflags table (Cargo then ignores CARGO_BUILD_RUSTFLAGS), or a crate that embeds env!("CARGO_MANIFEST_DIR") or env!("OUT_DIR") as a string, which path remapping does not reach, fails verification until its paths are mapped; that failure is the intended signal. Assembly requires a BuildKit builder for the bind mount, which current Docker uses by default. The package-build and context-generation procedure identities change, so existing wheel caches rebuild once.
Alternatives Considered
Section titled “Alternatives Considered”Map paths at build time, rewrite library search paths afterwards, and verify each finished package including device code, with patchelf, cuobjdump, and wheel from a release-pinned toolchain image environment (accepted)
Section titled “Map paths at build time, rewrite library search paths afterwards, and verify each finished package including device code, with patchelf, cuobjdump, and wheel from a release-pinned toolchain image environment (accepted)”- Pros: Covers host code, device code, debug information, and search paths, measured on the real vLLM build, Verification turns a backend that ignores the flags into a build failure instead of a silent leak, Tool versions are fixed by the release lock, available offline after one install, and enter the build evidence like AIPerf and lm-eval
- Cons: Image package builds now require toolchain install, Search paths are rewritten after the build rather than produced correctly by it
Keep the RFC-0007 exemption and accept build paths in binary payloads (rejected)
Section titled “Keep the RFC-0007 exemption and accept build paths in binary payloads (rejected)”- Pros: No change, and build paths in binaries are common in published packages
- Cons: Published images would expose the builder’s user name and workspace layout in every extension and in user-visible error messages, The DT_RPATH keeps pointing at a live host environment that precedes LD_LIBRARY_PATH
Build packages inside a mount namespace or container that presents the sources and the environment at the image’s fixed paths (rejected)
Section titled “Build packages inside a mount namespace or container that presents the sources and the environment at the image’s fixed paths (rejected)”- Pros: Every embedded path would be correct by construction without per-compiler flags
- Cons: Needs unprivileged user namespaces, which shared GPU hosts often disable, A Pixi environment does not relocate cleanly: activation, shebangs, and package configuration keep its original prefix
Take the tools ad hoc at build time with pixi exec at pinned versions (rejected)
Section titled “Take the tools ad hoc at build time with pixi exec at pinned versions (rejected)”- Pros: No toolchain change
- Cons: Builds need network or a warm cache, and the tool versions live only in a command line without a lock
Require each workspace to declare the tools in its stack’s Pixi environment (rejected)
Section titled “Require each workspace to declare the tools in its stack’s Pixi environment (rejected)”- Pros: Fully offline and locked with the stack
- Cons: Every workspace must add packaging tools to its serving environment, which they do not otherwise need
Strip debug information from built packages by default (rejected)
Section titled “Strip debug information from built packages by default (rejected)”- Pros: Removes debug paths and most of the size in one step
- Cons: Removes crash symbolization that operators may want, and leaves FILE and search paths untouched
Rewrite DWARF paths after linking with debugedit (rejected)
Section titled “Rewrite DWARF paths after linking with debugedit (rejected)”- Pros: Reaches debug information without compiler cooperation
- Cons: Cannot reach FILE strings in host or compressed device code, and adds another tool to the pinned toolchain