Decision record · ADR
ADR-0003: Use separate release-owned Eval and AIPerf Bench environments
Status: accepted | Date: 2026-07-01
References: ADR-0001, ADR-0002, RFC-0002:C-WORKSPACE-AUTHORITY, RFC-0004:C-INFERLAB-TOOLCHAIN, RFC-0004:C-MEASUREMENTS, RFC-0006:C-INTEGRATIONS, RFC-0005:C-EVIDENCE
Context
Section titled “Context”The workspace-authority contract in RFC-0002 separates the downstream serving closure from InferLab-owned generic measurement tooling, while ADR-0002 applied that separation only to Eval and deliberately left serving Bench clients in each workspace integration. During the prior design discussion, the project selected a common release-owned Bench engine before implementation and reflected that conclusion in an amendment to the governing RFC. This ADR records the rationale for that choice and fully replaces the earlier decision while preserving its still-valid release-owned Eval rationale.
An x86_64 technical spike with AIPerf 0.10.0 established that the documented headless CLI and file formats can express fixed synthetic input/output lengths, concurrency, request rates, arrival burstiness, and raw artifacts. It also exposed process-group cleanup behavior, a user-site isolation requirement, and an aarch64 source-build dependency. Native aarch64 installation and comparison against a real downstream serving workload remain implementation qualification gates rather than established evidence.
Problem Statement
Section titled “Problem Statement”Framework-native Bench clients duplicate translation and result normalization across integrations, couple a stable InferLab load model to each selected serving workspace, and keep LowerBench in an adapter protocol even though ordinary load generation has no framework-specific lowering requirement. InferLab needs one reusable Bench runtime without perturbing serving dependency resolution or coupling Eval and Bench dependency upgrades.
Constraints
Section titled “Constraints”- The InferLab Measurement Toolchain clause in RFC-0004 requires explicit, release-owned, side-effect-free-at-execution tool installation on glibc Linux x86_64 and aarch64.
- The Eval And Bench Measurements clause keeps case expansion, adaptive search, and prefix-cache reset orchestration in InferLab.
- The Framework Integration Boundary clause leaves serving commands and server-control capabilities in framework integrations while removing ordinary measurement clients.
- The Records And Evidence clause requires native configuration, results, logs, artifacts, and cleanup evidence.
- Pixi remains the resolver and installer for release-owned Python tooling.
- ADR-0001 continues to govern downstream workspace serving environments.
Options Considered
Section titled “Options Considered”We considered separate or combined release environments, one shared or independent release lock, a pinned CLI/file boundary or direct Python imports, and continued framework-native Bench clients.
Decision
Section titled “Decision”We will install separate release-owned Eval and AIPerf Bench Pixi environments from one embedded multi-platform workspace and lock through inferlab toolchain install.
This option was selected because:
- Common ownership: Eval and ordinary serving Bench are InferLab measurement capabilities rather than serving-framework dependencies.
- Dependency isolation: Separate named environments prevent lm-eval and AIPerf from forcing each other’s Python dependency versions and native build requirements.
- Shared release lifecycle: One manifest, lock, installation root, command, and completion identity keep the operator workflow and provenance model small.
- Narrow AIPerf boundary: The runner invokes the release-pinned AIPerf CLI with a generated configuration and consumes output artifacts instead of importing private package APIs.
- Rust authority: Rust remains responsible for case expansion, adaptive search, prefix-cache reset, timeout, interruption, process-group cleanup, artifact allocation, and records.
The Rust-to-runner boundary is a versioned typed request/result file pair. The runner-to-AIPerf boundary is the exact CLI, generated configuration, and artifact formats qualified for the pinned AIPerf release. Framework integrations expose serving commands and server-specific control capabilities but do not lower or normalize ordinary Bench operations.
Implementation Notes
Section titled “Implementation Notes”- Eval and Bench use separate named Pixi environments under the same release installation root.
- Installation verifies each runner and underlying tool version before publishing completion metadata.
- The Bench environment includes the native compiler closure required by supported aarch64 packages.
- AIPerf configuration and raw outputs are preserved as record artifacts.
- Rust continues to expand concurrency, request-rate, and adaptive cases so adopting AIPerf does not change existing BenchDefinition semantics.
Consequences
Section titled “Consequences”Positive
Section titled “Positive”- Serving workspaces and framework integrations no longer carry generic load-generation dependencies.
- One common engine provides a consistent measurement and evidence contract across serving frameworks.
- Eval and Bench dependency closures can evolve independently.
- One explicit installation command and one release identity remain sufficient for operators.
- Removing LowerBench reduces the framework integration protocol to serving and server-control concerns.
Negative
Section titled “Negative”- Eval and Bench environments duplicate some Python packages and consume more disk than one combined environment. The accepted trade-off is explicit dependency isolation within one installation root.
- Atomic installation and completion mean an AIPerf or compiler failure can block an Eval-only user from completing toolchain installation. Release qualification on both architectures and idempotent repair are required mitigations.
- InferLab must maintain translation against the pinned AIPerf CLI and artifact schema. Each AIPerf upgrade therefore requires explicit fixture and real-workload qualification.
- AIPerf can fail before emitting a summary. The runner and Rust process record must preserve exit, logs, partial artifacts, and bounded TERM-to-KILL cleanup evidence.
- Some framework-native Bench features may not map to the common contract. Server-specific controls remain integration-owned, and unsupported load semantics must be added deliberately rather than passed through.
- A common client removes one source of variance but does not make different servers intrinsically comparable. Records must retain endpoint, serving, and measurement provenance.
- Historical framework-native Bench metrics may not be numerically interchangeable with AIPerf metrics. Qualification records must document the transition rather than imply continuity.
Neutral
Section titled “Neutral”- Workspace-local Bench definitions, Rust case expansion, adaptive search, and prefix-cache reset policy remain part of InferLab.
- ADR-0001 continues to govern serving workspaces.
- ADR-0002 remains historical context after this ADR supersedes it.
Alternatives Considered
Section titled “Alternatives Considered”Install separate release-owned Eval and AIPerf Bench Pixi environments from one embedded multi-platform workspace and lock (accepted)
Section titled “Install separate release-owned Eval and AIPerf Bench Pixi environments from one embedded multi-platform workspace and lock (accepted)”- Pros: Keeps serving workspaces free of generic measurement dependencies, Allows Eval and Bench dependency closures to evolve independently, Preserves one explicit installation command and one release identity
- Cons: Installs two Python environments and duplicates some shared packages
Install Eval and AIPerf together in one release-owned Pixi environment (rejected)
Section titled “Install Eval and AIPerf together in one release-owned Pixi environment (rejected)”- Pros: Minimizes duplicated packages and installation concepts
- Cons: Couples unrelated lm-eval and AIPerf dependency resolution, The qualified closure forced dependency downgrades and makes aarch64 native-build requirements affect Eval
- Rejected because: The modest disk saving does not justify coupling two independent measurement runtimes and their platform-specific build requirements.
Keep framework-native Bench clients in each workspace integration (rejected)
Section titled “Keep framework-native Bench clients in each workspace integration (rejected)”- Pros: Uses benchmark commands shipped with each serving framework
- Cons: Makes comparable Bench behavior and metrics integration-specific, Keeps generic load generation inside serving dependency closures, Requires the LowerBench adapter operation and duplicated normalization
- Rejected because: Serving frameworks should own server lowering and capabilities, not InferLab’s stable load-generation and evidence contract.
Install Eval and Bench as independent release-owned Pixi workspaces with separate locks (rejected)
Section titled “Install Eval and Bench as independent release-owned Pixi workspaces with separate locks (rejected)”- Pros: Allows each runtime to be installed, qualified, and updated independently
- Cons: Creates two embedded package authorities and two installation identities, Makes one release command coordinate separate lock lifecycles
- Rejected because: Eval and Bench need dependency isolation, but separate named environments already provide it without multiplying release manifests, locks, and completion roots.
Import AIPerf Python APIs directly from the InferLab Bench runner (rejected)
Section titled “Import AIPerf Python APIs directly from the InferLab Bench runner (rejected)”- Pros: Could avoid a subprocess and parse in-memory objects
- Cons: AIPerf does not document its Python package API as a stable integration boundary, Tightly couples the runner to private modules and internal object schemas
- Rejected because: The release-pinned CLI, configuration, and artifact formats provide a narrower boundary that can be qualified without depending on private Python APIs.