Skip to content

Decision record · ADR

ADR-0002: Install Eval as an InferLab-owned versioned toolchain

Status: superseded | Date: 2026-06-30 Superseded by: ADR-0003

References: RFC-0004, ADR-0001

RFC-0002:C-WORKSPACE-AUTHORITY makes the downstream workspace authoritative for the serving source and package closure. Generic correctness smoke and lm-eval are InferLab capabilities, not serving-framework dependencies, so placing them in that closure changes framework resolution, repeats a large CPU-only installation in every workspace, and makes the same recipe depend on incidental workspace packaging.

ADR-0001 remains correct for workspace serving environments and their difficult editable-package lock bootstrap. This decision addresses a separate owner and lifecycle: the Eval runtime shipped with InferLab.

  • Ordinary dry-run and recipe execution cannot install or update packages.
  • The initial release targets glibc Linux on x86_64 and aarch64.
  • Pixi remains the resolver and installer for Python dependencies.
  • Installed Pixi prefixes cannot be relocated because their metadata and entry points contain absolute paths.
  • A direct runner artifact must have an identity independent of a release-local path entry in a Pixi lock.
  • Framework-native serving Bench clients remain in their selected workspace serving environment.

We will install one InferLab-owned Eval toolchain per InferLab version and platform through inferlab toolchain install.

The InferLab release embeds one CPU-only Pixi manifest and multi-platform lock plus a direct Eval runner artifact. The runner artifact is independently content-digested rather than treated as identified by a local-path lock entry. Installation writes these release inputs directly into the final XDG version-and-platform directory, runs a locked Pixi install there, verifies the runner handshake and installed lm-eval version, and writes completion metadata last.

Dry-run and recipe execution only verify completion metadata and required files. They never install, repair, update, or select another version. Missing or incomplete state fails with the exact explicit install action.

Rust invokes the direct runner with typed Eval request and result files and retains process timeout, cancellation, artifact placement, validation, and record ownership. There is no LowerEval adapter operation. Framework adapters continue to lower Serve and framework-native Bench operations inside the serving workspace.

  • Experiment workspaces declare only their serving closure and framework integrations.
  • One installed Eval runtime is reused by all workspaces using the same InferLab version and platform.
  • Eval provenance is explicit through manifest, lock, runner, lm-eval, platform, and InferLab identities.
  • The direct runner removes a pass-through adapter protocol operation.
  • Users must run one explicit install command before the first built-in Eval.
  • Multiple InferLab versions can consume substantial disk space until users remove old directories manually.
  • Installer correctness depends on Pixi locked-install behavior and native qualification on both supported architectures.
  • ADR-0001 continues to govern workspace serving manifests, locks, setup, and no-sync execution.
  • Framework-native Bench clients remain workspace-selected because they track the serving framework rather than InferLab.
  • Update, garbage collection, and broader toolchain management remain outside the initial design.

Install one release-owned Eval toolchain per InferLab version and platform through an explicit CLI command (accepted)

Section titled “Install one release-owned Eval toolchain per InferLab version and platform through an explicit CLI command (accepted)”
  • Pros: Keeps generic Eval dependencies out of serving environments, Makes the Eval runtime identity explicit and reusable across workspaces, Allows direct runner invocation without a framework adapter lowering hop
  • Cons: Adds a second versioned runtime input and consumes user data-directory storage, Requires an explicit Pixi-backed install before Eval recipes run

Install lm-eval and an Eval adapter in every experiment workspace serving environment (rejected)

Section titled “Install lm-eval and an Eval adapter in every experiment workspace serving environment (rejected)”
  • Pros: Uses the existing workspace Pixi install path
  • Cons: Perturbs framework dependency resolution and duplicates a large CPU-only dependency closure, Makes generic Eval availability depend on each workspace
  • Rejected because: Generic Eval is an InferLab capability and must not contaminate or be repeatedly declared by serving workspaces.

Materialize a recipe-specific Eval capsule dynamically at execution time (rejected)

Section titled “Materialize a recipe-specific Eval capsule dynamically at execution time (rejected)”
  • Pros: Could vary Eval dependencies per recipe
  • Cons: Restores generated environment identities, cache policy, and package mutation during execution, Makes dry-run and run responsible for dependency materialization
  • Rejected because: Recipes do not require dynamic Eval package closures, and ordinary execution must remain side-effect free before the experiment starts.