Skip to content

Decision record · ADR

ADR-0036: Unify measurement data-asset preparation without unifying dataset semantics

Status: accepted | Date: 2026-08-07

References: ADR-0003, ADR-0017, ADR-0024, ADR-0035, RFC-0004:C-MEASUREMENT-DATA-ASSETS, RFC-0005:C-MEASUREMENT-DATA-ASSET-EVIDENCE, RFC-0009:C-OPERATION-BUDGETS

InferLab already consumes two release-owned measurement engines with different dataset authorities. lm-eval derives dataset selection from an effective task and commonly delegates remote loading to Hugging Face. AIPerf supports synthetic, file, and registered public datasets and owns additional tokenized or memory-mapped representations. InferLab also owns a digest-addressed cache for release-qualified Bench snapshots. These paths can refer to related source content, but they do not identify the same representation or support the same invalidation rules.

The current product provides strong immutable acquisition for release-qualified Bench and AgentX sources, while task-owned lm-eval sources and native AIPerf public sources can rely on mutable aliases or tool caches whose resolved identity is not consistently recorded. Dataset loading can also occur only after a recipe has started serving processes, spending GPU time on network and preparation work that does not require an inference endpoint.

The design is constrained by ADR-0003, which isolates lm-eval and AIPerf in separate release-owned environments; ADR-0017, which keeps serving Bench request sources closed rather than exposing AIPerf’s configuration language; ADR-0024, which freezes qualified SPEED-Bench inputs before traffic; and ADR-0035, which delegates AgentX topology and scheduling to AIPerf while retaining strict source qualification. The effective task or Bench source remains the selection authority, Rust retains lifecycle and record ownership, and external tools retain their independently changing dataset and derived-cache semantics.

A common boundary is nevertheless justified because Eval and Bench share one independently testable invariant: before recipe-owned serving resources are occupied, InferLab can settle either a closed, integrity-checked source binding or an honest opaque-source classification, and the record can distinguish that source outcome from consumer-specific materialization. The boundary does not pretend that every dynamic task exposes enough information to pre-acquire its full closure.

Select a shared source-preparation manifest within RFC-0004:C-MEASUREMENT-DATA-ASSETS and RFC-0005:C-MEASUREMENT-DATA-ASSET-EVIDENCE. InferLab owns each preparation attempt from the effective source selection to one of two honest outcomes: a locally available, integrity-checked closed source with the strongest supported immutable binding, or an explicitly opaque and non-reproducible task/operator source when its owning interface cannot expose or bind the full closure. Release-qualified sources retain fail-closed behavior and cannot use the opaque outcome.

Recipe source preparation runs before serving launch. It is a distinct, intentionally unbounded but operator-interruptible operation under RFC-0009:C-OPERATION-BUDGETS. A manual Bench uses the same source boundary before its first request without acquiring lifecycle authority over the existing target server.

Consumer materialization remains with its current measurement owner. Eval task loading and AgentX native loading are case-owned and consume their case budget. Independent-request and linear-session frozen populations are owned once by the Bench child before its first case; their separate pre-case materialization operation is unbounded and interruptible, and no arbitrary case pays for shared work.

Source-specific resolution and acquisition semantics remain delegated to the authority selected by the measurement: the applicable release-pinned Hugging Face acquisition runtime for Hugging Face-backed content, InferLab’s verified content-addressed path for release-catalog URL snapshots, and an owning resolver plus immutable prepared snapshot when a local closure can be enumerated. An opaque task/operator source can defer further file or remote access to its consumer and records that limitation. A preparation attempt relates the declared selection, acquired closure or missing closure, physical cache observations, and later materialization without equating their identities.

Pass common explicit Hugging Face cache bindings to the isolated Eval and Bench runtimes and delegate unset-path derivation to each pinned runtime. Hugging Face source and dataset stores, lm-eval and AIPerf derived stores, and InferLab’s release-source store stay independently owned.

The manifest is represented once at recipe scope for shared source preparation, with workload children referring to it. Materialization evidence follows its owner: case scope for Eval and AgentX, Bench-child scope for request/session frozen populations. A standalone Bench owns both layers in its workload record. This decision adds neither a general dataset schema nor a top-level dataset-management command, and it does not open AIPerf’s source configuration language as an InferLab public surface.

Positive: release-qualified and otherwise closable source acquisition and integrity failures become visible before recipe-owned GPU processes start; preparation and measurement time remain separately attributable; records identify a complete immutable closure when one can actually be bound and otherwise explain the opaque non-reproducible conclusion; common cache bindings permit source-byte reuse without duplicating parser logic; and task, catalog, AIPerf scheduling, Bench-wide population, and case authorities remain intact.

Negative: the measurement integrations need a narrowly scoped preparation exchange that can report source, closure, cache, and identity facts without taking over consumer materialization. The same owning loader may observe a source during preparation and again during its owning materialization, relying on cache reuse. Opaque sources can still perform network or file acquisition after serving starts and cannot provide the resource-occupancy or reproducibility guarantee of a closed source. Separate physical stores continue to consume disk and retain independent cleanup behavior. Preparing every selected closable source before serving can download Bench data that a failed Eval gate later makes unnecessary.

Neutral: release-qualified sources fail when their complete expected immutable closure cannot be verified. Eval and AgentX materialization consume their case budgets, while Bench-wide request/session materialization has its own unbounded interruptible pre-case lifetime. Supporting user-authored Bench files, additional AIPerf formats, streaming datasets, or a public prefetch command still requires separate product evidence.

Let InferLab acquire, parse, and materialize every measurement dataset (rejected)

Section titled “Let InferLab acquire, parse, and materialize every measurement dataset (rejected)”
  • Pros: One implementation could expose uniform progress, paths, and digests, InferLab could control every byte before measurement
  • Cons: Duplicates Hugging Face, lm-eval, and AIPerf loaders and cache protocols, Makes InferLab responsible for fast-moving task, multimodal, trace, and streaming semantics
  • Rejected because: The apparent uniformity would create a second dataset authority and recreate external-tool behavior that InferLab deliberately consumes through versioned measurement boundaries.

Leave acquisition and provenance entirely to lm-eval and AIPerf (rejected)

Section titled “Leave acquisition and provenance entirely to lm-eval and AIPerf (rejected)”
  • Pros: Keeps the control plane small, Uses each tool exactly as its upstream authors intended
  • Cons: Allows downloads to begin after serving resources are occupied, Leaves immutable source identity, cache outcome, and cross-tool evidence inconsistent
  • Rejected because: Command and artifact capture alone cannot establish whether two runs consumed the same source or keep dataset acquisition outside the measured serving lifecycle.

Move all source and derived dataset bytes into one InferLab physical cache (rejected)

Section titled “Move all source and derived dataset bytes into one InferLab physical cache (rejected)”
  • Pros: One directory could appear easier to inspect and garbage-collect, Identical upstream downloads might be deduplicated
  • Cons: Conflates source snapshots, Hugging Face Arrow data, AIPerf mmap inputs, and frozen request populations, Requires InferLab to depend on private derived-cache layouts and invalidation rules
  • Rejected because: The stores represent different identities and ownership boundaries; physical consolidation would turn cache compatibility into an unsupported product contract.

Own one InferLab preparation manifest while delegating acquisition and materialization to their existing authorities (accepted)

Section titled “Own one InferLab preparation manifest while delegating acquisition and materialization to their existing authorities (accepted)”
  • Pros: Provides one preparation lifecycle and reproducibility conclusion across Eval and Bench, Retains lm-eval, AIPerf, Hugging Face, and release catalogs as the authorities for their existing semantics, Allows shared source caches without coupling consumer-specific derived representations
  • Cons: Requires both measurement runtimes to expose compatible preparation evidence, Some task-owned sources can remain explicitly non-reproducible when their consumer cannot bind an immutable identity

Require immutable binding for every measurement source (rejected)

Section titled “Require immutable binding for every measurement source (rejected)”
  • Pros: Provides one simple fail-closed reproducibility threshold, Prevents any mutable source from entering a completed experiment
  • Cons: Disables otherwise valid custom lm-eval tasks and operator sources whose upstream interface cannot bind an immutable revision, Treats an honestly observable reproducibility limitation as absence of measurement support
  • Rejected because: Release catalogs are qualified to promise strict immutable reproduction, while external task and operator sources need an honest capability boundary rather than unconditional disablement.

Prepare each data asset only immediately before its measurement executes (rejected)

Section titled “Prepare each data asset only immediately before its measurement executes (rejected)”
  • Pros: Avoids acquiring Bench data that a failed Eval gate will make unnecessary, Shortens the interval before serving startup
  • Cons: Performs network and preparation work while serving and GPU resources are already occupied, Moves acquisition failure into the middle of a recipe after successful serving launch
  • Rejected because: The demonstrated operator problem is resource occupancy during non-serving preparation, so avoiding speculative downloads does not outweigh separating acquisition failure from serving lifetime.