Decision record · ADR
ADR-0063: Source SPEED-Bench rows from the release-pinned AIPerf public datasets
Status: accepted | Date: 2026-10-07
References: RFC-0004, RFC-0005, ADR-0061, ADR-0036, ADR-0024
Context
Section titled “Context”InferLab froze raw nvidia/SPEED-Bench rows from a pinned parquet snapshot. Many rows are redistribution placeholders standing in for prompt text whose source dataset does not permit redistribution, so in each 33-prompt sample 9 of 11 qualitative categories carried between 6 and 33 placeholder prompts, and every InferLab SPEED-Bench run in those categories measured acceptance on placeholder text. Separately, the AIPerf integration build InferLab 0.21.0 ships (ADR-0061) rejects the speed_bench_<category> custom dataset format InferLab handed it, so every SPEED-Bench case fails at startup. The pinned AIPerf exposes SPEED-Bench as public datasets whose loader resolves placeholders with the dataset’s own upstream preparation code, checks access to the one gated source it needs, and refuses to serve partially resolved data; its public-dataset loaders share a generic materialize-and-load interface. The same AIPerf build computes per-run acceptance from per-request records, the run summary, or scraped server metrics, but its report CLI identifies a run’s category only from a speed_bench_ dataset selector, which a file dataset cannot carry. The operator decided that InferLab must not add dataset-specific gating or resolution logic, which goes stale as datasets and their sources change, and that existing definitions must keep working.
Decision
Section titled “Decision”A catalog source may name a release-pinned AIPerf public dataset instead of a download location, and the SPEED-Bench sources do. The catalog still pins the source’s content: the digest of the rows that AIPerf build’s public-dataset materialization produces. On a cache miss, source preparation asks that AIPerf build to materialize the dataset through its generic public-dataset interface, so AIPerf owns any upstream resolution, access requirement, and failure message; the control plane then holds the rows to the pinned digest and publishes them to the same digest-addressed cache a download fills, so a cache hit needs neither AIPerf nor the network. InferLab’s unchanged first-turn, input-bound, seeded-sample, and freeze pipeline then runs over the rows, and the frozen population reaches AIPerf as an ordinary file dataset. Acceptance length and rate come from the pinned AIPerf’s own SPEED-Bench sources, per-request records, run summary, and scraped server metrics in that order, labeled with the category InferLab already resolved from the profile, rather than from the report CLI. Because the digest is pinned, release-catalog sources keep ADR-0036’s fail-closed reproduction: rows that differ from the pin fail preparation until a release updates it. The first-turn population ADR-0024 decided is unchanged.
Consequences
Section titled “Consequences”Positive: SPEED-Bench Benches measure real prompts in every category, existing definitions keep their meaning and fields, and placeholder resolution and gated access stay current with each AIPerf release instead of being maintained in InferLab. Every machine either freezes the same rows or fails loudly, as with a downloaded snapshot, and a cache hit needs no network.
Negative: preparation and acceptance depend on internals of the pinned AIPerf build, its public-dataset materialization and its report computation, which a rebuild must requalify. A cache miss needs network access to every upstream source and an account permitted to read the gated source, failing with AIPerf’s message when it cannot. Several upstream sources are unpinned, so a change upstream makes preparation fail on machines without a cached copy until a release updates the pinned digest. Earlier SPEED-Bench results in the nine affected categories measured placeholder text and are not comparable with new ones; every SPEED-Bench population also changes its source identity and frozen digest.
Neutral: RFC-0004:C-BENCH-REQUEST-SOURCES, RFC-0004:C-MEASUREMENT-DATA-ASSETS, RFC-0004:C-MEASUREMENTS, and the RFC-0005 evidence clauses carry the obligations; acceptance values are now unrounded rather than the report CSV’s two decimals.
Alternatives Considered
Section titled “Alternatives Considered”Keep InferLab’s dataset pipeline and read SPEED-Bench rows from the release-pinned AIPerf public-dataset materialization (accepted)
Section titled “Keep InferLab’s dataset pipeline and read SPEED-Bench rows from the release-pinned AIPerf public-dataset materialization (accepted)”- Pros: AIPerf owns placeholder resolution, gated access, and their errors, so InferLab carries no dataset-specific logic, Existing SPEED-Bench definitions keep their meaning: first turn, input bound, seeded sample, frozen digest
- Cons: Couples preparation and acceptance to internals of the pinned AIPerf build
Hand SPEED-Bench Benches to AIPerf’s public datasets directly (rejected)
Section titled “Hand SPEED-Bench Benches to AIPerf’s public datasets directly (rejected)”- Pros: Deletes InferLab’s SPEED-Bench preparation entirely
- Cons: Changes what existing definitions mean: whole conversations instead of first turns, upstream order instead of seeded samples, and no max_input_tokens, Loses InferLab’s frozen population and its digest evidence
Resolve placeholders in InferLab, including a gated-access check (rejected)
Section titled “Resolve placeholders in InferLab, including a gated-access check (rejected)”- Pros: Independent of AIPerf internals
- Cons: Dataset-specific resolution and access logic in InferLab goes stale as the dataset, its sources, or their access terms change
Refuse to run any category that still holds placeholder rows (rejected)
Section titled “Refuse to run any category that still holds placeholder rows (rejected)”- Pros: Smallest change and never measures placeholder text
- Cons: Leaves only 2 of 11 qualitative categories usable
Materialize through AIPerf and record the observed digest without pinning it (rejected)
Section titled “Materialize through AIPerf and record the observed digest without pinning it (rejected)”- Pros: Never fails on upstream drift
- Cons: Two machines or two months can silently freeze different prompts, breaking the reproducibility release catalogs promise under ADR-0036
Add a category option to the integration build’s report CLI (rejected)
Section titled “Add a category option to the integration build’s report CLI (rejected)”- Pros: Keeps acceptance on a CLI surface
- Cons: Adds another fork patch that upstream has not reviewed and extends ADR-0061’s retirement condition