Skip to content

Decision record · ADR

ADR-0024: Stage SPEED-Bench as a frozen single-request population

Status: accepted | Date: 2026-07-31

References: RFC-0004, RFC-0005, ADR-0016, ADR-0017

SemiAnalysis InferenceX reproduction needs representative speculative-decoding workloads, while InferLab keeps AIPerf as the common serving measurement engine. SPEED-Bench source rows can contain several user turns. The release-pinned AIPerf loader treats all messages as one multi-turn session by default, whereas the InferenceX and SGLang benchmark path uses only the first user turn. InferLab’s established Bench population model assigns one transport request to each frozen population entry, including warmup, request counts, SLO evaluation, and evidence. Adopting the AIPerf default immediately would therefore change more than dataset selection: it would introduce session scheduling, dependent turns, and a second request-counting unit.

We will stage SPEED-Bench through the existing frozen request-population boundary, materializing the first user turn as one independent request. This direction wins because it reproduces the current InferenceX workload without silently changing InferLab’s unit of load, keeps AIPerf as the shared measurement engine, and leaves full multi-turn support visible as a separate product expansion instead of an accidental dataset side effect.

The initial SPEED-Bench capability can share static and adaptive load, warmup, SLO, timeout, and record behavior with other request sources. Using one AIPerf boundary keeps client-side measurement semantics consistent, while backend qualification, request representation, and framework behavior still bound result comparability. The initial capability does not measure conversational dependency, KV growth across turns, or session-level scheduling. RFC-0004 and RFC-0005 remain the authorities that must define those session and evidence semantics before a later implementation enables the loader’s multi-turn behavior. Keeping that expansion explicit avoids treating first-turn materialization as the final dataset model.

Stage SPEED-Bench as first-turn entries in the existing frozen request population (accepted)

Section titled “Stage SPEED-Bench as first-turn entries in the existing frozen request population (accepted)”
  • Pros: Matches the demonstrated InferenceX reproduction workflow, Reuses existing request-count, warmup, SLO, and evidence semantics, Keeps AIPerf as the measurement engine without importing a framework-native benchmark client
  • Cons: Does not exercise dependent multi-turn session behavior

Adopt AIPerf’s full multi-turn SPEED-Bench behavior now (rejected)

Section titled “Adopt AIPerf’s full multi-turn SPEED-Bench behavior now (rejected)”
  • Pros: Preserves the loader’s default session interpretation, Exercises conversation-history growth and dependent turn scheduling
  • Cons: Changes one entry into a variable number of transport requests, Requires new session, warmup, load, SLO, and evidence semantics before the demonstrated reproduction need can ship
  • Rejected because: The demonstrated reproduction target is single-request, and session semantics require a separate product contract rather than being inherited from a loader default.

Invoke each serving framework’s native SPEED-Bench client (rejected)

Section titled “Invoke each serving framework’s native SPEED-Bench client (rejected)”
  • Pros: Closely follows each upstream benchmark command
  • Cons: Reintroduces client and metric differences that AIPerf is intended to remove, Creates framework-specific benchmark execution outside the shared Bench boundary
  • Rejected because: It would violate the shared AIPerf measurement boundary and make cross-framework results depend on different native clients.