Decision record · ADR
ADR-0034: Separate serving Bench prompt authority from prefix-cache conditioning
Status: accepted | Date: 2026-08-05
References: RFC-0004, RFC-0005, ADR-0003, ADR-0017, ADR-0029, ADR-0031
Context
Section titled “Context”Serving Bench currently distinguishes prompt authority from transport. Structured datasets and live histories preserve production chat semantics, while controlled synthetic workloads can own the final prompt token sequence. A locally projected server template can target a synthetic ISL only when the local tokenizer, template, kwargs, generation marker, and server behavior agree; fallback preserves traffic but cannot make the nominal shape exact. Server-rendered framing also creates an implicit shared prefix that is inseparable from a controlled synthetic cache experiment.
Fixed-shape reproduction, variable-ISL prefix curves, and full-prompt reuse instead need the Bench to own the complete prompt presented to the completions tokenizer. Some speculative-decoding workloads still need chat-formatted token distributions, but InferenceX demonstrates that this does not require a chat transport: the template can be frozen before a completion request. Flattening every workload would fail in the opposite direction because source-owned roles, server template behavior, and live multi-turn histories are part of semantic dataset and agentic measurements.
Prefix geometry, cache initial state, and observed cache reads are also different facts. A shared-content ratio does not determine cache hits, which depend on cache contents, block boundaries, eviction, request ordering, routing, and replica topology. The existing reset boolean and AIPerf-native warmup order further mix runtime stabilization with cache conditioning because native warmup may mutate prompts and populate the prefix cache. The design therefore needs one boundary that preserves semantic chat traffic while making controlled token-shape and cache-state experiments reproducible without exposing backend endpoints or promised cache-hit outcomes.
Amended 2026-08-05: requiring every synthetic source to repeat a prompt-authority choice adds ceremony to the ordinary exact-shape workflow without enforcing an additional invariant. Omission can select flat while the resolved execution and record still carry one explicit authority. Defaulting omission to server_chat would instead make an apparently exact input-token declaration depend on an implicit model or server template and possible accounting fallback.
Decision
Section titled “Decision”Select the serving Bench request form from prompt authority, and model cache conditioning independently from prompt construction. Semantic datasets and dependent sessions retain structured messages, the chat-completions route, and server-owned rendering. For random and random_mixture, omission of the prompt table selects flat; an explicit table selects flat, rendered_chat, or server_chat. flat owns final token-shaped text, rendered_chat freezes one model-default or operator-declared template before transport, and server_chat retains structured messages when production server rendering is itself the workload. The first two use completions and can own exact final-prompt shape and prefix geometry; the last uses chat completions and treats final length and reuse as observations even when local projection succeeds.
Keep endpoint paths derived from those semantics rather than exposing an endpoint switch. Treat fixed shared-token length and proportional sharing as two declaration forms that lower to one per-entry shared-prefix token schedule over a canonical shared stream. Treat the cache start as a separate closed experiment choice: uncontrolled state, reset cold state, or reset followed by explicit conditioning. Runtime warmup remains runtime stabilization and precedes any reset or conditioning that defines the profiling cache state. Backend cache-read accounting remains an observed result rather than a requested percentage.
This direction preserves one frozen request-population authority and continues to use AIPerf for load scheduling, transport measurement, phase records, server token counts, and cache metrics. It prevails because the default synthetic workflow gets model-independent, exact final-token geometry without configuration ceremony, while explicit rendered or server chat remains available wherever chat shape, server rendering, message structure, or live history is the workload.
Consequences
Section titled “Consequences”Positive: ordinary synthetic Bench definitions become concise and exact by default: a declared input-token target resolves to a flat completion prompt unless the operator explicitly asks for chat shaping or server rendering. Existing explicit flat, rendered_chat, and server_chat definitions retain their semantics. Fixed and distributed synthetic ISL, zero through full shared-prefix geometry, locally chat-formatted speculative-decoding inputs, cold-cache cases, and explicitly primed cases remain distinguishable experiments. Semantic datasets and agentic sessions continue to exercise the production chat path. Records still separate resolved prompt authority, declared geometry, cache preparation, and observed cache reads.
Negative: configurations written before the explicit 0.9 prompt schema may have relied on omitted prompt configuration to imply chat; under the amended contract, omission on a synthetic source means flat, so preserving that historical behavior requires an explicit server_chat declaration. Prefix sharing still has two mutually exclusive declaration forms, and the reset boolean still needs migration to a cache-start policy. Frozen local rendering makes the Bench responsible for template identity and materialization evidence. Explicit conditioning can be qualified only on integrations and topologies whose reset, routing, and cache-reporting capabilities make the claimed start state defensible.
Neutral: effective prompt authority remains present in resolved execution and records even when selected by the default. AIPerf remains the measurement engine and backend-reported prompt and cache usage remain observational authorities. Server-rendered chat can still exhibit natural or incidental prefix reuse, but it is not relabeled as an exact controlled-prefix experiment. A future AIPerf version may replace release-specific phase coordination without changing the prompt-authority or cache-state decision.
Alternatives Considered
Section titled “Alternatives Considered”Derive request form from prompt authority and model cache start separately (accepted)
Section titled “Derive request form from prompt authority and model cache start separately (accepted)”- Pros: Preserves native chat semantics while giving controlled synthetic experiments exact final-prompt ownership, Separates workload geometry, cache preparation, and observed cache reads
- Cons: Requires a coordinated breaking schema migration and topology-qualified conditioning
Keep every serving Bench request on chat and improve local projection (rejected)
Section titled “Keep every serving Bench request on chat and improve local projection (rejected)”- Pros: Retains one transport family and the current structured population
- Cons: Cannot make the final token sequence or common-prefix boundary benchmark-owned when the server renders it
- Rejected because: Projection can expose drift but cannot establish exact token-shape or cache-prefix authority.
Flatten every serving Bench workload and use completions (rejected)
Section titled “Flatten every serving Bench workload and use completions (rejected)”- Pros: Makes every transmitted prompt locally inspectable
- Cons: Moves source-owned roles and live session rendering into the Bench and stops exercising the production chat path
- Rejected because: Exact prompt ownership is a shape-workload requirement, not a reason to erase semantic chat workloads.
Expose endpoint selection and target cache-hit percentage independently (rejected)
Section titled “Expose endpoint selection and target cache-hit percentage independently (rejected)”- Pros: Appears flexible and maps directly to familiar protocol and metric names
- Cons: Allows incoherent combinations and asks InferLab to promise a backend-dependent outcome
- Rejected because: Operators need reproducible workload causes and observed outcomes, not independent switches that can contradict each other.
Keep prompt authority mandatory for every synthetic source (rejected)
Section titled “Keep prompt authority mandatory for every synthetic source (rejected)”- Pros: Every synthetic source spells the selected authority
- Cons: Repeats a choice for the ordinary exact-shape workflow without enforcing a distinct invariant
- Rejected because: The resolved execution and record can remain explicit while omission selects the unambiguous exact flat authority.
Default omitted synthetic prompts to server-rendered chat (rejected)
Section titled “Default omitted synthetic prompts to server-rendered chat (rejected)”- Pros: Matches the historical transport behavior of synthetic Bench definitions before prompt authority became explicit
- Cons: Makes input-token targets and prefix shape depend on an implicit model or server template and possible accounting fallback
- Rejected because: The default synthetic scalar shape should provide exact model-independent token geometry; production chat behavior remains an explicit workload.