Decision record · ADR
ADR-0029: Keep structured chat transport while targeting exact synthetic prompt lengths
Status: superseded | Date: 2026-07-31 Superseded by: ADR-0034
References: RFC-0004, RFC-0005, ADR-0028
Context
Section titled “Context”Serving Bench keeps structured messages on the chat-completions route so the model server remains the authority that actually applies its chat template. ADR-0028 also made synthetic input_tokens an exact pre-template content length, but that means a nominal 8192-token request reaches the model with additional template tokens. The reproduction workflow needs synthetic fixed and distributed ISL targets to be as close as possible to the backend-visible prompt length. InferenceX estimates one constant template overhead from a dummy message, but complete template tokenization can vary with roles, kwargs, generation markers, and tokenizer boundary effects, so subtracting that constant cannot establish exactness. InferLab already resolves the model tokenizer and materializes every synthetic population before execution; the decision is whether that preparation may project the template for length planning without taking over transport rendering.
Decision
Section titled “Decision”Serving Bench will continue to transmit structured messages through the pinned AIPerf chat endpoint, and the model server will remain the authority that actually renders them. For random and weighted-random-mixture sources, input_tokens will denote the desired complete profiling prompt-token length. Population preparation will use the resolved model tokenizer, an explicit string request_body.chat_template when present or otherwise the tokenizer default, and declared chat_template_kwargs to tokenize the complete one-request chat projection with the generation marker. It will adjust only generated synthetic message content and will claim exact targeting only after the complete projection equals the selected target; it will not estimate or subtract a fixed overhead. Without prefix sharing it adjusts the one user message. With prefix sharing it keeps the shared system content at floor(input_tokens * shared_prefix_ratio) tokens and adjusts only the independently generated user suffix; the ratio therefore describes a shared content budget relative to the desired complete prompt target, not the realized pre-template content ratio. If the effective projection cannot be determined or an exact synthetic content length cannot be constructed, preparation will keep the unadjusted selected content length or suffix budget and record a fallback reason instead of failing or silently claiming exactness. Dataset inputs and linear sessions are not rewritten because their content is source-owned and later session histories depend on live model output. Backend-reported usage.prompt_tokens remains the authority for the prompt length actually executed.
Consequences
Section titled “Consequences”Synthetic random workloads default to the closest defensible interpretation of a requested profiling ISL while retaining the same structured chat traffic used in production. Population and case evidence must distinguish the selected prompt target, realized pre-template content length, locally predicted prompt length, exact-targeted versus fallback entries, fallback reasons, and backend-observed prompt-token metrics. A locally exact projection is not relabeled as backend observation; disagreement remains visible as configuration drift. Templates that are absent, backend-private, invalid for local projection, or unable to produce the requested total length lead to recorded fallback without altering the unadjusted content generator. The AIPerf-native warmup marker is not part of the profiling target projection and may increase warmup prompt_tokens; warmup evidence must not claim it attained the profiling target. Real datasets and agentic sessions preserve their source semantics and report observed prompt lengths rather than being padded or truncated.
Alternatives Considered
Section titled “Alternatives Considered”Project the complete chat template only to target synthetic prompt length (accepted)
Section titled “Project the complete chat template only to target synthetic prompt length (accepted)”- Pros: Makes fixed and distributed synthetic ISL targets exact when the effective tokenizer and template are locally determinable, Keeps structured chat transport and server-side template application unchanged
- Cons: Requires explicit fallback and drift evidence because a server may use an unseen template override
Subtract one constant template overhead estimated from a dummy message (rejected)
Section titled “Subtract one constant template overhead estimated from a dummy message (rejected)”- Pros: Is simple and resembles the current InferenceX heuristic
- Cons: Cannot guarantee the final token count when template branches or token boundaries change
Render prompts locally and send them through completions (rejected)
Section titled “Render prompts locally and send them through completions (rejected)”- Pros: Makes the flat prompt bytes entirely client-owned
- Cons: Replaces the native chat path and duplicates live multi-turn behavior
Retain exact pre-template content targets and only report observed prompt tokens (rejected)
Section titled “Retain exact pre-template content targets and only report observed prompt tokens (rejected)”- Pros: Keeps the existing meaning simple and never needs local template projection
- Cons: Makes a nominal fixed ISL systematically larger at the model boundary and prevents direct reproduction of backend-visible fixed shapes