Skip to content

Decision record · ADR

ADR-0072: Separate generated Bench prefix reuse by namespace instead of resetting the server cache

Status: accepted | Date: 2026-11-06

References: RFC-0004, ADR-0034, ADR-0041, ADR-0071, RFC-0003, RFC-0005, RFC-0009

A prefix-sharing Bench answers one question: given the reuse opportunities its workload defines, how much did the server actually reuse, and what did that reuse buy? Its metrics — prompt-cache-read tokens, available_miss_ratio, available_reuse_efficiency, and the time-to-first-token gain — and every cross-configuration comparison built on them hold only when the reuse opportunities come from the measurement alone. Any hit on content written earlier — by a previous case, by warmup, by an earlier Bench or run, or by another user of a shared tier — is contamination.

Today a cold or primed start removes that contamination by resetting the server’s prefix cache through the reset capability each integration declares (RFC-0004:C-BENCH-CACHE-STATE). The reset changes state InferLab does not own. Planning rejects a cold or primed start on a server without one; a prefill/decode server must reset every target; declined resets need a retry policy; and an integration that cannot clear its store, as SGLang with a storage backend cannot, declares no reset at all. With a KV tier on a shared file system (ADR-0071), a reset also clears other users’ entries, and other users can clear ours; InferLab neither guards nor detects either, by the operator’s decision of 2026-11-05.

Generated populations are content InferLab owns. A Bench’s population is generated once from its seed. Every case takes its slice from index zero: warmup consumes the leading entries and profiling the following ones. So cases replay byte-identical prompts — one case’s warmup entries can be another’s profiling entries — and every run with the same seed repeats them; all entries of one prefix group also share one seed-derived canonical prefix (RFC-0004:C-BENCH-PREFIX-GEOMETRY). Only the reset keeps one case, Bench, or run from warming the next, with or without prefix sharing.

AIPerf’s AgentX replay already avoids resets this way (AIPerf src/aiperf/timing/strategies/cache_bust.py): it inserts a per-run, per-trajectory [rid:…] marker at the start of the first user message, so trajectories and runs never share conversation content while the system prompt and tool definitions, identical across production sessions, stay shared on purpose.

For every request population InferLab generates — random and random_mixture, with or without prefix_sharing, prefix groups, or shared_system_content — InferLab separates reuse by namespace instead of resetting the server.

What the namespace replaces. The Bench seed alone keeps determining each entry’s selected input and output targets, mixture shape, resolved shared-prefix length, and prefix group, so geometry, length distribution, and group structure are the same in every run. The namespace replaces the seed only in the draws that choose content: canonical prefix streams, unique suffixes, corpus offsets, shared system content, and the seed InferLab gives AIPerf for synthetic image decoration.

How it is derived. A cold or primed phase’s namespace is derived from a nonce drawn once per Bench execution, the Bench seed, the case id, and the phase, so no two phases executed in one run — across all its Benches, including one Bench run twice — share a namespace. Warmup and profiling are separate phases; primed conditioning belongs to the profiling phase it prepares. The nonce is drawn when the execution is resolved and carried in the resolved plan, so dry-run, execution, and the record show the same namespaces. Each phase’s namespace is recorded; it lets a reader regenerate any recorded prompt, and no setting pins it as an input. A manual Bench invocation is its own execution with its own nonce.

Cache starts.

  • cold: nothing the window can reuse exists when it opens. No reset is issued.
  • primed: conditioning runs in the profiling namespace, followed by the store’s publication wait when one applies. No reset is issued.
  • uncontrolled: the namespace is derived from the seed alone, exactly today’s content, so runs and cases share content; it remains the one start whose reuse may include history, for a developer who means to measure that.

Group prefixes are inside the namespace. A generated population is meant to be reproducible independently of history: if group prefixes were shared across runs, the first run of a configuration would be cold and the second warm. A developer who wants a resident system prompt asks for it with primed.

Content under different namespaces shares no prefix beyond incidental framing, such as a chat template’s fixed tokens. With a corpus (ADR-0041), the namespace selects the canonical and unique slice offsets, replacing “the Bench seed and entry index alone”; an overlap between namespaces falls back to ADR-0041’s incidental natural reuse, which stays unmeasured, and a corpus too small to make collisions improbable cannot give cold its meaning. Operator directory images are operator content and keep their reuse unmeasured in the same way.

cold therefore changes from “empty cache” to “no reusable content”: the cache may hold unrelated entries. No empty start is added now; when a concrete capacity or eviction workflow needs a physically empty cache, empty is a reset plus a namespace.

Replay and dataset sources keep resetting: their content is the operator’s, and changing it would change the workload. AgentX keeps AIPerf’s own cache bust.

  • A generated cold or primed Bench no longer depends on a prefix-cache reset, so it runs on servers whose integration declares none or declines it, on prefill/decode servers without per-target resets, and on shared tiers whose other users’ entries it neither clears nor needs cleared.
  • Replay and dataset Benches keep every reset dependency and the shared-tier interference this decision removes for generated sources. The reset capability, per-target resets, and the declined-reset retry remain for them and a future empty start, and planning’s rejection of a server without a reset applies only to them.
  • Without a reset, a cold window runs in a cache that may be full of stale entries from earlier cases, Benches, runs, invocations, and tier users. Its own reuse is independent of history only while the framework or tier evicts stale entries before the window’s own; measurements near cache capacity can differ by history and need empty.
  • A run’s prompts are no longer byte-identical to another run’s; reproducibility moves from literal prompts to the recorded seed and namespaces, while geometry, length distribution, and prefix-group structure stay identical.
  • Population generation moves from once per Bench to once per case and phase, and a materialized population can no longer be reused across runs. For long prompts this multiplies preparation time. Measuring it, and deciding whether it runs inside the case budget, belong to the implementation; whether a short position-zero marker, as AgentX uses, replaces regeneration — at the price of accounting its tokens in the prefix geometry — is left open until that measurement exists.
  • Amendments the implementation carries:
    • RFC-0004:C-BENCH-CACHE-STATE: the meanings of cold and primed for generated sources, the reset requirement narrowed to replay and dataset, and the reuse opportunity re-anchored from “sent after the case’s prefix-cache reset” to “sent in the case’s profiling namespace, conditioning included”;
    • RFC-0004:C-BENCH-PREFIX-GEOMETRY: content draws keyed by namespace, selection draws by the seed;
    • RFC-0003:C-SERVE-KV-STORE: the rule that a reset clears the store matters only for replay and dataset;
    • RFC-0005:C-EVIDENCE: the run nonce and each phase’s namespace, and population evidence per phase instead of one population digest per Bench with slice offsets;
    • RFC-0009:C-MEASUREMENT-CASE-BUDGETS, if per-phase generation moves into the case;
    • ADR-0071’s reliance on the reset for a cold tier start holds only for replay and dataset sources.

Separate generated content by a per-run, per-case, per-phase namespace with group prefixes inside it, and keep resets for replay and dataset sources (accepted)

Section titled “Separate generated content by a per-run, per-case, per-phase namespace with group prefixes inside it, and keep resets for replay and dataset sources (accepted)”
  • Pros: Removes contamination by construction, without touching state InferLab does not own, Works where no reset exists, is declined, or would clear a shared tier’s other users
  • Cons: Population generation multiplies, and long-prompt preparation grows, cold no longer means an empty cache, Near cache capacity, stale entries left by history can evict the window’s own, A small corpus can collide across namespaces

Keep resetting the server cache before every cold and primed case (rejected)

Section titled “Keep resetting the server cache before every cold and primed case (rejected)”
  • Pros: Literal prompts stay identical across runs, and cold keeps meaning an empty cache
  • Cons: Depends on every integration’s reset, per-target resets on prefill/decode servers, and a retry policy for declined resets, On a shared tier it clears other users’ entries and is undone by theirs, which InferLab neither guards nor detects
  • Rejected because: It controls reuse by changing state InferLab does not own, where changing the content InferLab does own removes the same contamination without those dependencies

Namespace only each entry’s unique content and share group prefixes across cases and runs, as AgentX shares the system prompt (rejected)

Section titled “Namespace only each entry’s unique content and share group prefixes across cases and runs, as AgentX shares the system prompt (rejected)”
  • Pros: Matches production, where a system prompt stays resident across sessions
  • Cons: The first run of a configuration is cold and later runs warm, so comparisons between runs depend on history
  • Rejected because: A generated population must be reproducible independently of history; a resident shared prefix is what primed expresses explicitly

Give uncontrolled a per-run namespace too, differing from cold only in issuing no preparation (rejected)

Section titled “Give uncontrolled a per-run namespace too, differing from cold only in issuing no preparation (rejected)”
  • Pros: No start would share content across runs
  • Cons: uncontrolled would become cold without preparation, and no start would remain for measuring reuse across history
  • Rejected because: It erases the only start meant to measure reuse that includes history

Add an empty start that resets and namespaces now (rejected)

Section titled “Add an empty start that resets and namespaces now (rejected)”
  • Pros: Capacity and eviction experiments could start from a physically empty cache
  • Cons: Adds a public start no current workflow needs
  • Rejected because: No concrete capacity or eviction workflow needs it yet; when one does, empty is a reset plus a namespace, and the reset remains available for replay