Skip to content

Decision record · ADR

ADR-0038: Make raw Bench capture an explicit diagnostic level and keep it the default

Status: accepted | Date: 2026-08-17

References: RFC-0004, RFC-0005

A paired InferenceX versus InferLab/AIPerf C8 implementation review (same server process, frozen 8K/1K population, C8, 16-request warmup, 80-request profiling) observed 206.12 versus 174.47 output tok/s and established that InferLab requests full per-request raw SSE export for every Bench: for this workload the raw JSONL held 82,080 SSE messages and about 36 MiB. The raw writer is not on the credit-refill critical path, so the retention is a measurement-cost and data-volume issue that is not yet proven to explain the throughput gap. The review’s verified facts also showed that AIPerf 0.12 writes both warmup and profiling records into the normalized per-request records export, and that disabling raw export still produces normalized records and summary while dropping response and payload retention inside the pipeline. InferLab had no way to express a low-observability run: the raw request was unconditional, and warmup plus population-identity reconciliation read the raw artifacts. Session and agentic evidence legitimately depend on raw payloads and responses for content-token and source-coordinate dimensions.

We will make the raw request/response capture an explicit serving-Bench artifact level, performance or diagnostic, governed by RFC-0004:C-BENCH-ARTIFACT-LEVEL, and keep diagnostic as the default. Warmup validation and population-identity reconciliation move to the normalized per-request records under either level. Session-source and agentic-source cases remain accepted at the performance level, with the raw-dependent evidence dimensions recorded as unavailable rather than rejected, per the degradation paragraphs in RFC-0005:C-BENCH-LINEAR-SESSION-EVIDENCE and RFC-0005:C-BENCH-AGENTIC-TRACE-EVIDENCE. The default stays diagnostic because the observed throughput gap is not yet attributed; flipping the default is a separate decision gated on the A/B/C artifact-level ablation the new knob enables.

Operators can run ordinary performance benches without multi-megabyte raw SSE retention and can execute the raw-export and server-metrics ablation without code changes, while diagnostic runs retain today’s full evidence by default. Warmup and population reconciliation no longer depend on raw artifacts, which also removes the interrupted-run raw-shard fallback from the ordinary validation path. The cost is that performance-level session and agentic cases lose content-token, pairwise response-history, and source-coordinate evidence; that loss is recorded per case rather than hidden, and interrupted performance runs have no raw shards to recover. Because the default is unchanged, no existing definition changes behavior, and any later default flip must cite the ablation evidence.

Add an explicit artifact level, keep diagnostic as the default, and degrade session/agentic evidence with recording (accepted)

Section titled “Add an explicit artifact level, keep diagnostic as the default, and degrade session/agentic evidence with recording (accepted)”
  • Pros: Gives operators and the ablation experiment a low-observability run mode without changing any existing definition’s behavior, Moves warmup and population reconciliation onto normalized records that already carry both benchmark phases, Keeps session and agentic sources usable at the performance level with honestly recorded evidence gaps
  • Cons: Performance-level session and agentic cases lose content-token, pairwise-history, and source-coordinate evidence, Interrupted performance runs have no raw shards for post-hoc diagnosis

Make performance the default artifact level immediately (rejected)

Section titled “Make performance the default artifact level immediately (rejected)”
  • Pros: Removes raw retention cost from every ordinary run without requiring operators to opt in
  • Cons: Weakens diagnostic evidence for all existing definitions before the throughput gap is attributed
  • Rejected because: The paired C8 gap is not yet proven to come from raw retention; the default flip must wait for the A/B/C ablation evidence the new level enables.

Reject the performance level for session and agentic sources (rejected)

Section titled “Reject the performance level for session and agentic sources (rejected)”
  • Pros: Keeps every evidence obligation fully satisfiable at every valid combination
  • Cons: Fragments the definition model with a source-dependent validation rule for a purely observational concern
  • Rejected because: Raw dependence is an evidence property, not a validity property; recording the degraded dimensions keeps the combination usable without fabricating evidence.

Keep raw capture unconditional and change nothing (rejected)

Section titled “Keep raw capture unconditional and change nothing (rejected)”
  • Pros: No protocol, reconciliation, or evidence changes
  • Cons: Leaves every Bench paying full SSE retention with no way to run the recommended low-observability ablation
  • Rejected because: The review demonstrated material raw volume on an ordinary long-output workload and recommended an explicit diagnostic mode; doing nothing blocks both the cost relief and the ablation.