Decision record · ADR
ADR-0041: Corpus-sliced content for the random serving-Bench source
Status: accepted | Date: 2026-08-23
References: RFC-0004:C-BENCH-REQUEST-SOURCES, RFC-0005:C-BENCH-REQUEST-SOURCE-EVIDENCE, ADR-0040
Context
Section titled “Context”The random serving-Bench source synthesizes prompts from hash words (sha256-derived tokens) to get model-independent, byte-exact token geometry under RFC-0004:C-BENCH-REQUEST-SOURCES. That supply is high-entropy unnatural text, and a class of measurements is content-sensitive rather than shape-sensitive: speculative-decoding acceptance rates, MoE expert routing distributions, and similar behaviors measured on hash-word prompts are systematically unrepresentative of natural-language load. The demonstrated driver is acceptance-rate evaluation of speculative decoding on the Specialized Engine and vLLM workspaces, where random remains the right source for everything except content naturalness. Existing alternatives do not close the gap: the dataset catalog is conversation-oriented and closed to operator corpora, and the replay source fixes the population in a file, discarding the random source’s seeded extension, warmup/formal partitioning, and size derivation from case shapes.
Decision
Section titled “Decision”Extend the random serving-Bench request source with an optional corpus declaration: a workspace-relative path to an operator-supplied text corpus with an optional expected SHA-256, content-addressed exactly like the replay source’s population file. When declared, entry content is drawn as exact token-length slices of the corpus token stream at offsets determined by the Bench seed and entry index alone, preserving seed determinism, exact token geometry, selector semantics, prefix-sharing geometry, and the extension invariant that a larger population preserves the same first N entries. Slices are drawn independently and may overlap; that incidental sharing is declared as unmeasured natural reuse, while a declared prefix_sharing keeps its exact controlled geometry with one fixed corpus slice as the shared prefix. A corpus whose token stream cannot serve the largest selected input target fails preparation, and a declared digest mismatch fails before any transport request. random_mixture and the catalog dataset boundary are unchanged.
Consequences
Section titled “Consequences”Positive: content-sensitive measurements such as speculative-decoding acceptance rates and MoE routing distributions run on natural-language prompts with the full random-source tooling (seeded extension, warmup/formal partitioning, prefix geometry, SLO and evidence paths); operator corpora enter the schema through the same digest-pinned carve-out as replay instead of beside it.
Negative: incidental slice overlap adds an uncontrolled prefix-reuse component that operators must not mistake for declared shared-prefix geometry; corpus selection and cleaning are unvalidated operator responsibilities, and a stale or edited corpus silently changes the workload unless its digest is pinned; two content-pinned local-file uses now exist, so the carve-out needs one general statement rather than per-feature exceptions.
Neutral: the synthetic hash-word supply remains the default and its determinism claims are unchanged; the Bench runner remains the population authority and now owns corpus tokenization and slicing; a broadly reusable corpus can still be promoted into the release catalog later without invalidating workspace-local corpora.
Alternatives Considered
Section titled “Alternatives Considered”An optional corpus declaration on the random source (accepted)
Section titled “An optional corpus declaration on the random source (accepted)”- Pros: Keeps every random-source semantic (seed determinism, exact token geometry, selectors, prefix sharing, extension stability) and replaces only the content supply, Corpus identity is content-addressed with the same digest-pinned local-file carve-out the replay source established, so no new trust model, Population sizing, warmup/formal partitioning, and seed-stable extension remain InferLab-owned instead of moving to operator scripts
- Cons: Independent slice draws can overlap, introducing unmeasured incidental sharing that must be declared rather than controlled, Corpus quality and representativeness become the operator’s responsibility, with only length and digest validated
A separate corpus_random request-source kind (rejected)
Section titled “A separate corpus_random request-source kind (rejected)”- Pros: Keeps the hash-word random source untouched
- Cons: Duplicates the entire random contract (selectors, seed sequence, extension invariant, prefix geometry, TPOT rules) for a one-field difference, and the two kinds would drift
- Rejected because: Content supply is orthogonal to population semantics; a new kind forks a contract that should evolve once.
Pre-slice corpora into replay population files (rejected)
Section titled “Pre-slice corpora into replay population files (rejected)”- Pros: No new surface: replay already admits arbitrary curated content
- Cons: Population size is fixed by the file, so every concurrency tier or extension requires re-slicing; warmup/formal partitioning and seed-stable extension move into operator scripts
- Rejected because: Acceptance-rate studies sweep concurrency and population sizes; freezing the population per experiment repeats exactly the out-of-schema workflow replay was meant to absorb.
Release-catalog raw-text datasets (rejected)
Section titled “Release-catalog raw-text datasets (rejected)”- Pros: Release-qualified identity for shared corpora
- Cons: Catalog materialization is conversation-oriented; raw-text slicing semantics would have to be invented there anyway, and operator-specific corpora do not belong in a release
- Rejected because: The demonstrated need is operator-local corpora with random-source geometry; release qualification is a later step if a corpus proves broadly reusable.