Specification · RFC
RFC-0004: Measurement And Toolchain Contract
Version: 0.44.0 | Status: normative | Phase: spec Owners: @Zihua Wu
References: RFC-0009
1. Summary
Section titled “1. Summary”[RFC-0004:C-SUMMARY] Summary (Informative)
Section titled “[RFC-0004:C-SUMMARY] Summary (Informative) ”This RFC governs measurements and the toolchains that execute them: Eval and Bench semantics and result normalization, the InferLab-owned release-versioned toolchain, whose image-packaging runtime also serves RFC-0007:C-IMAGE-BUILD, and workload-attached Nsight Systems profiling. Servers, server cases, server records, and readiness semantics are defined by RFC-0003; records by RFC-0005. RFC-0001 holds the specification overview and the authority map.
Since: v0.1.0
2. Specification
Section titled “2. Specification”[RFC-0004:C-MEASUREMENTS] Eval And Bench Measurements (Normative)
Section titled “[RFC-0004:C-MEASUREMENTS] Eval And Bench Measurements (Normative) ”InferLab MUST support reusable workspace-local Eval and serving-Bench definitions.
A static serving-Bench definition MAY omit its definition kind; omission MUST resolve to serving. An explicit definition kind MUST remain serving or adaptive-serving. The effective definition kind MUST be preserved in canonical merged workspace output, dry-run, execution, and record evidence whether it was declared or defaulted.
An Eval definition MUST be able to express the evaluated task or dataset selection, bounded sample policy, positive fixed repeated-trial count, primary metric, pass threshold, and timeout when those values apply.
A serving-Bench definition MUST be able to express prompt-source parameters, prompt representation, input and output shape, concurrency-limited load, request-rate-limited load, arrival burstiness, request-count or duration policy, cache start policy, optional server-metric collection, artifact level governed by RFC-0004:C-BENCH-ARTIFACT-LEVEL, timeout, and the request-body fragment governed by RFC-0004:C-INFERENCE-REQUESTS when those values apply.
A concurrency serving-Bench definition MAY declare the non-negative integer field warmup_prompts_per_concurrency. Omission MUST resolve to zero. The field applies to every request source governed by RFC-0004:C-BENCH-REQUEST-SOURCES, where one materialized population entry produces one request.
For each expanded concurrency case with concurrency c, InferLab MUST resolve the warmup request count as c * warmup_prompts_per_concurrency. A value that cannot be represented by the supported request-count domain MUST fail definition resolution.
A zero resolved warmup request count MUST preserve the existing single profiling-phase behavior.
A positive resolved warmup request count MUST lower to an AIPerf-native warmup phase before the case profiling phase. The warmup phase MUST retain the case concurrency, effective request-source policy, selected route, served model, and effective request-body fragment. AIPerf MAY add its native phase marker to the effective warmup prompt or message; InferLab MUST preserve the phase-tagged native request artifact rather than claiming byte-identical profiling input.
The frozen request population MUST contain the resolved warmup request count plus the profiling request count, use sequential sampling through the one sampler shared by the native phases, and contain enough entries that neither phase wraps or reuses an entry. InferLab MUST preserve the native conversation identity and benchmark phase for emitted warmup and profiling request records.
Warmup requests MUST be excluded from normalized metrics and from the case completed and failed profiling-request counts. A positive warmup succeeds only when the native phase artifacts contain exactly the resolved number of warmup request records and every such record is terminal without an error or cancellation. A missing, incomplete, errored, or cancelled warmup request MUST fail the case. The failure MUST report the expected, observed-completed, errored, cancelled, and missing warmup counts and preserve the available process outcome and, at the diagnostic artifact level governed by RFC-0004:C-BENCH-ARTIFACT-LEVEL, the raw AIPerf phase artifacts.
The declared warmup multiplier and each concurrency case’s resolved warmup request count MUST be preserved in dry-run and execution evidence.
A workload suite MUST be able to select an Eval gate and one or more Eval and Bench measurements; a recipe selects those measurements only through its named workload suite.
A Bench guarded by an Eval gate MUST be skipped when that gate does not succeed.
An OpenAI-compatible completions smoke definition MAY declare its prompt, maximum-token limit, and timeout independently of a server endpoint or execution engine. Omission MUST resolve the prompt to Hello, the maximum-token limit to 16, and the timeout to 60 seconds. Explicit values MUST replace those defaults subject to the same validation. The effective prompt, maximum-token limit, and timeout MUST be preserved in canonical merged workspace output, dry-run, execution, and record evidence.
The Eval definition kind MUST determine the execution boundary without a separate user-selectable engine setting.
An OpenAI-compatible completions smoke measurement MUST NOT require or launch an installed measurement-toolchain runtime.
The smoke request MUST use the resolved public completions path and served model, the effective prompt and maximum-token limit, non-streaming generation, and zero temperature. It MUST NOT accept the Eval and Bench request-body fragment.
A successful smoke measurement MUST receive a 2xx HTTP response containing a JSON object with a non-empty choices array whose first element contains a string text value.
A transport error, timeout, interruption, non-2xx response, malformed JSON response, missing or empty choices array, or a missing or non-string first-choice text value MUST fail the smoke measurement.
A smoke definition MAY declare vision = true; omission MUST resolve to false. A vision smoke request MUST instead use the resolved public chat-completions path and MUST carry exactly one user message whose content is the effective prompt as one text part followed by one image_url part whose data URI embeds the fixed test image bytes owned by the invoking InferLab release. Every vision smoke of one release MUST embed identical image bytes, and the image content MUST NOT participate in the success judgment. A successful vision smoke measurement MUST receive a 2xx HTTP response containing a JSON object with a non-empty choices array whose first element’s message object contains a string content value; the completions smoke failure conditions apply with the first-choice message.content string in place of the first-choice text value. The effective vision selection MUST be preserved wherever the smoke definition is preserved.
Every smoke attempt MUST preserve the effective HTTP request, elapsed time and terminal outcome, plus failure evidence when the outcome is failure.
When the HTTP operation returns a complete response, smoke evidence MUST additionally preserve its status, body size, and raw body artifact.
When a valid choices array is parsed, smoke evidence MUST preserve its choice count.
An Eval measurement of any kind governed by this clause, including the OpenAI-compatible smoke measurement, that is bound to a server whose resolved configuration carries synthetic acceptance MUST fail measurement planning as governed by RFC-0003:C-SERVE-SYNTHETIC-ACCEPTANCE.
lm-eval measurements MUST execute through the Eval runtime fixed by the invoking InferLab release.
Serving Bench measurements MUST execute through the AIPerf runtime fixed by the invoking InferLab release.
A serving-Bench definition MAY set server_metrics = true; omission MUST resolve to false. When false, InferLab MUST NOT ask AIPerf to scrape a server-metrics endpoint. When true, the resolved target server capability MUST supply one AIPerf-compatible server-metrics endpoint containing its logical path and public-or-named-port binding plus the exact control-plane-resolved URL. The measurement runtime MUST pass that exact frozen URL to AIPerf and MUST NOT reconstruct or alter it. InferLab MUST fail preflight before the first transport request when the capability is absent or invalid.
Server-metric collection MUST run through AIPerf’s native serverMetrics configuration and MUST request JSON export. InferLab MUST NOT substitute framework logs, a framework-native benchmark client, or an independently timed scraper. The required native raw artifact is the case run directory’s unchanged server_metrics_export.json. An AIPerf launch failure, a non-zero profile outcome not accepted by the complete all-inference-error rule in RFC-0004:C-BENCH-SLO, or a completed profile without that artifact MUST fail the case while preserving available process and raw evidence. When AIPerf returns zero and emits the required artifact, InferLab MUST accept the native AIPerf classification of recoverable scrape errors and MUST NOT parse log text, infer failure from metric gaps, or inspect private in-process state to impose a second collection-success policy.
For a successful speed_bench case with server metrics enabled, the measurement runtime MUST take acceptance length and acceptance rate from the first of these release-pinned AIPerf SPEED-Bench sources that yields a value for the one case run directory: per-request records, then the run summary, then the scraped server metrics. Each value MUST be labeled with the case’s resolved SPEED-Bench category, which InferLab supplies because the frozen population reaches AIPerf as a file dataset whose selector does not name it.
The computation MUST yield exactly one value for the case’s resolved category. InferLab MUST expose the acceptance-length value as the scalar acceptance_length and the acceptance-rate value as the scalar acceptance_rate, unrounded. These normalized values inherit the release-pinned AIPerf report computation’s aggregation semantics; InferLab MUST NOT recompute them from server-metric series, transport-token counts, or framework logs.
A missing value or a value for any other category, a non-numeric or non-finite value, acceptance length below one, acceptance rate outside the inclusive range from zero through one, or a failure of the computation MUST fail successful-result normalization and identify the metric. A non-SPEED-Bench source MUST omit acceptance_length and acceptance_rate even when raw server metrics are collected.
For any serving Bench, a completed profiling request whose backend usage reports accepted and rejected prediction tokens contributes its accepted tokens and its drafted tokens, the sum of the two. When the contributions sum to a positive drafted total, the schema MUST expose draft_accepted_tokens and draft_tokens as the profiling-window totals and draft_acceptance_rate as accepted over drafted tokens, token-weighted across the window; otherwise it MUST omit all three. A request reporting only one of the two counts, or a negative or non-integer count, MUST fail successful-result normalization. These values MUST NOT replace or be presented as the SPEED-Bench acceptance_rate, and InferLab MUST NOT derive an acceptance length from them, because usage carries no verify-step count.
Evidence MUST preserve each acceptance value and the source it was taken from, as governed by RFC-0005:C-EVIDENCE, together with the unchanged native artifacts that source read. InferLab MUST NOT maintain a second server-metric-name matcher or claim a framework metric identity that the AIPerf computation does not expose.
A measurement client result MUST carry result envelope version 1.
The measurement runtime MUST reject an Eval or Bench result whose envelope version it does not support, failing that case with an error naming the unsupported version.
Static concurrency and request-rate definitions MUST expand into deterministic ordered cases before execution.
Adaptive-serving definition, SLO evaluation, request-count success semantics, and rate search MUST follow RFC-0004:C-BENCH-SLO.
Manual and closed-loop execution under the same invoking InferLab release and toolchain identity MUST use the same case planning, cache preparation, timeout, AIPerf version, result normalization, and evidence semantics for the same effective Bench definition and serving snapshot.
Each Bench case MUST report profiling-phase completed_requests and failed_requests as integer counts derived only from native records tagged for the profiling phase.
Bench normalization schema aiperf-summary-v1 MUST map AIPerf request_throughput.avg, output_token_throughput.avg, and total_token_throughput.avg to the scalar InferLab metrics request_throughput, output_throughput, and total_token_throughput, respectively.
The schema MUST expose backend-observed prompt-token count, request latency, and TTFT as the stable distribution families prompt_tokens, request_latency_ms, and ttft_ms. With AIPerf server token counting enabled, prompt_tokens MUST retain the backend-reported final input-token count after the locally frozen or server-owned rendering selected by the resolved prompt kind; it MUST NOT use a source-content estimate.
For a Bench whose request source fixes every effective output-token limit at one, TPOT is not applicable and the normalized result MUST omit the tpot_ms family.
For a Bench whose request source guarantees every effective output-token limit is at least two, the schema MUST expose TPOT as the stable distribution family tpot_ms.
For each applicable distribution family, the schema MUST map AIPerf statistics avg, min, max, std, p50, p90, p95, and p99 to InferLab metric-name prefixes mean, min, max, stddev, p50, p90, p95, and p99, respectively.
Each normalized distribution key MUST be composed as <prefix>_<family>. The prompt-token keys MUST be mean_prompt_tokens, min_prompt_tokens, max_prompt_tokens, stddev_prompt_tokens, p50_prompt_tokens, p90_prompt_tokens, p95_prompt_tokens, and p99_prompt_tokens. The request-latency keys MUST be mean_request_latency_ms, min_request_latency_ms, max_request_latency_ms, stddev_request_latency_ms, p50_request_latency_ms, p90_request_latency_ms, p95_request_latency_ms, and p99_request_latency_ms. The TTFT keys MUST be mean_ttft_ms, min_ttft_ms, max_ttft_ms, stddev_ttft_ms, p50_ttft_ms, p90_ttft_ms, p95_ttft_ms, and p99_ttft_ms. When applicable, the TPOT keys MUST be mean_tpot_ms, min_tpot_ms, max_tpot_ms, stddev_tpot_ms, p50_tpot_ms, p90_tpot_ms, p95_tpot_ms, and p99_tpot_ms.
The schema MUST source the prompt-token family from AIPerf input_sequence_length, the request-latency family from request_latency, the TTFT family from time_to_first_token, and the TPOT family from inter_token_latency.
InferLab MUST treat AIPerf inter_token_latency as a private source name for AIPerf’s per-request (request latency - TTFT) / (output tokens - 1) value. The normalized result MUST NOT expose a duplicate ITL metric family for that same value.
When per-request cache-read observations are required by RFC-0004:C-BENCH-CACHE-STATE, the schema MUST expose prompt_cache_read_ratio from the summed per-request prompt-cache-read tokens divided by the summed per-request prompt tokens. When those observations are not required and AIPerf reports a finite overall_usage_prompt_cache_read_pct.avg profiling value in the inclusive range from zero through one hundred, the schema MUST expose prompt_cache_read_ratio as that value divided by one hundred. Both paths retain token-volume-weighted profiling-window semantics.
When per-request cache-read observations are not required and AIPerf does not report overall_usage_prompt_cache_read_pct.avg, the normalized result MUST omit prompt_cache_read_ratio. A reported non-numeric, non-finite, or out-of-range aggregate value MUST fail successful-result normalization.
The normalization schema MUST preserve AIPerf’s profiling request window, token accounting, and percentile semantics for those source fields.
A successful Bench case with at least one completed profiling request MUST fail validation when a required throughput scalar, prompt-token statistic, request-latency statistic, or TTFT statistic is absent, non-numeric, or non-finite. A successful Bench with at least one completed profiling request whose request source makes TPOT applicable MUST also fail validation when a required TPOT statistic is non-numeric or non-finite, or is absent while at least one completed profiling response produced two or more output tokens. Under output_stop = "eos", when no completed profiling response produced two or more output tokens, the normalized result MUST omit the TPOT family, and an aggregate TPOT constraint MUST count as unsatisfied.
A request-level SLO case with zero completed profiling requests MUST follow RFC-0004:C-BENCH-SLO and MUST NOT fabricate distribution metrics for an empty completed-request population.
A failed Bench case MUST finalize as one failed Bench-record case with explicit error text, stdout and stderr paths, available request counts, produced partial raw artifacts, and an empty or explicitly partial metric set.
A failed Bench case MUST preserve process outcome when a measurement process started.
Serving-Bench cache start, reset, conditioning, phase ordering, preparation failures, case-budget ownership, and cache-derived metric requirements MUST follow RFC-0004:C-BENCH-CACHE-STATE.
InferLab MUST record each Eval or Bench measurement’s identity, effective parameters, effective invocation, structured metrics, produced raw result artifacts, failure, and skip reason when applicable.
A measurement runs only against a server whose every process is alive. Before and after each Eval and each Bench case, InferLab MUST observe every process of the target server; a process observed exited MUST fail that Eval or case, naming the process and the command that stops the server. A static-matrix Bench MUST then skip its remaining cases, and a recipe MUST skip its remaining workloads and proceed to its cleanup, which ends the server’s remaining processes. InferLab does not supervise a ready server between measurements; serve status reports each process’s observed state.
Rationale: Eval and serving performance are stable InferLab measurement concepts even when serving frameworks change. The built-in smoke is an endpoint protocol check that needs no external evaluation engine, while lm-eval and AIPerf remain release-owned external engines for dataset evaluation and serving Bench. A native excluded warmup phase lets a declared workload exercise shape-dependent JIT, allocation, and connection setup without discarding measured requests or teaching the control plane to parse framework-specific logs. Keeping case, gate, cache preparation, warmup, adaptive, lifecycle, and evidence semantics in InferLab preserves one experiment workflow across those execution boundaries.
Since: v0.1.0
[RFC-0004:C-INFERLAB-TOOLCHAIN] InferLab Measurement Toolchain (Normative)
Section titled “[RFC-0004:C-INFERLAB-TOOLCHAIN] InferLab Measurement Toolchain (Normative) ”The public CLI MUST provide inferlab toolchain install to install the lm-eval, Bench, and image-packaging runtimes fixed by the running InferLab release.
One explicit installation MUST provide independently isolated lm-eval, Bench, and image-packaging dependency closures under one version-and-platform installation identity.
The installed Eval runtime MUST contain the release-pinned lm-eval engine and every InferLab-bundled Eval task closure fixed by that release.
The installed Bench runtime MUST contain the release-pinned AIPerf engine.
The installed image-packaging runtime MUST contain the release-pinned ELF search-path editor, CUDA device-code extractor, and wheel repacker that RFC-0007:C-IMAGE-BUILD uses for locally built packages.
toolchain install, lm-eval or Bench preflight, and image package-build preflight MUST support glibc Linux x86_64 and aarch64.
toolchain install, lm-eval or Bench preflight, and image package-build preflight MUST fail with an explicit unsupported-platform error on other operating systems, C libraries, and architectures.
A workflow that selects no lm-eval measurement, Bench, or image package build MUST NOT require a supported toolchain platform or an installed toolchain.
The toolchain MUST be installed directly at $XDG_DATA_HOME/inferlab/toolchains/<inferlab-version>/<platform>/, using the platform’s conventional XDG data default when XDG_DATA_HOME is unset.
Installed prefixes MUST NOT be relocated after installation.
A complete matching installation MUST be identified by completion metadata that matches the invoking InferLab version, host platform, release manifest and lock digests, runner digests and versions, installed lm-eval and AIPerf versions, installed image-packaging tool versions, and the bundled Eval task-closure digest fixed by the release.
The bundled Eval task-closure digest MUST cover every bundled task definition, prompt asset, dataset asset, and scorer implementation consumed by lm-eval.
Installation MUST serialize concurrent writers.
Installation MUST consume the package definition fixed by the InferLab release without updating it.
Installation MUST verify every runtime closure and the bundled Eval task closure before publishing completion metadata.
A matching complete installation MUST make toolchain install an idempotent no-op.
An explicit toolchain install MUST either replace an incomplete or release-mismatched installation with a complete matching installation or fail without publishing matching completion metadata.
When replacing an installation fails because the installation path is held by live processes, the failure MUST identify the holding processes. Identification covers the holders visible to the invoking user; holders beyond that visibility can exist and the identified set is not exhaustive.
Dry-run and execution MUST validate only the matching installed runtimes required by selected lm-eval or Bench measurements and, for an image build, by its selected packages (RFC-0007:C-IMAGE-BUILD).
Dry-run, experiment execution, and image builds MUST NOT install, repair, update, or select a different toolchain version.
When a selected lm-eval or Bench measurement requires a missing or incomplete matching runtime, dry-run and execution MUST fail before the first experiment effect and identify inferlab toolchain install as the required action.
Different InferLab versions MUST be able to coexist. Automatic update, garbage collection, registry publication, and general-purpose toolchain management commands are outside this contract.
Rationale: Dataset evaluation and serving performance measurement use release-owned external engines rather than the model-serving dependency closure. One explicit versioned installation keeps workspaces small and serving locks stable, while isolated lm-eval and Bench closures prevent their unrelated dependencies and platform requirements from perturbing each other. Binding bundled Eval assets and scorer code to the same completion identity prevents two installations of one InferLab release from silently executing different tasks. The built-in OpenAI smoke remains available without this installation because it adds no external evaluation engine. Image packaging follows the same model, so locally built packages are rewritten and verified by tools pinned with the release rather than by whatever a host or workspace happens to provide.
Since: v0.1.0
[RFC-0004:C-WORKLOAD-PROFILING] Workload-Attached Profiling (Normative)
Section titled “[RFC-0004:C-WORKLOAD-PROFILING] Workload-Attached Profiling (Normative) ”InferLab supports two workload-attached capture mechanisms: managed_collection, in which InferLab wraps each target process with a managed Nsight Systems collection, and engine_trace, in which the serving framework’s internal profiler writes per-rank trace artifacts under InferLab-directed storage. A server, its cases, and invocation-scoped overrides MAY declare profiler.mechanism; omission MUST resolve to managed_collection under RFC-0003:C-RESOLUTION.
When profiling is enabled for a server, InferLab MUST prepare every launched process rank of each model-serving replica declared as profile-capable by the selected integration as a capture target of the effective mechanism for later Eval or Bench measurements.
A server, server case, or invocation-scoped server setting MAY request capture-target preparation. A recipe run whose server resolves profiling on but whose invocation selects no capture runs instrumented and captures nothing; InferLab MUST say so on standard error when the run starts and again when it ends, naming --capture and the recipe’s workloads, without changing the run’s outcome.
Profiler declarations MUST NOT silently drop: when profiling resolves off for a server (no profiling = true and no requested capture), an effective declaration of profiler.mechanism or profiler escape inputs MUST fail resolution with a typed error naming the declaration and the enable-profiling remediation.
A server prepared with capture targets MUST NOT bound its readiness wait by the effective server readiness timeout (RFC-0003:C-RESOLUTION): capture instrumentation multiplies startup cost by an unpredictable factor, and a budget chosen for uninstrumented startup converts progressing servers into false failures. The capture-armed readiness wait terminates only by readiness success, process-group exit, or operator interruption, each with deterministic evidence.
Each blocking process-status or endpoint attempt within capture-armed readiness MUST use the effective readiness_attempt_timeout_seconds governed by RFC-0003:C-RESOLUTION. Attempt expiry MUST return to the unbounded readiness loop so it can retry and re-evaluate process exit or operator interruption.
An Eval or Bench execution MUST support an explicit profiling-capture request without changing the selected named measurement definition.
A usable capture target MUST identify the target, its logical role, replica, process, and rank boundary, its capture mechanism, its supported framework-neutral window-control kinds, its control mechanism, its profiler storage, and its report finalization or retrieval strategy.
The selected framework integration MUST declare which capture mechanisms it supports and the framework-specific profiler controls and launch inputs each supported mechanism needs. The integration MUST reject a requested mechanism it does not support before launch.
InferLab MUST retain authority for profiler process lifecycle, capture planning, window control, collection finalization, report verification, cleanup, and authoritative evidence.
Preparing profiler output and arming managed collection across every selected target MUST consume one effective capture_arm_deadline_seconds budget. That budget MUST begin immediately before the first profiler-output preparation or collection-arm external action. A later target or command MUST consume the remaining arm budget and MUST NOT restart it. Failure to prepare output or arm any selected target, including expiry of that budget, MUST fail the containing profiled measurement before measured client work begins.
For an engine-trace capture target, InferLab MUST assign a persistent trace directory under record-owned runtime storage, keyed by server definition and replica, and supply it to the integration at render time, and the integration MUST render that directory into the target’s framework-native launch configuration so engine processes write trace artifacts into it directly. Trace artifacts are measurement evidence: they MUST NOT be directed to machine-local scratch or cache storage, and InferLab MUST NOT recopy them into the record after the fact.
An engine-trace capture target MUST be rejected when the selected placement is not entirely local to the controlling machine or when the launch is image-backed: InferLab defines no remote or in-container trace retrieval.
For an engine-trace target, artifact flush completion is judged by coverage verification alone, never by the window-closing control response, whose latency carries no completion information. Coverage verification MUST confirm that the storage delta since collection arming contains at least one new trace artifact per device of the target replica’s whole-replica device count, counted without framework-specific file naming. Engine-internal profilers write one artifact per engine worker process, which the device count bounds, while the control-plane rank model counts entry processes. The trace directory is dedicated to the target replica, so unrelated writes cannot fabricate coverage. A delta smaller than the device count MUST be treated as capture failure. Engine-trace targets declare no profiler escape inputs in this version.
For the managed-collection mechanism, a server MAY declare profiler escape inputs, and each of its model-serving roles MAY declare escape inputs that merge into the server’s: scalar escapes replace, launch and start option lists concatenate with the role’s options after the server’s, the trace set replaces, and environment entries merge per key with the role value winning.
Managed-collection escape option lists are declared separately for the profiler launch command and the collection-start command, and InferLab MUST splice each list ahead of its managed argv tail so that a managed value wins on collision.
The managed-collection profiler executable, trace set, sampling, and context-switch escapes are dedicated fields: the trace set replaces the managed default trace set, sampling and context switch each replace their managed default, and the free-list option form of a dedicated field MUST be rejected toward the dedicated field.
A managed-collection escape environment map MUST apply to every InferLab-managed Nsight Systems command for the target, including profiler launch, collection start, session inspection, and collection stop. Entries on the profiler launch command MUST reach the wrapped server process through launch-command inheritance. For collection-start, session-inspection, and collection-stop commands, InferLab MUST apply that environment only to the command process and MUST NOT separately propagate it to or modify the wrapped server process. Environment keys MUST be POSIX identifiers, rejected at workspace load naming the escape field and the key; the rendered environment prefix terminates option parsing with a separator before the first assignment, so no key can be parsed as an option of the environment utility.
A managed-collection escape option that names a managed profiler fact — session identity; report storage, export, and overwrite lifecycle; capture-range mechanics; or launch wait behavior — MUST be rejected when the workspace is loaded, naming the escape field and the offending option, in long, short, attached short-option-value, and abbreviated long forms — any strict prefix of a managed long option name, which either resolves to the managed option or is an ambiguity. A standalone - or -- token in an escape option list MUST likewise be rejected at workspace load, because it ends option parsing and displaces the managed argv tail into positionals of the wrapped command.
The record MUST preserve the effective mechanism of every capture target, both the raw escape declaration and the effective profiler invocations it produced when escapes apply, and the assigned trace storage of engine-trace targets.
Profiling an Eval or Bench MUST remain an observation mode attached to that measurement and MUST NOT create a separate workload kind.
A profiling request MUST bind to a managed server record and resolve every prepared process-rank target before measured client work starts.
Each capture window MUST have a semantic identity and a binding to the measured Eval phase or Bench case before the window opens.
A static capture plan for a range-backed target MUST enumerate every expected capture window and compute the expected range count for each selected target before collection is armed.
A static capture plan MUST preserve the mapping from each framework range index to its semantic capture-window identity.
A profiling request that uses a range-backed target MUST fail before measured client work when its capture windows cannot be enumerated in advance.
A dynamic measurement MAY use an external-session control kind instead when every selected target advertises that control kind.
InferLab MUST open a capture window before its bound measured phase begins.
InferLab MUST close the capture window after the measured phase completes or fails.
InferLab MUST attempt to close an opened capture window when measured client work fails.
Each framework window-control action MUST establish its effective capture_control_deadline_seconds budget immediately before dispatching the control request. That budget MUST cover the complete HTTP request, including connection establishment and response consumption. The action MUST NOT impose a shorter fixed connection or response cap. Expiry of a window-opening action budget MUST fail the containing profiled measurement. For a managed-collection target, expiry of a window-closing action budget MUST be treated as a window-closing control failure and adjudicated by coverage verification below. For an engine-trace target, window closing is a dispatch into the finalization phase: InferLab MUST dispatch the close request when the measured phase ends, and the dispatch, response consumption, flush waiting, and coverage verification MUST consume the one global finalization budget below rather than a window-control action budget. A delivery failure of an engine-trace close — connection refusal, a dead engine process, or a prompt error status — MUST be treated as a window-closing control failure and adjudicated by coverage verification; a slow or absent response MUST be preserved as neutral flush-pending evidence and MUST NOT by itself fail the capture.
For asynchronous report generation, InferLab MUST finalize managed profiler collection before verifying report coverage or performing destructive server cleanup.
Collection-state inspection, any required collection stop, engine-trace window-close dispatch and response consumption, asynchronous report completion, engine-trace flush completion, and coverage verification across all selected targets MUST consume one effective capture_finalization_deadline_seconds budget. That budget MUST begin when the measurement business result becomes terminal, before the first finalization action. A fallback action or later target MUST consume that budget’s remaining time and MUST NOT restart the configured duration. Expiry of the finalization budget MUST fail the containing profiled measurement while preserving the terminal measurement business result.
After collection finalization, InferLab MUST verify coverage for every planned range on every required range-backed target and every required rank trace artifact on every required engine-trace target.
A required report reference or rank trace artifact missing after collection finalization MUST be treated as capture failure.
A measurement requiring profiling MUST fail when target resolution, window-opening control, measured client work, collection finalization, required report generation, or required coverage verification fails.
A window-closing control failure MUST be preserved as evidence and adjudicated by coverage verification: the capture succeeds despite it when every required report and rank trace artifact verifies, and fails carrying both the coverage and control evidence when one does not.
Measurement evidence MUST preserve the capture plan, window identities, selected role, replica, process, and rank targets, capture mechanism per target, range mapping, control actions, collection finalization, report and trace-artifact references, verification outcomes, and failure evidence when those facts apply.
Rationale: Profiling observes ordinary correctness and performance work. Two mechanisms share one window, budget, and evidence spine because the operator question — what happened inside this measured phase — is mechanism-neutral, while the artifact producers differ: managed collection owns its report lifecycle outright, whereas an engine-internal profiler sees op-level behavior no host collector can observe but writes its own files, so record-owned storage and per-rank delta verification replace the managed report lifecycle for it. Engine trace is confined to local non-containerized placements because retrieval over SSH or out of a container is undefined, not because the mechanism is weaker. Binding semantic windows to stable role, replica, process, and rank targets keeps traces comparable across single-role and multi-role topologies, while static range planning preserves the Nsight Systems repeat-count behavior needed by framework-native CUDA profiler controls. The capture-armed readiness wait is unbounded because the readiness loop already fails instantly on process-group death and cleanly on interruption, so a timeout’s only unique detection is the alive-but-stuck class — while the failure such a budget actually produces is a progressing instrumented warmup killed by a limit written for uninstrumented startup. Window-opening control stays fail-fast because a lost start silently shifts every later range index off its planned window identity; window-closing control defers to coverage verification because the artifacts either landed or they did not, and a late stop response must not fail a capture whose evidence exists. Escape option lists stay split between launch and start because the qualified Nsight Systems option surfaces are disjoint where it matters — trace-shaping options exist only at launch while metrics, sampling, and capture-range options exist only at start — so a single list could not be spliced correctly. Managed facts stay managed because they are the static capture plan’s spine: session identity binds control actions to their target, the report lifecycle underwrites coverage verification, and capture-range mechanics carry the planned range mapping. Dedicated fields for the trace set, sampling, and context switch dissolve the default-collision problem: those managed defaults exist to be replaced, so replacement flows through a typed field instead of an option-list collision rule. The environment map is shared across every managed Nsight invocation so binary selection and runtime dependencies do not drift between launch, collection, inspection, and finalization; only the launch invocation passes that environment onward to the wrapped server.
Since: v0.1.0
[RFC-0004:C-RECIPE-MEASUREMENT-OVERRIDES] Recipe Measurement Overrides (Normative)
Section titled “[RFC-0004:C-RECIPE-MEASUREMENT-OVERRIDES] Recipe Measurement Overrides (Normative) ”recipe run MUST accept invocation-scoped typed patches for Eval and Bench definitions selected by the recipe’s workload suite.
An Eval patch path MUST be rooted at evals.<eval-id>. A Bench patch path MUST be rooted at benches.<bench-id>. A path naming an unselected definition MUST be rejected before the server starts.
Measurement patches MUST use the same repeatable TOML key-path and value syntax and command-order precedence as server patches in RFC-0003:C-RESOLUTION.
A measurement patch MAY assign a typed optional field that the workspace declaration omitted. It MUST NOT change measurement identity or kind, workload-suite membership or order, gate selection, recipe identity, or selected server.
Each measurement child record MUST preserve the declared definition, the complete effective definition, and the ordered invocation overrides that produced it.
The framework-neutral override surface MUST NOT define a generic array append or removal language.
Rationale: Operators need to vary bounded sample counts and concurrency while keeping the recipe’s server and measurement composition reproducible. Typed definition patches provide that iteration point without creating a second recipe or a general workflow language.
Since: v0.8.0
[RFC-0004:C-LM-EVAL] lm-eval Task And Endpoint Contract (Normative)
Section titled “[RFC-0004:C-LM-EVAL] lm-eval Task And Endpoint Contract (Normative) ”An lm-eval definition MUST select exactly one task source: one individual task name built into the release-pinned lm-eval runtime, one InferLab-bundled task selected by a bundled name, or one workspace-local lm-eval YAML file resolved relative to the workspace root.
A bundled task MUST resolve its task definition, prompt and dataset assets, and scorer from the bundled Eval task closure fixed by the invoking InferLab release. It MUST NOT require a downstream workspace copy or fetch task content from the network.
A workspace-local task YAML path MUST resolve to a regular file inside the workspace. The selected file and the complete closure of YAML files reached through relative or absolute include declarations MUST remain inside the workspace and satisfy the source-containment and identity rules of RFC-0002:C-WORKSPACE-AUTHORITY. Selection MUST fail before inference when the file or any included source is absent, escapes the workspace, is not identity-covered, cannot be loaded by the release-pinned lm-eval runtime, or does not resolve to exactly one named task. InferLab MUST let lm-eval own task parsing and include composition in the Python Eval integration rather than copying lm-eval task semantics into the Rust control plane.
The resolved lm-eval task configuration is the sole authority for dataset path, dataset name, evaluation split, few-shot split, prompt construction, output type, filters, metrics, scoring direction, task-internal response multiplicity, and scorer behavior. A built-in task uses the configuration shipped by the release-pinned lm-eval runtime. A bundled task uses the configuration and contained assets from the release-fixed bundled Eval task closure. A workspace-local task uses its selected YAML and contained include closure. InferLab MUST NOT maintain independent task-specific prompt, answer, parsing, or scoring rules.
The definition MAY apply bounded sample count, few-shot count, seed, generation-token limit, request concurrency, fixed trial count, and timeout where those invocation controls apply. Such invocation controls MUST NOT replace task-owned dataset, prompt, output-type, filter, metric, or scoring declarations.
Every lm-eval definition used as an InferLab measurement gate MUST declare one primary metric name, one finite pass threshold, and MAY declare one primary-metric filter. The metric, optional filter, and threshold form InferLab’s gate selection and MUST NOT be passed as replacements for the task’s native metric or filter declarations.
An lm-eval definition MAY declare model_thresholds, a table from workspace model identifiers to pass thresholds; its declared threshold is then the default. Every key MUST name a model declared by the workspace, and every value MUST satisfy the same rules as the declared threshold; a violating table MUST fail workspace loading. When the Eval is planned against a server, the entry keyed by that server’s model MUST be the effective threshold, and the declared threshold MUST be effective otherwise. Invocation overrides MUST apply to the declared fields before this selection. The effective definition MUST carry only the selected threshold.
InferLab MUST resolve the selected source through the installed Eval runtime before starting inference. The resolution MUST preserve exactly one task identity, effective task configuration, and output type. A selection that resolves to zero or multiple tasks MUST fail before inference and direct the operator to compose separate Eval definitions in a recipe. Task resolution or validation failure MUST fail the measurement with task-source context.
An lm-eval definition MAY declare trials as a positive fixed count. When omitted, its effective value MUST be one. A trial is one logical repetition of the complete resolved Eval over the same task and sample selection.
The first repeated-trial surface MUST accept trials greater than one only when the resolved task uses generate_until, selects exactly one sample, and produces exactly one scored generation response for that sample in each trial. A task outside that shape MUST remain usable with one effective trial and MUST fail before inference when a greater trial count is requested, naming the unsupported repeated-Eval shape.
Each repeated-trial request MUST use OpenAI n=1. Repeated Eval MUST NOT perform prefix-cache reset or issue additional endpoint requests for scouting or warm-up. The existing fixed request-concurrency control MUST retain its existing meaning for repeated-trial requests; InferLab MUST NOT add a second trial-concurrency setting or select concurrency adaptively.
For a repeated Eval, the effective base seed MUST be the definition seed or 1234 when absent. Trial indices MUST start at one, and trial index i MUST use request seed base_seed + i - 1. InferLab MUST assign stable trial identity and effective seed before releasing that trial for request execution.
A repeated Eval’s selected primary metric and optional filter MUST yield one binary per-trial score whose only valid values are 0 and 1, with higher-is-better scoring direction. Score 1 MUST classify the completed trial as passing. Score 0 MUST classify the completed trial as non-passing. A missing or non-binary score for an otherwise completed trial MUST fail repeated-result normalization.
A repeated Eval threshold MUST be finite and within the inclusive range [0, 1]. At terminal result publication, requested_trials MUST equal issued_trials + unissued_trials; issued_trials MUST equal completed_trials + request_failure_trials; and passed_trials MUST be no greater than completed_trials. A requested trial becomes issued when its generation request is released for execution. An issued trial without a terminal completed classification when the Eval case reaches its terminal point MUST count as a request-failure trial.
When issued_trials is greater than zero, pass_rate MUST equal passed_trials / issued_trials; otherwise pass_rate and its metric-gate conclusion MUST be absent. An unissued trial MUST NOT enter the pass-rate denominator. Whenever issued_trials is greater than zero, InferLab MUST compare the observed pass rate with the effective threshold even when fewer than the requested trials completed. The aggregate metric gate MUST pass when the observed pass rate is at least the threshold and MUST otherwise fail.
A timeout or interruption of the overall repeated Eval case MUST fail that case and prevent its guarded Bench work even when the observed metric gate passed. Endpoint or task-specific non-pass classifications MAY be preserved as evidence, but the task and its scorer remain their authority.
The resolved model locator MUST be the default Hugging Face tokenizer source for the release-pinned lm-eval OpenAI client selected under RFC-0004:C-INFERENCE-REQUESTS. InferLab MUST set tokenizer_backend=huggingface and tokenized_requests=false: lm-eval uses that tokenizer for context lengths and alignment while requests remain text. The first supported surface MUST NOT introduce a second tokenizer-path authority.
A resolved task using generate_until MUST NOT require prompt-token log probabilities. A resolved task using loglikelihood, loglikelihood_rolling, or multiple_choice MUST pass the default endpoint conformance probe before evaluation starts.
The default conformance probe MUST use the exact UTF-8 prompt Inferlab prompt logprob probe: 0123456789. The resolved tokenizer MUST encode that prompt into at least two tokens or the probe MUST fail with tokenizer context. The probe MUST send one non-streaming request to the resolved public completions endpoint with the resolved served model, temperature=0, max_tokens=1, echo=true, and logprobs=1. Its tokenizer loading and HTTP operation MUST each consume the remaining Eval case budget and MUST NOT acquire a shorter fixed attempt cap.
A successful conformance probe MUST receive a 2xx JSON response with one indexed choice whose text is a string and whose logprobs object contains equal-length tokens, token_logprobs, top_logprobs, and text_offset arrays covering the echoed prompt and generated token. It MUST establish that the echoed prompt token count and text align with the resolved tokenizer. It MUST establish that prompt positions needed for continuation scoring have numeric log probabilities and top-logprob objects. It MUST establish that the generated position has a numeric log probability and top-logprob object.
The probe conclusion MUST be one of supported, unsupported, or inconclusive. A conforming 2xx response with generated-token logprobs but no scored prompt positions MUST conclude unsupported. A conforming 2xx response whose echoed prompt does not align with the resolved tokenizer MUST conclude unsupported for that model and route. A transport failure, timeout, non-2xx response, or malformed response MUST conclude inconclusive because it does not prove endpoint capability. Every conclusion other than supported MUST fail before lm-eval inference with an explicit typed failure. InferLab MUST NOT silently fall back from loglikelihood semantics to generation semantics.
When a primary-metric filter is declared, the native lm-eval metric key MUST match that metric and filter exactly. When no filter is declared, selection MUST succeed only when exactly one finite native metric for the declared metric name exists at the selected task result. Missing, ambiguous, non-numeric, or non-finite selections MUST fail. Result ordering, bare aliases, and single-result fallback MUST NOT choose a metric.
For an Eval with one effective trial, the primary metric MUST be selected from the one resolved task; unrelated task or aggregate results MUST NOT be substituted. The task’s resolved higher_is_better declaration is the comparison authority: a finite selected value passes at or above the threshold when true and at or below the threshold when false. A missing or ambiguous comparison direction MUST fail the gate. For a repeated Eval, the pass-rate rules above replace that single-result comparison without replacing the task’s per-trial metric or scorer.
A successful lm-eval result MUST preserve the complete raw result and a normalized finite metric map with task, native metric key, and filter provenance. A started lm-eval attempt MUST preserve its native command, terminal process outcome, stdout, stderr, raw output directory, and any partial result files even when execution, result discovery, or normalization fails.
Rationale: A task YAML and release-fixed task bundle are already reproducible units for dataset, prompt, output, filter, and scoring behavior, so InferLab keeps those semantics with the task rather than creating a second task language in the control plane or its RFCs. The model locator remains the tokenizer authority. A bounded prompt-logprob probe makes loglikelihood support an observed property of the complete public route. Exact metric provenance prevents successful gates from depending on map ordering or whichever filter happens to sort first. Fixed outer trials add reproducible pass-rate measurement for the demonstrated single-sample generation workflow without generalizing Eval into a workflow language or changing existing request-concurrency semantics.
Since: v0.9.0
[RFC-0004:C-INFERENCE-REQUESTS] Task-Directed Inference Requests (Normative)
Section titled “[RFC-0004:C-INFERENCE-REQUESTS] Task-Directed Inference Requests (Normative) ”An lm-eval or serving-Bench definition MAY declare one structured OpenAI request-body fragment in the field request_body.
request_body MUST be a TOML table that projects to a JSON object. TOML tables project recursively to JSON objects; arrays preserve order and project to JSON arrays; strings, booleans, integers, and finite floating-point values preserve their JSON scalar type. TOML date and time values MUST be rejected. TOML has no null value, so a definition or command-line override MUST NOT synthesize JSON null.
The request-body fragment MUST preserve operator-declared JSON member names, nesting, values, and scalar types. InferLab MUST NOT translate those members into a framework-neutral sampling or reasoning profile. InferLab MUST NOT silently remove a member because one backend does not support it.
An Eval or Bench definition MUST NOT expose a sibling temperature, reasoning, sampling, logprob, chat-template, or endpoint-selection field that duplicates a member owned by request_body or the resolved measurement semantics. A serving-Bench local prompt representation and any locally applied chat template MAY be declared only inside the synthetic request source prompt table governed by RFC-0004:C-BENCH-PROMPT-AUTHORITY; it MUST NOT be duplicated in request_body or exposed as an independent endpoint selector. An lm-eval prompt authority MAY be declared only inside the Eval prompt table governed by RFC-0004:C-EVAL-PROMPT-AUTHORITY and is subject to the same prohibition.
The resolved task and measurement runtime MUST remain authoritative for the served model, prompt or messages, stream mode, requested completion count, output-token bound, and stop conditions. A request-body fragment that declares model, prompt, messages, stream, n, max_tokens, max_completion_tokens, or stop MUST fail definition resolution and identify the conflicting member. Server-side template controls such as chat_template and chat_template_kwargs MAY remain non-structural request-body members only when the resolved task or serving-Bench prompt semantics assigns rendering to the model server. In that case they MUST be preserved under their real JSON names, and InferLab MUST claim only that those members were forwarded. A serving-Bench prompt kind whose rendering authority is local MUST reject those members rather than preserve two template authorities.
For an lm-eval definition, the resolved definition seed and repeated-trial seed schedule MUST remain authoritative for the request seed. An lm-eval request_body fragment that declares seed MUST fail definition resolution and identify that conflicting member.
A serving-Bench output-shape control is also authoritative for request members that AIPerf emits to enforce that shape. A serving-Bench request_body MUST reject min_tokens, min_new_tokens, and ignore_eos under every output_stop value, because InferLab alone derives those members, or their absence, from the declared output shape and output_stop.
Request-body members supplied by a release-pinned task or measurement client are task defaults. A definition request-body fragment MAY replace a non-reserved default. Resolution MUST recursively merge object members. At the same member path, a later scalar or array replaces the earlier value atomically, and a later object recursively merges with an earlier object. Traversing a nested override through a non-object value MUST fail and identify that path.
The precedence order MUST be release-pinned task or client defaults, definition request_body, then invocation-scoped overrides in command-line order. The resolved evidence MUST retain an overridden default, its replacement, and their authority sources.
Every request emitted for an Eval or Bench attempt MUST request exactly one completion.
Each lm-eval definition MUST resolve to exactly one individual lm-eval task. A selection that resolves to zero or multiple tasks MUST fail before inference and identify that the operator must select each task as a separate Eval definition in a recipe.
lm-eval prompt authority, its declaration domain, the derived protocol route, and the derived native client MUST follow RFC-0004:C-EVAL-PROMPT-AUTHORITY.
The resolved model locator MUST be the sole tokenizer source for serving-Bench generation, local prompt-length projection, and pre-template content-token measurement. The Bench runtime MUST load it locally with the Hugging Face tokenizer backend fixed by the release-pinned Bench toolchain. A Bench definition MUST NOT declare another tokenizer locator, revision, or backend. The resolved execution MUST preserve the model locator, tokenizer backend, and installed Transformers version.
Serving-Bench synthetic prompt authority, request representation, selected input-token meaning, exactness policy, and derived AIPerf route MUST follow RFC-0004:C-BENCH-PROMPT-AUTHORITY. Dataset-backed independent requests and every linear-session turn on the valid path through its first failure MUST retain structured messages and server rendering. A native linear-session turn emitted after that failure is retained only as failure evidence and is not required to contain a missing assistant response.
For dataset request sources, admission and source-shape evidence MUST measure the sum of structured message-content token lengths with add_special_tokens=false. Server-added template tokens are outside the source admission bound. Native AIPerf request artifacts remain authoritative for the structured messages actually sent, and backend-reported profiling prompt_tokens remains authoritative for the observed post-template input length.
The resolved prompt representation, rendering authority, and derived route MUST be one frozen authority used by dry-run, population preparation, execution, and records. The request form MUST NOT be exposed as an independent operator-selected endpoint switch.
The built-in OpenAI smoke MUST use the declared completions path and MUST NOT consume a definition request-body fragment.
A serving-Bench server_chat attempt MUST fail with the backend diagnostic context when the selected model has no usable server-side chat template. InferLab MUST NOT switch prompt kind or fall back to completions after that failure.
A request-body fragment belongs to its one Eval task or serving-Bench definition. Operators compose multiple tasks as separate Eval definitions in a recipe, so each task retains an explicit request fragment and route.
Invocation-scoped measurement overrides MUST address evals.<eval-id>.request_body.<path> or benches.<bench-id>.request_body.<path>. They MUST use the precedence, declared-definition, effective-definition, and ordered-override evidence rules in RFC-0004:C-RECIPE-MEASUREMENT-OVERRIDES.
The selected named path and complete effective request_body fragment MUST be fixed before the native measurement client starts. The client MUST apply that fragment to every real request body after task or client defaults and before transport. Dynamic task-owned prompt or message content remains part of the measurement engine’s native sample or request artifacts rather than a fabricated single body for a multi-request attempt. Evidence MUST satisfy RFC-0005:C-INFERENCE-REQUEST-EVIDENCE.
Rationale: The OpenAI request is the backend-visible experiment input. Preserving a body fragment directly supports evolving framework fields such as temperature, logprobs, and reasoning_effort without creating a lossy InferLab parameter language. Task and prompt semantics retain authority for structural request members, rendering, and route selection; the route is derived rather than independently configured. Recipe-level composition keeps multi-task experiments explicit without requiring per-task routing inside one lm-eval invocation.
Since: v0.11.1
[RFC-0004:C-EVAL-PROMPT-AUTHORITY] Generative Eval Prompt Authority (Normative)
Section titled “[RFC-0004:C-EVAL-PROMPT-AUTHORITY] Generative Eval Prompt Authority (Normative) ”An lm-eval definition whose resolved task output type is generate_until MAY declare one prompt table whose kind is flat or server_chat. Omission of that table MUST resolve to flat. An lm-eval definition whose resolved task output type is loglikelihood, loglikelihood_rolling, or multiple_choice, or whose resolved task does not reduce to one request type, MUST NOT declare that table and MUST resolve to flat, because the release-pinned lm-eval chat client does not implement loglikelihood scoring, so a task that emits any loglikelihood request cannot place any of its requests on the chat route.
flat MUST use the declared completions path with the release-pinned lm-eval local-completions client and MUST NOT enable lm-eval structured chat construction. The task-owned context therefore crosses as ordinary text and the resolved task retains the continuation and scoring format its filters expect.
server_chat MUST use the declared chat-completions path with the release-pinned lm-eval local-chat-completions client and MUST enable lm-eval structured chat construction so task context becomes a message list before request lowering. With tokenized_requests=false, InferLab MUST let the model server apply its model-owned chat template and MUST NOT locally render those messages into a completion prompt.
The resolved prompt authority MUST determine one protocol route and one native client. An independent operator-selected endpoint switch MUST NOT permit a second route choice, and InferLab MUST NOT derive the authority from a task identity table it owns.
Server-side template controls such as chat_template and chat_template_kwargs MAY remain non-structural request_body members only under server_chat. Under flat they MUST fail definition resolution and identify the conflicting member. That legality MUST be evaluated against the effective prompt authority after invocation-scoped overrides, so an override that changes the authority also changes which request-body members are legal.
A server_chat attempt MUST fail with the backend diagnostic context when the selected model has no usable server-side chat template. InferLab MUST NOT switch prompt authority or fall back to completions after that failure.
The resolved prompt authority, the derived route, and the derived native client MUST be fixed before the native measurement client starts and MUST be recorded with the effective Eval definition. Records MUST distinguish a declared authority from a defaulted one. Native sample evidence MUST preserve the effective task context or messages that lm-eval exposes. A reported Eval metric MUST NOT be presented without its resolved prompt authority, because the same task scored under a different authority is a different measurement. Evidence MUST satisfy RFC-0005:C-INFERENCE-REQUEST-EVIDENCE.
Rationale: Chat rendering and continuation scoring are different evaluation workloads, not two spellings of one request. A task whose filters expect its demonstration continuation format loses accuracy under server chat rendering, while a task or model whose behavior depends on the model chat template and its reasoning controls needs that rendering. Defaulting an omitted authority to flat runs the resolved task in the form its own demonstrations and filters were authored for, so InferLab interposes no rendering unless an operator asks for it; the deviation is declared, not the baseline. Because the choice still moves a metric that a definition compares against a threshold and a recipe can use as a gate, records separate a declared authority from a defaulted one, and InferLab derives the default from one fixed rule rather than from a task table it maintains. The remaining output types declare nothing because the release-pinned client implements no alternative for them.
Since: v0.28.0
[RFC-0004:C-BENCH-SLO] Serving Bench SLO And Adaptive Rate Search (Normative)
Section titled “[RFC-0004:C-BENCH-SLO] Serving Bench SLO And Adaptive Rate Search (Normative) ”A static or adaptive serving-Bench definition MAY declare the ordered field aggregate_slos as a non-empty list of constraints over stable InferLab Bench metrics governed by RFC-0004:C-MEASUREMENTS or this clause.
Each aggregate constraint MUST name one known stable metric and MUST declare exactly one finite inclusive bound, at_most or at_least.
All declared aggregate constraints MUST pass for a case to satisfy its aggregate SLOs.
An aggregate TPOT constraint MUST fail definition resolution when the selected request source makes TPOT inapplicable.
The metrics good_request_ratio and goodput MUST be valid aggregate-constraint metrics only when the same definition declares a request-level SLO.
An unavailable prompt_cache_read_ratio MUST produce an unavailable aggregate-constraint outcome that does not satisfy that constraint without invalidating an otherwise complete measurement.
A configured distribution metric that has no completed-request population MUST produce an unavailable aggregate-constraint outcome that does not satisfy that constraint when a request-level SLO measurement is otherwise complete and internally consistent.
A missing, non-numeric, or non-finite value for any other configured aggregate metric that should exist for the observed profiling population MUST invalidate the measurement result. Under output_stop = "eos", a completed response with fewer than two output tokens lacks request-level TPOT and counts as not good when tpot_ms is declared.
A static or adaptive serving-Bench definition MAY declare one request_slo containing optional inclusive upper bounds request_latency_ms, ttft_ms, and tpot_ms, plus the required field minimum_good_request_ratio.
A request-level SLO MUST declare at least one request-metric upper bound.
Every request-metric upper bound MUST be finite and non-negative.
minimum_good_request_ratio MUST be finite and greater than zero and less than or equal to one.
A request-level tpot_ms bound MUST fail definition resolution when the selected request source makes TPOT inapplicable.
InferLab MUST lower the request-level bounds to the release-pinned AIPerf runtime without exposing AIPerf’s arbitrary SLO-tag surface as InferLab definition fields.
A profiling request MUST count as good only when it terminates without error or cancellation, supplies every request metric required by the definition, and satisfies every declared request-metric upper bound.
An errored request, cancelled request, or completed request missing a required request metric MUST count as not good.
The AIPerf native per-request metrics export for the profiling phase MUST be the file-bound authority for attempted, completed, failed, and good request counts used by request-level SLO evaluation.
Each exported profiling record MUST carry its native session_num transport-request identity, and those identities MUST be unique within the profiling phase. Independent-request population reconciliation follows RFC-0004:C-BENCH-REQUEST-SOURCES; linear-session identity and turn reconciliation follow RFC-0004:C-BENCH-LINEAR-SESSIONS.
The export MUST contain exactly the resolved profiling request count frozen in the case before launch. A missing or duplicate request identity, a non-terminal profiling record, or a terminal-record count that differs from that resolved count MUST invalidate the measurement.
Each terminal profiling record without an error or cancellation MUST count as completed.
Each terminal profiling record with an error or cancellation MUST count as failed.
The attempted-request denominator MUST equal the resolved profiling request count, the number of unique terminal profiling records, and completed requests plus failed requests.
For a completed profiling record that supplies every request metric required by the definition, its native per-request good_request_count value MUST be present, integral, and either zero or one, and MUST be the authority for whether that request is good. A value of one MUST be invalid when any required request metric is absent.
A completed profiling record that observably lacks at least one required request metric MAY omit its native per-request good_request_count value and MUST count as not good. An errored or cancelled record MUST count as not good without fabricating request metrics or a good marker.
The case good_request_count MUST equal the sum of the valid per-request native good values; records governed as not good above contribute zero. When AIPerf emits an aggregate good_request_count, it MUST be integral, MUST equal that file-bound sum, and MUST lie in the inclusive range from zero through completed requests. The aggregate value MAY be absent.
The profiling duration MUST equal the maximum request_end_ns minus the minimum request_start_ns across all terminal native profiling records, converted from nanoseconds to seconds.
The profiling duration MUST be positive and finite, and its stable source identity MUST be native-profiling-request-window.
good_request_ratio MUST equal good requests divided by attempted requests.
goodput MUST equal good requests divided by the profiling duration and MUST use requests per second.
InferLab MUST derive good_request_ratio and goodput from those file-bound counts and duration instead of depending on optional AIPerf derived-summary fields.
A request-level SLO result MUST satisfy its ratio requirement only when good_request_ratio is greater than or equal to minimum_good_request_ratio.
A request-level SLO measurement MUST be invalid when no profiling request is attempted, a profiling record is non-terminal or malformed, or its counts, timestamps, duration, ratio, or goodput are missing, non-finite where applicable, or internally inconsistent.
An independent-request case with a request-level SLO MUST remain a successful measurement when one or more profiling requests fail, including when every attempted request fails, if the profiling phase and native per-request export are complete and internally consistent and no timeout, interruption, launch, protocol, or export failure occurred.
For an independent-request case, a non-zero AIPerf exit caused solely by every profiling request returning an inference error MUST remain service-quality evidence rather than a measurement failure when the preceding completeness conditions hold, and its native exit outcome MUST remain recorded.
A case without a request-level SLO MUST retain the existing requirement for at least one completed profiling request and zero failed profiling requests.
Measurement status MUST describe execution and result integrity independently of SLO satisfaction.
Each successfully measured case MUST evaluate every declared aggregate and request-level SLO and MUST expose one case-level SLO conclusion formed by AND across those evaluations.
A static Bench MUST execute every resolved case despite an earlier case’s SLO failure.
A static Bench with SLOs MUST pass only when every resolved case executes successfully and satisfies its case-level SLO conclusion.
A static Bench without SLOs MUST retain its existing execution-success pass meaning.
An adaptive serving-Bench definition MUST declare one or more aggregate constraints, a request-level SLO, or both.
An adaptive serving-Bench definition MUST declare non-empty positive finite initial_request_rates, non-negative integer max_search_steps, and an optional positive finite min_rate_resolution.
max_search_steps MUST bound the total automatically added doubling and bisection probes after all distinct initial request rates have been probed.
The fields target_metric, target_threshold, and max_refinement_steps MUST NOT remain accepted adaptive-serving definition fields.
The adaptive policy identity MUST be highest-feasible-rate-v1.
The policy MUST probe all declared initial rates in ascending unique order unless a measurement failure, timeout, or interruption aborts the search.
The policy MUST classify a finite observed failure of only at_least constraints as below the feasible region.
The policy MUST classify any finite observed at_most failure or request-level ratio failure as above the feasible region.
The policy MUST classify a probe as feasible only when every configured aggregate and request-level SLO passes.
An unavailable aggregate constraint MUST make its probe infeasible but MUST NOT supply directional evidence for an automatically added probe.
After the initial probes, the policy MUST double the highest observed rate while that rate is feasible or has only below-region failures and no higher observed above-region rate bounds it, until it observes such a higher above-region rate or exhausts max_search_steps.
When automatically added bisection is possible, the active lower endpoint MUST be the highest observed rate classified as feasible or below-region that has a higher observed above-region rate.
The active upper endpoint MUST be the lowest observed above-region rate greater than that lower endpoint.
This one tightest observed bracket MUST take precedence over automatic doubling.
The next bisection probe MUST be the arithmetic midpoint of the active endpoints.
A feasible or below-region midpoint MUST replace the lower endpoint for subsequent bisection, while an above-region midpoint MUST replace the upper endpoint.
A feasible midpoint MUST additionally remain eligible for final rate selection.
The policy MUST NOT execute a midpoint already present in the observation set.
The policy MUST stop adding probes when max_search_steps is exhausted, when an active bracket width is less than or equal to a configured min_rate_resolution, or when no distinct directional probe can be derived.
The policy MUST NOT automatically probe below the lowest declared initial request rate.
The initial rates SHOULD include a low-rate anchor expected to remain below any overload boundary.
The policy MUST select the highest observed feasible rate and MUST NOT extrapolate an unmeasured rate.
The policy MUST report whether the selected rate has a higher observed above-region boundary.
Exhausting max_search_steps without a higher above-region observation MUST remain a normally completed search and MUST NOT claim that the selected rate is a saturation limit.
A normally completed adaptive Bench MUST pass only when it selects an observed feasible rate.
A complete probe whose request failures reduce its request-level ratio MUST remain a valid infeasible observation and MUST NOT abort adaptive search.
A probe whose launch, timeout, interruption, protocol, native-artifact, or required-result validation fails MUST abort adaptive search as a measurement failure and MUST NOT be treated as above-region evidence, except for the complete all-inference-error outcome governed above.
The policy MUST describe its result as the highest observed feasible rate under the executed probes and MUST NOT claim a global optimum across unobserved rates or non-contiguous feasible regions.
Manual and closed-loop execution MUST apply the same resolved SLO evaluation and adaptive policy for the same effective Bench definition and serving snapshot.
Rationale: Serving operators need both distribution-level limits and the rate of individually good requests. Under the ordinary serving regime, throughput floors and latency or quality ceilings form an approximately contiguous feasible-rate interval; deterministic doubling plus directional bisection finds useful observed boundaries without importing a general optimizer or claiming statistical certainty that the measurements do not provide.
Since: v0.15.0
[RFC-0004:C-BENCH-REQUEST-SOURCES] Serving Bench Request Sources (Normative)
Section titled “[RFC-0004:C-BENCH-REQUEST-SOURCES] Serving Bench Request Sources (Normative) ”A serving-Bench definition that selects independent requests MUST declare one request_source table whose kind is random, random_mixture, dataset, or replay. A definition that selects dependent linear sessions MUST instead follow RFC-0004:C-BENCH-LINEAR-SESSIONS; a definition that selects recorded agentic session trees MUST instead follow RFC-0004:C-BENCH-AGENTIC-TRACE-REPLAY. The former top-level input_tokens and output_tokens fields MUST be rejected rather than retained as a compatibility representation.
A random, random_mixture, or dataset request source MAY declare output_stop, whose value MUST be length or eos; omission MUST resolve to length. Under length, every request’s resolved output-token limit is an exact requested length: InferLab MUST request at least and at most that many output tokens and MUST suppress end-of-sequence stopping. Under eos, the resolved limit is a cap: InferLab MUST request at most that many output tokens and MUST NOT set a minimum output length or suppress end-of-sequence stopping. output_stop MUST NOT change the selected samples, prompts, or resolved output-token limits; it changes only the stop controls each request carries, and a frozen population entry MUST carry the controls its request uses. The resolved value MUST appear in merged workspace output, dry-run, execution, and record evidence. A replay source MUST NOT declare output_stop, because each replayed entry carries its own request members. Every replayed request MUST start from end-of-sequence suppression, the length default, and an entry’s extra members MUST override it member by member: an entry without a stop member generates its exact output length, and an entry whose extra sets ignore_eos false stops at end of sequence with its output length as a cap. An extra that is not a JSON object MUST fail preparation naming the line. Populations InferLab writes under eos MUST therefore set ignore_eos false explicitly rather than omit it, so that replaying them reproduces end-of-sequence stopping. Under eos a response MAY end after a single output token; TPOT applicability still follows the requested limits, and the absence of per-token timing is governed by RFC-0004:C-MEASUREMENTS. Replay evidence MUST record each entry’s resolved output stop: eos when its extra sets ignore_eos false, length otherwise.
A random request source MUST declare input_tokens and output_tokens. Each token-length selector MUST be either a positive integer, meaning one fixed value, or a table with positive integer min and max fields whose min is strictly less than its max. Such a table MAY omit kind; omission MUST resolve to inclusive_uniform. When the table declares kind, its value MUST be inclusive_uniform. Both forms MUST resolve to the same canonical inclusive-uniform selector identity in merged workspace output, dry-run, execution, and record evidence. Other distributions, ratio shorthands, means, standard deviations, and unbounded ranges MUST be rejected.
For every generated entry, InferLab MUST sample the input and output selectors independently. An inclusive_uniform selector MUST assign equal probability to every integer from min through max, inclusive. The Bench seed and selector identity MUST determine the sequence, and generating a larger population MUST preserve the same first N selected pairs.
For random and random_mixture, the selected input-token target and synthetic prompt materialization MUST follow RFC-0004:C-BENCH-PROMPT-AUTHORITY. The effective output-token value MUST be the requested output-token limit, exact or a cap as output_stop selects. A fixed output value of one MUST remain valid for the existing prefill-dominant workflow. An output selector whose possible values include both one and values of at least two MUST be rejected so one Bench retains one unambiguous normalized metric schema.
A random request source MAY declare one corpus table whose path is a workspace-relative path to an operator-supplied text corpus and whose optional expected_sha256 binds the corpus bytes. The same locator rules as the replay population file apply: the path MUST NOT be absolute and MUST NOT escape the workspace, and a declared digest MUST match the observed corpus bytes. Resolution MUST read the corpus: a missing or unreadable corpus, or a declared digest that differs from the observed bytes, MUST fail resolution, so a dry-run and a run alike fail before any process launches, naming the path and, for a mismatch, the expected and observed digests. Corpus content MUST NOT be supplied inline or through a remote reference. An invocation override MUST NOT change the corpus path or expected_sha256.
When corpus is declared, entry content MUST be drawn from the corpus instead of the synthetic word supply. InferLab MUST tokenize the corpus once with the resolved model tokenizer, and each entry MUST take exactly its selected input-token target as one slice of the corpus token stream whose offset is determined by the Bench seed and the entry’s population index alone, decoded back to text under the same round-trip verification as synthetic prompts. The offset MUST NOT depend on the total population size, so generating a larger population MUST preserve the same first N selected entries. A corpus whose token stream is shorter than the largest selected input-token target MUST fail preparation.
Corpus slices MUST be drawn independently and MAY overlap. Incidental sharing between overlapping slices is natural reuse: it MUST NOT be measured, promised, or presented as controlled prefix geometry. A declared prefix_sharing MUST retain its exact controlled semantics, with the shared prefix taken as one fixed corpus slice and each unique suffix drawn independently. Omission of corpus MUST resolve to the synthetic word supply with unchanged semantics.
A dry-run MUST resolve the corpus path, declared and observed digest state, and the corpus token length when the corpus bytes and tokenizer are locally available, without fabricating population facts.
A random request source MAY declare one images table attaching image content parts to every generated entry. The table MUST declare fixed positive integer width and height pixel values and MAY declare a positive integer count; omission of count MUST resolve to one. Distribution selectors, means, standard deviations, and unbounded ranges MUST be rejected for image dimensions, as for token-length selectors. Declared input_tokens retain their text-token meaning: images MUST NOT be counted in the selected input-token target, and a server’s image-derived position count remains its own behavior.
An images table MAY declare one source table. Omission MUST resolve to the measurement runtime’s synthetic noise image supply. A declared source MUST name a workspace-relative directory path containing operator-supplied images; the path MUST NOT be absolute and MUST NOT escape the workspace. The enumeration digest MUST cover the directory’s regular files recursively, ordered by byte-wise relative path, combining each file’s content SHA-256; the exact composition rule is release-owned and SHOULD remain stable across releases. A source MAY declare expected_sha256 binding the directory content through that digest; a missing, unreadable, or empty directory, or a declared digest that differs from the observed digest, MUST fail resolution before any process launches, as for a corpus. A source directory that does not exist, is unreadable, or enumerates no regular file MUST fail preparation before the first transport request. A source MAY declare sampling as exactly one of random-with-replacement, shuffle-cycle, or sequential-cycle; omission MUST resolve to shuffle-cycle. An invocation override MUST NOT change the image source path, expected_sha256, or sampling.
A request source that declares images MUST NOT declare the flat or rendered_chat prompt authority; when its prompt table is omitted, omission MUST resolve to server_chat under RFC-0004:C-BENCH-PROMPT-AUTHORITY, and the measurement runtime attaches each request’s images as image_url content parts on the chat-completions route. Where the sampling policy involves randomness, the Bench seed MUST determine the image selection sequence, exactly as it governs the request-source sequence. The per-request image sequence is the measurement runtime’s request-time behavior governed by the declared source form and sampling policy; frozen population entries MUST NOT carry image identity, and the sequence MUST NOT be presented as frozen or replayable. Replaying a population generated under an image decoration attaches no images: the replayed requests carry text content only. A random_mixture, dataset, or replay source MUST NOT declare images. A dry-run MUST resolve the effective image count, dimensions, source form, and — for a declared directory — its declared and observed digest state, without fabricating acquisition facts.
The frozen population MUST preserve each entry’s selected input- and output-token targets. A configured selector describes target request shapes, not the realized output length returned by the server.
Synthetic shared-prefix declarations and their lowering MUST follow RFC-0004:C-BENCH-PREFIX-GEOMETRY. Omission declares no controlled shared-prefix geometry; it MUST NOT be interpreted as a promised zero cache-hit outcome.
A random_mixture request source MUST declare at least two ordered, distinct shapes. Each shape MUST declare positive exact input_tokens, positive exact requested output_tokens, and a positive unsigned 32-bit integer weight. Its token lengths MUST have the same complete prompt-target and requested output-limit meanings as random. Two shapes with the same input- and output-token lengths MUST be rejected. The checked sum of weights MUST fit an unsigned 64-bit integer.
For each generated mixture entry, InferLab MUST select one shape independently with replacement from the categorical distribution whose probability is shape.weight / sum(shape.weight). The Bench seed and ordered shape list MUST determine the selection sequence, and generating a larger population MUST preserve the same first N selections. Weights MUST NOT be presented as exact realized request quotas.
All shapes in one random_mixture MUST declare output_tokens equal to one, or all shapes MUST declare output_tokens of at least two. A mixture spanning those two TPOT applicability classes MUST be rejected so one Bench retains one unambiguous normalized metric schema. Each generated entry MUST materialize according to its resolved prompt kind and selected input-token target under RFC-0004:C-BENCH-PROMPT-AUTHORITY.
For each effective random source and Bench seed, InferLab MUST define one logical deterministic entry sequence. Every static case and adaptive probe MUST independently instantiate that sequence and consume its prefix of the resolved warmup-plus-profiling length. Within a case, warmup MUST consume the first resolved warmup entries and profiling MUST consume the immediately following entries. The sequence MUST NOT wrap, and one generated entry MUST produce exactly one independent transport request.
A dataset request source MUST declare a release-qualified dataset name and positive max_input_tokens. It MAY declare a release-qualified profile name and positive output_tokens as a fixed override. A catalog entry MAY require a profile and MUST define its closed valid profile set. Omission of output_tokens MUST derive each request’s output-token limit from its held-out assistant target only for catalog entries that provide such targets.
The Bench-level seed MUST remain the authority for request-source randomness and the existing stochastic load behavior. An invocation override MUST NOT change request_source.kind.
A dataset name MUST resolve through the catalog fixed by the invoking InferLab release. The definition MUST NOT supply an arbitrary source URL, repository, revision, local path, data format, loader, or AIPerf dataset type.
Each catalog entry MUST bind one stable name to one immutable upstream snapshot with its expected SHA-256 digest, or to one release-pinned AIPerf public dataset, together with its source format and versioned materialization semantics. A profiled catalog entry MUST additionally bind each stable profile name to one source split and closed filter policy. A release MUST NOT resolve one catalog or profile name to different source bytes or selection semantics.
The release-qualified catalog MUST provide speed_bench profiles whose rows come from the release-pinned AIPerf’s NVIDIA SPEED-Bench public datasets. Its profile set MUST distinguish supported qualitative categories and supported throughput input-length and entropy subsets without accepting an arbitrary category, split, Hugging Face selector, or AIPerf plugin name. A speed_bench request source MUST declare a profile and fixed output_tokens.
A catalog source MAY name a release-pinned AIPerf public dataset instead of a download location. Such a source still binds an expected SHA-256 digest: the digest of the rows that AIPerf release’s public-dataset materialization produces, serialized one JSON object per line. On a cache miss, InferLab MUST obtain the rows through that materialization, which owns any resolution of upstream content, any access requirement, and the reporting of their failures; InferLab MUST NOT add dataset-specific acquisition, access, or completeness logic. A materialization failure MUST fail dataset preparation without starting the measurement client and MUST preserve AIPerf’s own failure message. The materialized rows MUST match the catalog digest before InferLab publishes them to the source cache; rows that differ, for example because an upstream source changed, MUST fail preparation until a release updates the digest. Once published, the rows are used exactly as a verified byte snapshot.
toolchain install MUST NOT acquire catalog dataset bytes. Before executing a Bench that selects a dataset source, InferLab MUST resolve the catalog entry and make its verified source snapshot available before the first transport request.
A missing source snapshot MUST be downloaded, or for an AIPerf public-dataset source materialized, with operator-visible progress. The obtained bytes MUST match the catalog digest before InferLab publishes them to the cache. InferLab MUST run an AIPerf materialization so that AIPerf’s own cache resolves to a machine-local location outside any workspace and MUST record that location. A download, transport, materialization, or digest failure MUST fail dataset preparation without starting the measurement client.
The source cache MUST use $XDG_CACHE_HOME/inferlab/datasets/sha256/<digest>, with the conventional $HOME/.cache default when XDG_CACHE_HOME is unset. A verified cache hit MUST avoid network access. The cache MUST remain machine-local, recoverable state and MUST NOT be placed in a workspace or versioned toolchain prefix.
Dataset materialization MUST use the resolved model tokenizer governed by RFC-0004:C-INFERENCE-REQUESTS to measure each input as the sum of its structured message-content token lengths before server chat-template rendering. It MUST preserve the source messages as structured roles and content. Every admitted independent entry MUST remain structured messages for the pinned AIPerf chat endpoint; InferLab MUST NOT freeze a flat prompt or apply a chat template in the measurement runtime.
For a source conversation ending in an assistant target, materialization MUST first treat every preceding message through the last user message as one candidate request and hold out the assistant target. When that candidate exceeds max_input_tokens, materialization MUST remove one complete trailing user-and-assistant exchange and retry with the preceding assistant target. It MUST NOT drop leading history, truncate message content, or leave an assistant message as the final input message.
A conversation for which no non-empty user input and assistant target fit MUST be ineligible. When output length is dataset-derived, an assistant target shorter than two tokenizer tokens MUST be ineligible so TPOT applicability holds for every admitted request. A fixed output_tokens override MUST replace the target-derived length and MAY be one.
For a speed_bench request_source profile, catalog materialization MUST map the first non-empty user turn of the AIPerf-materialized row to exactly one structured message whose role is user and whose content is that turn’s unchanged string. That message MUST produce one independent chat-completions request without client rendering. Materialization MUST NOT replay later turns or expose the AIPerf loader’s multi-turn default in this RFC version. A row with no non-empty first user turn or whose first-turn input exceeds max_input_tokens before template rendering MUST be ineligible; materialization MUST NOT truncate it.
For a speed_bench request_source, AIPerf MUST consume the frozen population through its release-pinned dataset and reporting boundary. InferLab MUST NOT invoke a vLLM, SGLang, TensorRT-LLM, or other serving-framework benchmark client for that population.
Within request_source, one admitted dataset sample MUST produce exactly one independent transport request. Prior assistant messages in that request MUST come from the source rows. InferLab and AIPerf MUST NOT replay source turns as a stateful multi-request session.
InferLab MUST deterministically order candidate samples from the definition seed, select without replacement, and materialize enough admitted entries for the largest resolved warmup-plus-profiling population required by the Bench. Every case MUST consume a prefix of that one frozen population. Warmup and profiling within a case MUST consume disjoint sequential entries.
When the source cannot provide the required admitted population, preparation MUST fail before the first case. InferLab MUST NOT repeat or oversample entries to fill the deficit. AIPerf MUST consume the frozen materialized population sequentially and MUST NOT fetch, reshuffle, resample, or wrap the source.
A dry-run MUST resolve the catalog identity, request-source policy, cache path, and observed cache state without downloading missing bytes. Dry-run MUST NOT materialize an AIPerf public-dataset source; a cache miss is reported as a planned materialization. When the verified source and tokenizer inputs are already available, dry-run SHOULD report the same determinable materialized-population facts as execution. A cache miss MUST be reported as a planned acquisition rather than fabricated population evidence.
A replay request source MUST declare a workspace-relative path naming a population file in the release’s materialized population format and MUST declare its prompt kind. It MAY declare expected_sha256; when declared, preparation MUST verify that the file content digest matches before any transport request. The file content MUST be the sole population authority: entry order, subset selection, duplication, and content edits are expressed by the file itself, and InferLab MUST NOT offer selection, filtering, or transformation of a replay source.
A replay source MUST NOT declare token-length selectors or output overrides; each entry’s recorded input- and output-token targets remain authoritative. Entries mixing the output-one and output-at-least-two TPOT applicability classes MUST be rejected, as for generated sources.
A replay source MAY declare prefix_sharing; its geometry MUST be resolved from the file entries, and cache-state conditioning MUST follow RFC-0004:C-BENCH-CACHE-STATE unchanged. Downstream of population preparation, a replayed population MUST be indistinguishable from a generated one: every case consumes a prefix of the replayed sequence, warmup and profiling consume disjoint sequential entries, and an insufficient population MUST fail preparation without repeating entries.
Preparation MUST validate each entry against the declared prompt kind and the materialized format. A missing or unreadable file, or a declared digest that differs from the observed content, MUST fail resolution before any process launches, as for a corpus. A malformed entry or prompt-shape mismatch MUST fail preparation before the first transport request. An invocation override MUST NOT change the replay path or expected_sha256.
A dry-run MUST resolve the path, declared and observed digest state, prompt kind, and entry count without fabricating population facts.
Rationale: Fixed random shapes, weighted exact-shape mixtures, controlled final-prompt prefix geometry, server-rendered shared-content compatibility, and bounded conversational data are demonstrated request-origin workflows with distinct validation rules but the same serving measurement lifecycle. A closed request source makes those rules explicit while preserving one load, warmup, SLO, timeout, and evidence path. Shared geometry controls reproducible input construction without promising a backend-dependent cache-hit outcome. Relative shape weights expose a realistic mixed workload without promoting the open AIPerf distribution language. Release-qualified source identity and control-plane acquisition keep network behavior and reproducibility visible without exposing the AIPerf dataset surface. Tail rollback preserves a real conversation prefix; deterministic sampling without replacement avoids artificial prefix-cache reuse. An optional corpus declaration replaces only the content supply of the random source so content-sensitive measurements such as speculative-decoding acceptance rates run on natural-language prompts while population geometry stays release-owned. Whether a response runs to its full limit or stops at end of sequence is a measurement choice that changes acceptance and timing, so the operator chooses it with one uniform output_stop rather than per benchmark, and the default preserves existing definitions. Rows that only a dataset’s own tooling can reconstruct come from the release-pinned AIPerf materialization, so InferLab carries no dataset-specific resolution or access logic, while the pinned digest keeps such a source as reproducible as a downloaded snapshot.
Since: v0.16.0
[RFC-0004:C-BENCH-LINEAR-SESSIONS] Serving Bench Linear Sessions (Normative)
Section titled “[RFC-0004:C-BENCH-LINEAR-SESSIONS] Serving Bench Linear Sessions (Normative) ”A static serving Bench definition MAY declare session_source. A session source represents a frozen population of dependent linear conversation templates, not independent transport requests, and MUST NOT expose AIPerf dataset, loader, scenario, context-mode, or plugin names.
A session_source MUST declare a release-qualified dataset and positive max_input_tokens. It MAY declare a release-qualified profile, positive fixed output_tokens, and output_stop with the meaning and default that RFC-0004:C-BENCH-REQUEST-SOURCES defines, applied to every turn’s resolved output-token limit. It MAY declare finite non-negative inter_turn_delay_scale, which defaults to 1.0, and finite non-negative max_inter_turn_delay_seconds; omission of the maximum means that no cap applies. A typed invocation override MAY change either delay control within the selected session-source boundary. The catalog, immutable snapshot, acquisition, cache, tokenizer, structured-message routing, and dry-run rules of RFC-0004:C-BENCH-REQUEST-SOURCES apply. The selected catalog entry MUST define versioned linear-template materialization and whether every source turn has a held-out assistant target from which to derive its output-token limit. Omitting output_tokens MUST be rejected when any admitted turn lacks such a target.
A linear-session definition MUST declare session_source and MUST reject request_source and agentic_source. A session-backed definition MUST declare non-empty positive concurrency, positive sessions_per_concurrency, and MAY declare non-negative warmup_sessions_per_concurrency, which defaults to zero. It MUST reject prompts_per_concurrency, warmup_prompts_per_concurrency, request_rates, request_count, duration_seconds, and burstiness. adaptive-serving MUST reject session_source, and invocation overrides MUST NOT change the selected source boundary.
For concurrency c, the profiling and warmup session counts MUST be c * sessions_per_concurrency and c * warmup_sessions_per_concurrency. An unrepresentable count MUST fail resolution. Concurrency is the maximum number of active native conversations: a session occupies one slot from first-turn admission until its final planned native turn returns or the phase terminates, including while an inter-turn delay or post-failure continuation elapses, and has at most one request in flight. InferLab MUST adjudicate success from the first failing turn without treating that adjudication point as slot release. The phase MUST admit frozen templates in order whenever a native slot is available until the resolved session count has started.
A linear template MUST have one stable template identity and at least two ordered source turns. The supported catalog materialization MUST consist only of those source turns and MUST NOT add a separate system message. Each turn MUST have one non-empty structured user message and one positive requested output-token limit. A turn after the first MAY have finite non-negative inter_turn_delay_seconds, which is the source think time before that turn; omission resolves to zero. A catalog materializer that derives this value from recorded timing MUST use the idle interval from the preceding source response’s terminal receipt to the current source request’s start. It MUST NOT treat a request-start-to-request-start interval as post-response think time. A source trajectory in which the current request starts before the preceding response terminates MUST NOT be admitted as a linear template. A template MUST NOT contain an executable branch, tool step, replayable source assistant turn, or model-selected action.
For every turn after the first, InferLab MUST multiply the source inter_turn_delay_seconds by the effective inter_turn_delay_scale and, when max_inter_turn_delay_seconds is declared, cap that product at the declared maximum. The resulting effective delay MUST be finite and non-negative; an unrepresentable result MUST fail materialization. These controls MUST NOT synthesize a delay distribution, reorder turns, or incorporate live inference latency. The effective controls and delays MUST contribute to the frozen population identity.
A fixed output_tokens value MUST replace every target-derived turn limit. Otherwise a held-out assistant target MAY determine only that turn limit; its content MUST NOT enter runtime history, and a template containing a target shorter than two tokenizer tokens MUST be ineligible. A fixed value of one makes TPOT inapplicable to the population; every other admitted population makes TPOT applicable. A template whose first pre-template content-token count exceeds max_input_tokens MUST be ineligible and MUST NOT be truncated.
InferLab MUST deterministically order eligible templates from the Bench seed and catalog materialization identity and freeze enough for the largest resolved warmup-plus-profiling population before the first case. Cases MUST use prefixes of that population; warmup and profiling within a case MUST use disjoint sequential slices. Selection MUST be without replacement or wrap, and insufficient eligible templates MUST fail preparation.
Each admitted template instance MUST receive one x_correlation_id unique within the complete Bench execution, which is the runtime session identity used by InferLab and AIPerf. The template identity remains separate, and a turn identity is the runtime session identity plus its zero-based source-turn index. All turns MUST use the case-resolved chat-completions route, model, structured-message representation, and request fragment. Propagating x_correlation_id MUST NOT be presented as backend routing affinity unless the resolved serving capability establishes that fact.
Turn zero MUST contain the first source user message. On the valid path through the first session failure, turn n > 0 MUST begin only after turn n - 1 returns a valid live assistant response and the effective inter-turn delay, measured from terminal response receipt, has elapsed. Its history MUST contain every preceding source-user and live-assistant pair followed by source user message n. Source assistant targets MUST NOT replace live responses, and turns from one session MUST NOT overlap. A native turn emitted after an invalid predecessor belongs only to failed-session evidence, is not required to contain the missing assistant response, and cannot satisfy this dependency.
The frozen first turn MUST satisfy max_input_tokens before the measurement client starts. For every attempted turn, InferLab MUST measure the pre-template content-token count from the native structured-message request as the sum of tokenizer lengths of all message content with add_special_tokens=false. The count includes source-user and live-assistant content and excludes roles, server template delimiters, and the generation prompt. Because AIPerf owns dependent-turn dispatch, a later turn that exceeds max_input_tokens MAY already have reached the server; it MUST fail session adjudication, MUST preserve the native request and backend outcome, and MUST NOT be presented as a successful bounded session. InferLab MUST NOT drop, truncate, replace, or locally render the history.
A transport error or cancellation, malformed or missing terminal response, missing assistant text, model-selected action payload, or content-bound violation MUST fail the session at its first affected turn. Under output_stop = "eos", a turn whose response ends without assistant text remains a session failure. A native request emitted after that failing turn MUST remain failure evidence and MUST NOT restore session or case success. Other sessions MAY continue under AIPerf scheduling. A session succeeds only after every planned turn returns a valid text-only assistant response without a bound violation. A case deadline, interruption, client failure, or cleanup cancellation MUST fail affected sessions and preserve partial evidence.
A positive warmup session count MUST use a complete AIPerf-native session-bounded warmup phase with the same source-population policy, route, served model, request fragment, and dependent-session construction as profiling. The AIPerf-native phase marker allowed by RFC-0004:C-MEASUREMENTS MAY add or prepend a warmup-only system message at transport time; this is not a catalog-materialized source turn, MUST be preserved in the phase-tagged native request artifact, and contributes to the warmup request’s observed prompt-token count. Warmup MUST drain before profiling, use distinct session identities and population entries, and fail the case if any warmup session is unsuccessful or missing. Warmup traffic MUST be excluded from profiling metrics. Server-metric collection during a session warmup MUST follow the profiling-window exclusion requirements in RFC-0004:C-BENCH-CACHE-STATE.
The frozen profiling slice MUST determine planned session count and planned transport-request count, where the latter is the sum of template turn counts. Existing normalized latency, token, cache, throughput, and request-level SLO metrics retain transport-request semantics; request_throughput is not session throughput, and native session_num is a phase-local transport-request identity despite its name, and only the pair of benchmark phase and session_num is case-unique. A session-backed case succeeds only when every planned session succeeds, every planned turn has one terminal native request record, and all identities and counts reconcile. SLO evaluation occurs only after that completeness check, and the independent-request all-inference-error exception does not apply.
Only fixed linear source turns are supported. Executable environments, tool use, and source- or model-directed next actions require a separate contract.
Rationale: A linear session is a distinct load unit whose later requests depend on live responses. Source think time plus two scalar controls preserves trace shape, permits explicit compression or removal, and bounds long tails without importing AIPerf configuration or synthetic-delay models. One template identity, one runtime session identity, and native per-request evidence are sufficient to keep session scheduling and request metrics separate.
Since: v0.18.0
[RFC-0004:C-BENCH-PROMPT-AUTHORITY] Serving Bench Prompt Authority (Normative)
Section titled “[RFC-0004:C-BENCH-PROMPT-AUTHORITY] Serving Bench Prompt Authority (Normative) ”A random or random_mixture Bench request source MAY declare one prompt table whose kind is flat, rendered_chat, or server_chat. Omission of that table MUST resolve to flat, except on a random source that declares images under RFC-0004:C-BENCH-REQUEST-SOURCES, where omission MUST resolve to server_chat and a declared flat or rendered_chat kind MUST be rejected. A dataset request source and a dependent linear session MUST NOT declare that table and MUST resolve to server_chat. The resolved prompt authority MUST determine one protocol route and MUST be recorded with the effective Bench definition; independent endpoint selectors MUST NOT permit a second route choice. flat MUST synthesize ordinary text prompts on the completions route and MUST treat the sampled input-token count as the final transport-prompt target. rendered_chat MUST synthesize text content, place it in a declared message template, render that conversation locally using an explicit model chat-template authority, and submit the frozen rendered prompt on the completions route; exact token-length construction MUST apply to that final rendered prompt. server_chat MUST submit declared messages on the chat-completions route and delegate template rendering to the server. Under server_chat, the sampled input-token count MUST constrain message-content tokens before server rendering; implementations SHOULD use tokenizer metadata returned by the serving integration to compensate for fixed template overhead when that overhead is known exactly, MUST fall back to uncompensated content-token control when it cannot establish that overhead, and MUST record which accounting mode was used and why. Successful requests MUST retain the backend-observed input-token count when the endpoint reports it. Failure to load an explicit local template authority, deterministic rendering failure, or inability to construct the requested final rendered length MUST fail before measurement rather than silently selecting another prompt authority.
Rationale: Message semantics and final-token shape are different workload authorities. Defaulting an omitted synthetic prompt authority to flat makes declared input-token targets exact by default without hiding a model- or server-specific chat template. Server rendering remains explicit when production chat behavior is itself the workload, while datasets and live histories retain it by construction. Freezing either flat text or a locally rendered chat prompt gives controlled synthetic measurements one exact transport prompt without exposing protocol paths as independent experiment controls.
Since: v0.24.0
[RFC-0004:C-BENCH-PREFIX-GEOMETRY] Serving Bench Prefix Geometry (Normative)
Section titled “[RFC-0004:C-BENCH-PREFIX-GEOMETRY] Serving Bench Prefix Geometry (Normative) ”A random or random_mixture source whose prompt kind is flat or rendered_chat MAY declare one prefix_sharing table. That table MUST contain exactly one of shared_prefix_tokens or shared_prefix_ratio, MAY declare groups and popularity, and MUST reject any other key.
shared_prefix_tokens MUST be a non-negative integer no greater than the minimum possible selected input-token target. For every population entry it MUST resolve directly to that same shared-prefix token count.
shared_prefix_ratio MUST be finite and lie in the inclusive range from zero through one. For an entry with selected input-token target I, it MUST resolve to floor(I * shared_prefix_ratio) shared-prefix tokens. The resolved independently generated suffix length MUST be I minus that entry’s resolved shared-prefix length. A zero shared prefix and a zero independently generated suffix MUST both remain valid.
The Bench seed and resolved per-entry shared-prefix schedule MUST determine one canonical shared token stream long enough for the maximum resolved shared-prefix length in the frozen population. In a Bench that separates reuse by namespace under RFC-0004:C-BENCH-CACHE-STATE, the seed keeps every selection draw — each entry’s selected input and output lengths, mixture shape, resolved shared-prefix length, and prefix group — while each case phase’s namespace replaces the seed in every content draw: the canonical streams, the unique suffixes, corpus offsets, and shared system content; every phase thus has its own canonical streams over the same geometry. Each materialized final prompt MUST begin with the prefix of that canonical stream whose length is resolved for that entry. Content after that prefix MUST be generated independently for the entry. With a proportional declaration and variable input lengths, the resulting shared prefixes MUST therefore be nested prefixes of the same stream rather than unrelated per-length prefixes.
groups MUST be a positive integer and defaults to one. With G groups, the Bench seed MUST determine G independent canonical shared streams, one per group, the first of which is the single-group stream, so one group reproduces the ungrouped population. Each population entry MUST draw its group under the Bench seed, independently of its selected input length, and begin with the prefix of its group’s stream; entries of different groups share no controlled prefix. popularity MUST be either uniform, the default, or a table { zipf = s } with a finite exponent s greater than zero, which requires more than one group; the group of rank k, counting from one, is drawn with probability proportional to 1 / k^s, and uniform popularity is the exponent zero. Increasing the required population length MUST preserve the first N entries’ groups. A replay source MUST NOT declare groups or popularity.
For a fixed selected input length, a token declaration and a ratio declaration that resolve to the same shared-prefix token count MUST produce the same materialized population under the same effective prompt kind, template inputs, seed, and population length. Increasing the required population length MUST preserve the first N selected input targets, resolved shared-prefix lengths, and materialized prompts.
A flat source with a resolved shared-prefix length of zero MUST generate independent prompts from the first tokenizer-controlled payload token. A rendered_chat source MUST include its frozen template framing in the final common-prefix calculation. If the declared prefix geometry is incompatible with invariant template framing, or the final rendered prompts cannot satisfy both their exact input targets and resolved common-prefix lengths, population preparation MUST fail rather than weaken the declared geometry.
Omission of prefix_sharing MUST declare no controlled final-prompt prefix geometry. Materialized prompts MAY still share tokenizer special tokens or invariant rendered-template framing, and evidence MUST NOT describe those incidental tokens as a declared zero-cache-hit outcome.
A server_chat random source MAY instead declare one shared_system_content table containing exactly one positive integer tokens or one finite ratio strictly between zero and one. It MUST generate a canonical shared system-message content stream and an independently generated user message. For an entry with selected input target I, the token form MUST resolve to its declared fixed pre-template system-content length and the ratio form MUST resolve to floor(I * ratio). Definition resolution MUST reject any declaration for which the resolved system-content length is zero or is not strictly less than I for any possible selected input target; the unadjusted user-content length is I minus that resolved system-content length. This table describes source message content before server rendering; it MUST NOT be presented as an exact final-prompt prefix, combined with prefix_sharing, or used by the primed cache-start policy in RFC-0004:C-BENCH-CACHE-STATE.
Neither a declared shared-prefix length nor its ratio to a selected or observed input length is a requested cache-hit percentage. InferLab MUST preserve it as workload geometry and MUST use backend cache-read accounting governed by RFC-0004:C-BENCH-CACHE-STATE for observed reuse.
Rationale: Fixed tokens model a stable system or repository context across variable request sizes, while a ratio models proportional reuse and naturally produces nested prefixes for distributed ISL. Both declaration forms lower to one per-entry token schedule and one population authority. Inclusive endpoints represent independent flat prompts and full-prompt reuse without claiming that a backend must realize the same cache-read ratio.
Since: v0.24.0
[RFC-0004:C-BENCH-CACHE-STATE] Serving Bench Prefix-Cache State (Normative)
Section titled “[RFC-0004:C-BENCH-CACHE-STATE] Serving Bench Prefix-Cache State (Normative) ”A serving-Bench definition MAY declare one cache table whose start is uncontrolled, cold, or primed. Omission MUST resolve to uncontrolled. The former reset_prefix_cache field MUST be rejected rather than retained as a second cache-state authority.
For uncontrolled, InferLab MUST NOT reset or condition the prefix cache for the case. The record MUST NOT claim a known profiling cache state.
A static-matrix Bench whose request source is random or random_mixture and whose start is cold or primed separates reuse by namespace instead of resetting the server. Resolution MUST draw one nonce per Bench execution and carry it in the resolved plan, so dry-run, execution, and record show the same namespaces. Each case MUST read its own population: its warmup entries draw their content from the case’s warmup namespace and its profiling entries from its profiling namespace, each derived from the nonce, the Bench seed, the case id, and the phase, so no two phases executed in one run share a namespace; the seed alone keeps every selection draw under RFC-0004:C-BENCH-PREFIX-GEOMETRY. Such a case MUST NOT issue a prefix-cache reset, and its server need not provide a reset capability. Its cold start means that nothing the window can reuse exists when it opens; the cache may still hold unrelated entries. Its primed start conditions the canonical prefixes of the case’s profiling namespace. The record MUST carry the nonce and each case’s warmup and profiling namespaces. Replay and dataset sources, linear sessions, and adaptive searches keep the reset this clause describes; uncontrolled keeps one seed-keyed population that every case shares.
For cold on a Bench that does not separate reuse by namespace, the resolved target server MUST provide its server-specific prefix-cache reset capability. A server’s prefix-cache reset capability is either its public endpoint’s reset action or the per-target reset of every model-serving role under RFC-0006:C-INTEGRATIONS. A per-target reset MUST attempt its action on every model-serving replica of every role — every prefill and decode replica of a prefill_decode server, or the serve replica of a routed single server — even after an earlier target fails, MAY attempt them concurrently, and MUST succeed only when the reset of every target succeeds.
A reset attempt is declined when it receives a 2xx response other than HTTP 206 whose declared success predicate expects a JSON boolean and whose body holds the opposite boolean at the predicate’s pointer: the framework answered that it cannot reset yet, for example while KV blocks remain held for an unfinished transfer. Any other predicate mismatch, including an unparseable body, a missing value, or a mismatched string, is a failure, not a decline. InferLab MUST attempt a declined reset again — for a per-target reset, only on its declined targets; for a public reset, the whole action — after a bounded, increasing delay that never exceeds the remaining case budget, until the reset succeeds, an attempt fails in any other way, or the case budget is spent. A failure other than a decline, including a transport, deadline, redirect, or non-2xx outcome and HTTP 206, MUST NOT be retried. A reset still declined when the case budget is spent has failed. Evidence MUST preserve each attempt under RFC-0005:C-BENCH-PROMPT-CACHE-EVIDENCE. Any declared native runtime warmup MUST complete before InferLab invokes that reset. Profiling MUST begin after the successful reset without an intervening conditioning request.
For primed, the request source MUST use flat or rendered_chat, MUST declare prefix_sharing, and MUST resolve at least one positive shared-prefix length. Unless the case is namespaced, the resolved target server MUST provide its server-specific prefix-cache reset capability. Any declared native runtime warmup MUST complete before reset, or before conditioning in a namespaced case. After a successful reset where one applies, and before profiling, InferLab MUST complete recorded conditioning whose transport prompt is exactly the canonical shared token stream — of the case’s profiling namespace in a namespaced case — through the maximum resolved shared-prefix length. On a direct Engine endpoint, InferLab MUST issue one conditioning request per attention data-parallel rank of the public serving role, pinning each request to its rank through the framework’s rank-routing request header. When the integration declares a shared prefix cache under RFC-0003:C-SERVE-KV-STORE, the server has exactly one cache-owning target on a direct Engine endpoint and behind a Gateway alike, and InferLab MUST issue one conditioning request through the ordinary serving flow without rank pinning; after it completes, InferLab MUST wait the store’s declared or observed publication bound under RFC-0003:C-SERVE-KV-STORE before releasing profiling and MUST record that wait. Otherwise, on a Gateway-fronted endpoint, conditioning coverage is defined over the cache-owning targets: every model-serving prefill replica and every attention data-parallel rank of that replica. For a role without an effective parallelism shape under RFC-0003:C-SERVE-PARALLELISM, each replica’s attention data-parallel rank count is the count its Engine reported at readiness. When the serving shape resolves to exactly one such target, ordinary frontend routing cannot miss it, and InferLab MUST issue one conditioning request through the ordinary serving flow without rank pinning, exactly as on a direct Engine endpoint with one rank; a declared fan-out capability is not required for that shape. When more than one cache-owning target exists, InferLab MUST invoke the frontend’s conditioning fan-out capability, which MUST route one conditioning request through the ordinary serving flow for every model-serving prefill replica and every attention data-parallel rank of that replica, pinning each request to its rank; decode-side cache state produced by that flow is incidental and MUST be preserved as evidence rather than claimed as coverage. That prompt MUST contain no profiling entry unique suffix and MUST be locally verified to tokenize to exactly those canonical prefix tokens. With more than one prefix group under RFC-0004:C-BENCH-PREFIX-GEOMETRY, every conditioning obligation of this clause applies once per group, using that group’s canonical stream through the population’s maximum resolved shared-prefix length, and a publication wait begins after the last group’s conditioning completes. Conditioning MUST use the same served model, derived completions route, and complete effective request_body fragment as profiling, request exactly one output token, and ignore the generated response content. Inability to construct or verify that conditioning prompt MUST fail preparation. Conditioning MUST NOT consume a profiling population entry. A Bench whose cache start is primed or whose request source declares prefix sharing MUST be rejected at planning when the resolved server endpoint does not expose backend cache-read usage: the measurement’s cache-reuse evidence would otherwise be unobservable, and a completed run MUST NOT fail at normalization for a configuration that was knowable before launch. A primed start MUST be rejected before preparation when the public workload endpoint belongs to a Gateway whose serving shape resolves to more than one cache-owning target and whose selected backend does not declare the conditioning fan-out capability: ordinary frontend routing cannot address an individual replica or data-parallel rank, so conditioning could not cover every cache-owning rank. When the target set depends on a data-parallel rank count or a store access the server reports only at readiness, InferLab MUST apply the observed facts and make this rejection after the server is ready and before any reset or conditioning request, naming the observed counts; the primed-start publication wait is then the observed bound.
When a case has a positive AIPerf-native warmup, InferLab MUST hold profiling before its first credit, wait for warmup to drain successfully, then perform the selected reset and conditioning actions before releasing profiling. A case without native warmup MUST complete the selected reset and conditioning actions before its first profiling request. InferLab MUST NOT insert another prefix-cache reset between successful conditioning and profiling.
Warmup and conditioning requests MUST be excluded from normalized profiling counts, SLO evaluation, transport metrics, prompt-token distributions, cache-read metrics, and server-metric deltas. A profiling capture window MUST open after successful reset and conditioning and before profiling is released, so native warmup, reset, and conditioning activity stays outside it. Observing first-execution behavior therefore uses an ordinary case whose resolved warmup request count is zero, not a capture policy over warmup.
A missing required reset capability, failed reset, failed or incomplete conditioning request, lost profiling barrier, or interruption before profiling release MUST fail the case. InferLab MUST NOT intentionally release profiling before preparation succeeds. Any profiling request that escapes the barrier MUST be retained as invalid partial evidence and MUST NOT contribute to successful profiling counts or metrics. Evidence MUST preserve every completed or partial preparation action under RFC-0005:C-BENCH-PROMPT-CACHE-EVIDENCE.
The one case budget governed by RFC-0009:C-MEASUREMENT-CASE-BUDGETS MUST cover native warmup, prefix-cache reset, conditioning, profiling, and result handling. Cleanup MUST retain its separate grace.
A successful profiling request in a case with prefix_sharing, shared_system_content, or cache.start = "primed" MUST provide backend-reported prompt-cache-read tokens as well as backend-reported prompt tokens. An integration endpoint MAY declare that its backend protocol omits cache-read detail exactly when the reported value is zero; only for such an endpoint, that omission MUST be interpreted as the backend-defined representation of integer zero. Without that explicit endpoint capability, missing cache-read usage for any completed profiling request MUST fail successful-result normalization for that case.
For those observed requests, InferLab MUST expose prompt_cache_read_tokens and uncached_prompt_tokens distribution families, where each request’s uncached value equals its backend-reported prompt tokens minus its backend-reported prompt-cache-read tokens. A negative derived value MUST fail normalization. The distribution prefixes and percentile semantics MUST follow RFC-0004:C-MEASUREMENTS.
For a cold or primed case whose population InferLab generated with prefix_sharing, InferLab MUST record each observed profiling request’s reuse opportunity: its reusable tokens — the tokens of its shared prefix that an earlier request of its prefix group sent after the case’s prefix-cache reset — or, in a namespaced case, in the case’s profiling namespace — conditioning included, already sent — and its opportunity state when it was sent: first when no earlier request sent that prefix; available when some earlier request that sent it had produced its first output token at least the store’s declared or observed publication bound under RFC-0003:C-SERVE-KV-STORE earlier, zero without a store; and in_flight otherwise. Opportunity describes the workload and its timing, not a promised hit; an uncontrolled start has no defined earlier writers, so it records none. InferLab MUST expose available_miss_ratio, the share of available requests whose backend-reported prompt-cache-read tokens are zero, and available_reuse_efficiency, the summed prompt-cache-read tokens of available requests divided by their summed reusable tokens, and MUST omit both when no request is available. On a server with more than one cache-owning target, an available request may miss because routing sent it to a target that does not hold the prefix; only where the server has one cache-owning target do these measure whether the cache served what it could.
The existing prompt_cache_read_ratio MUST remain the profiling window total of backend-reported prompt-cache-read tokens divided by the total of backend-reported prompt tokens. It is an observed token-volume-weighted result. When this clause requires per-request cache-read usage, that one recorded integer population MUST be the normalized ratio authority; any AIPerf aggregate remains native corroborating evidence and MUST NOT replace those request observations. A Bench definition MUST NOT declare a target cache-hit or cache-read percentage.
Server-metric collection MAY accompany native warmup and cache preparation. Its normalized deltas and exported profiling window MUST exclude warmup, reset, and conditioning activity.
Rationale: Runtime warmup, cache reset, cache conditioning, prefix geometry, and observed cache reads answer different experiment questions. A closed start-state policy makes their order explicit while preserving AIPerf as the profiling engine and treating backend cache behavior as an outcome rather than a promise.
Since: v0.24.0
[RFC-0004:C-MEASUREMENT-AUTHORING-DISCOVERY] Measurement Authoring Discovery (Normative)
Section titled “[RFC-0004:C-MEASUREMENT-AUTHORING-DISCOVERY] Measurement Authoring Discovery (Normative) ”The version-matched InferLab CLI long-help surface MUST make the ordinary measurement-authoring path discoverable without requiring an installed agent skill, website, or source checkout. Across workspace show, recipe run, and bench long help, it MUST include accepted TOML examples for a built-in OpenAI smoke using its defaults, a fixed synthetic serving Bench that omits the default definition and prompt kinds, and an inclusive-uniform synthetic Bench whose range tables omit kind.
Every published configuration or invocation example in that help MUST be accepted by the same InferLab release and MUST name only supported override paths. The workspace show --json help MUST identify that output as the canonical merged workspace representation in which authoring defaults are explicit.
Bundled authoring references and website documentation MAY provide advanced prompt, dataset, session, cache, metrics, SLO, and profiling guidance, but they MUST supplement rather than replace the ordinary CLI-local examples.
Rationale: Operators need a short, version-aligned path from an empty definition to a valid ordinary measurement. Keeping those examples beside the parser and canonical workspace projection prevents an agent-only guide or a stale website from becoming the practical schema authority.
Since: v0.25.0
[RFC-0004:C-BENCH-AGENTIC-TRACE-REPLAY] Serving Bench Agentic Trace Replay (Normative)
Section titled “[RFC-0004:C-BENCH-AGENTIC-TRACE-REPLAY] Serving Bench Agentic Trace Replay (Normative) ”A static serving Bench definition MAY declare agentic_source instead of request_source or session_source. An agentic source represents a release-qualified population of recorded root and subagent session trees whose request geometry, parent-child dependencies, and timing are replayed for serving performance measurement. It MUST NOT be accepted by adaptive-serving and MUST NOT expose AIPerf scenario, dataset-loader, timing-mode, cache-bust, plugin, DAG, or arbitrary workflow configuration names.
An agentic_source MUST declare a release-qualified dataset and profile. The selected catalog entry MUST bind the public names to an immutable upstream dataset snapshot, expected content digest, source format, materialization semantics, and the upstream benchmark-script revision that the profile reproduces. A profile MUST bind one closed replay policy including its admissible context-length population, trajectory-start range, global-idle treatment, warmup policy, cache-bust policy, request construction, failure thresholds, minimum and default profiling duration, and required native artifacts. One InferLab release MUST NOT resolve the same catalog or profile name to different source bytes or replay semantics.
A definition MUST declare exactly one of request_source, session_source, and agentic_source. An agentic-backed definition MUST declare a non-empty list of positive concurrency values and MAY declare positive duration_seconds; omission of duration MUST resolve to the selected profile default, while an explicit value below the profile minimum MUST be rejected. It MAY use the ordinary Bench seed, timeout, server-metric selection, and served-model resolution; omission of the seed MUST resolve to the selected profile’s default seed. It MUST reject request rates, request counts, prompts-per-concurrency, sessions-per-concurrency, ordinary warmup counts, burstiness, prompt declarations, prefix-sharing and cache-start declarations, and open request-body fragments because the release-qualified profile owns those meanings. Invocation overrides MUST NOT change the selected source boundary, dataset, or profile.
The Bench seed MUST deterministically govern trace ordering, trajectory-boundary sampling, and every other replay choice assigned to randomness by the profile. Concurrency MUST mean the maximum number of admitted root session trees. A root tree occupies one lane from admission until its root and every spawned descendant have reached a native terminal state; parent suspension, recorded think time, and child execution do not release that lane. Descendant requests MAY therefore make instantaneous transport-request concurrency greater than the declared root-tree concurrency. When a tree drains during profiling, the native replay scheduler MAY admit the next trace selected by its release-pinned sequential sampling policy.
The immutable upstream trace snapshot MUST remain the authority for source topology, source coordinates, source token and cache-block geometry, and recorded timestamps. During client startup, the release-pinned AIPerf loader MUST acquire or reuse that qualified snapshot, validate and materialize it through its native dataset path, and classify invalid or unsupported traces before transport begins. Acquisition, loader, or materialization failure MUST fail the case without a transport request. InferLab MUST retain the qualified source identity, expected and observed source digest, resolved native configuration, and produced native run artifacts; it MUST NOT require AIPerf to publish an inputs.json or a separate materialized graph that the qualified Weka path does not produce, and MUST NOT reconstruct a second authoritative graph.
The release-pinned AIPerf AgentX replay capability MUST own runtime session-tree scheduling. For every admitted trace it MUST reproduce the source parent and subagent topology, admit descendants at their recorded spawn boundaries, suspend a parent at a blocking join, resume it only after the required descendants terminate, preserve per-stream request order, and keep one root lane until the whole tree drains. InferLab MUST NOT translate that topology into its own general DAG, operation graph, or workflow runtime.
The profile MUST use the trace materializer’s response-inclusive delta context. Each request history MUST therefore be reconstructed from the source-derived synthetic turn content and response geometry that establish the recorded prefix and cache relationships. A measured live server response MUST remain transport evidence and MUST NOT choose, rewrite, or semantically feed a later source turn. The Bench measures replay serving behavior; it does not claim successful execution of the source agent task or evaluate response quality.
Profiling MUST preserve each stream’s recorded request timing relative to its sampled trajectory boundary and preserve recorded inter-turn spacing and dependency order. The release-qualified profile MAY declare a per-trace idle-gap cap that compresses recorded idle periods within one trace before per-turn delays are derived; such a cap MUST be a recorded profile property and MUST NOT reorder requests or merge distinct streams. The profile MAY cap a period in which the complete replay would otherwise be globally idle, but it MUST shift all pending timers together. Beyond these profile-declared caps, InferLab MUST NOT apply an unrecorded per-trace think-time cap, reorder timers, or replace the trace timeline with an InferLab delay distribution. Records MUST preserve the source-timing and idle-cap policies plus native source coordinates and observed request timestamps; they MUST NOT claim that the release-pinned AIPerf reports each internal timer shift or sampled trajectory boundary when it does not.
Before profiling, AIPerf MUST synthesize one snapshot warmup from the same trajectory population and boundary choices. Every stream active before the boundary MUST replay its last eligible pre-boundary request with its reconstructed prefix; streams that begin at or after the boundary need no snapshot request. When the profile declares additional cache-pressure warmup, it MUST declare it in exactly one closed form — a warmup duration or a per-lane warmup request budget — and the same live trajectories MUST continue without recorded idle waits and with the profile’s bounded warmup output until the declared budget is spent, drain under the profile grace policy, and then resume profiling from the resulting state with residual next-request delay preserved. Warmup traffic and warmup failures MUST remain separate from profiling metrics and MUST be preserved as warmup evidence. The release-pinned AIPerf owns the warmup abort decision: a failed or cancelled root-stream warmup request, whether a snapshot or a cache-pressure request, MUST abort the case before profiling, and a descendant warmup request failure MUST NOT by itself abort it. InferLab MUST NOT add a warmup abort criterion of its own or relax the native one; after warmup, case acceptance remains governed by the native scenario outcome and its failure threshold. Evidence MUST be limited to the effective policy, emitted warmup records, native phase outcomes, and other fields actually exported by the release-pinned AIPerf artifacts; it MUST NOT fabricate internal handoff state.
Profiling MUST be duration-driven. At the duration boundary, native scheduling MUST stop admitting new work and finalize or classify already admitted trees according to the release-pinned replay contract and ordinary case deadline. A tree or branch classified by AIPerf as truncated solely because the requested profiling window ended is an observable duration outcome and MUST NOT be silently counted as a completed tree or converted into an unrelated transport failure. Cancellation, deadline expiry, missing terminal evidence, or scheduler failure remains a case failure.
The profile MUST use streaming chat-completions requests, server-owned chat rendering, and source-defined output limits, and MUST retain its native cache-bust and scenario-invariant policy. A case succeeds only when AIPerf exits successfully, its native scenario outcome reports a valid submission, the required aggregate artifact including native branch statistics and, at the diagnostic artifact level governed by RFC-0004:C-BENCH-ARTIFACT-LEVEL, the required raw-request artifact are present, and InferLab can reconcile the exported fields. InferLab MUST preserve AIPerf’s profiling failure-rate and context-overflow decisions and counts and MUST NOT recompute a second acceptance threshold. An early native failure-threshold abort, an invalid scenario outcome, or a required-artifact or reconciliation failure MUST fail the case.
Normalized latency, token, transport-request throughput, and optional server metrics MUST retain the meanings of the measured native transport requests. They MUST NOT be relabeled as root-session, agent, or task-completion metrics. Agentic summaries MUST separately preserve the branch, join, warmup, ordinary-failure, context-overflow, and duration outcomes that the native artifacts actually report. Request-level SLO and adaptive-rate search semantics do not apply to agentic_source in this RFC version.
A dry-run MUST resolve the public source and profile, catalog identity, cache path and state, effective replay policy, concurrency cases, seed, duration, endpoint, and planned native invocation without downloading missing data or claiming materialized topology, warmup, runtime, or outcome facts that were not observed.
Rationale: SemiAnalysis AgentX is a concrete duration-driven serving workload whose root and subagent trees, timing, and synthetic cache geometry already have a release-pinned native scheduler in AIPerf. A closed agentic_source keeps that operator workflow distinct from finite live-response linear sessions while avoiding an InferLab-owned DAG language, avoiding direct dependence on a serving engine’s benchmark client, and staying within the release-pinned AIPerf’s qualified public artifact boundary. The profile reproduces the cataloged InferenceX replay revision using only replay parameters that the release-pinned AIPerf accepts, so that default InferLab runs remain comparable with submissions produced by that revision, and warmup evidence stays evidentiary rather than becoming a second acceptance authority.
Since: v0.26.0
[RFC-0004:C-MEASUREMENT-DATA-ASSETS] Measurement Data Asset Preparation (Normative)
Section titled “[RFC-0004:C-MEASUREMENT-DATA-ASSETS] Measurement Data Asset Preparation (Normative) ”A measurement data asset is non-synthetic source content selected by an effective Eval task or Bench source and needed to construct measurement inputs. The effective task, Bench definition, or release catalog that selected the content MUST remain the sole authority for dataset choice, subset, split, filtering, request construction, and scoring semantics. Preparation MUST NOT introduce a second dataset selector or let a common asset declaration override that authority.
InferLab MUST distinguish three identities. A preparation-attempt identity identifies one effective source selection before external effects begin. An acquired-source identity identifies the resolved content closure made locally available by that attempt. A consumer-materialization identity identifies the lm-eval task data, AIPerf-native representation, or InferLab-frozen population later derived by the owning measurement execution. A revision, content digest, framework fingerprint, derived-cache key, and frozen-population digest MUST NOT be presented as interchangeable identities.
One preparation owner MUST use one preparation attempt for all consumers whose pre-effect selection keys are equal. That key MUST be established before the first source-resolution or acquisition external effect and MUST include the selection authority, acquisition provider and release-pinned acquisition-runtime identity, locator, requested revision or alias, subset, split, filter, data-file selection, local path, and release-catalog identity whenever the field is already selected by effective configuration or catalog data without that effect. Fields discoverable only by the owning resolver after the attempt begins MUST be preserved in effective-selection and acquired-source evidence and MUST NOT retroactively change the attempt identity. Unequal pre-effect selection keys MUST remain separate attempts even if they later resolve to the same effective selection, bytes, or physical cache.
Source preparation is a separately owned, intentionally unbounded, operator-interruptible operation under RFC-0009:C-OPERATION-BUDGETS. It begins immediately before the first source-resolution or acquisition external effect and terminates when every planned attempt is source-ready, one attempt fails, or the operator interrupts it. An external attempt MAY retain a source- or tool-owned responsiveness cap, but source preparation MUST NOT gain an arbitrary overall timeout.
Before a recipe launches any serving process, InferLab MUST prepare every source selected by its workload suite, including a source for a measurement that a later Eval gate may skip. A manual Bench targeting an existing server MUST perform the same source preparation before its first transport request. Recipe preparation failure MUST fail the recipe before serving launch. Manual-Bench preparation failure MUST fail the Bench without stopping, restarting, or otherwise changing its target server.
For a release-qualified source, source-ready means that all declared remote and local source bytes are available, the complete catalog-owned content closure is immutably identified and bound, and every required integrity check succeeded. For a task-owned or operator-owned source, preparation MUST establish that same closed result when the owning interface supports it. When that interface cannot expose or bind a complete closure, preparation MAY instead declare the source opaque and ready only for a non-reproducible execution. An opaque source MAY perform further file access or remote acquisition during consumer materialization; evidence MUST state that preparation did not establish complete local availability. Source preparation MUST NOT send an inference request.
Consumer materialization MUST follow its existing measurement owner rather than being assigned uniformly to a case. Eval task and tokenizer materialization remain Eval-case actions under RFC-0009:C-MEASUREMENT-CASE-BUDGETS. Independent-request and linear-session frozen populations remain owned once by the Bench child before its first case, and every case MUST reference the shared population under RFC-0004:C-BENCH-REQUEST-SOURCES or RFC-0004:C-BENCH-LINEAR-SESSIONS. That Bench-wide materialization is a separate intentionally unbounded, operator-interruptible operation beginning immediately before its first materialization effect and terminating when the population is ready, materialization fails, or the operator interrupts it; it MUST NOT consume an arbitrary case’s timeout. AgentX native validation and materialization remain client-startup actions of each AIPerf case and consume that case budget under RFC-0004:C-BENCH-AGENTIC-TRACE-REPLAY and RFC-0009:C-MEASUREMENT-CASE-BUDGETS.
A materialization failure MUST retain the classification of its owner. Eval or AgentX case-owned failure is a case failure. An independent-request or linear-session shared-population failure is a pre-case Bench-child failure. A prelaunch source-preparation failure MUST NOT be relabeled as either materialization failure.
Preparation MUST delegate source-specific resolution and acquisition semantics to the authority selected by the effective source. Hugging Face-backed content MUST use the applicable release-pinned Hugging Face acquisition runtime. A release catalog entry with an explicit URL and digest MUST continue to use the InferLab content-addressed acquisition governed by RFC-0004:C-BENCH-REQUEST-SOURCES. A release catalog entry that names a release-pinned AIPerf public dataset and a digest MUST use the same content-addressed acquisition, with that AIPerf release’s public-dataset materialization in place of the download; the materialization runs during source preparation, before any serving process launches or any transport request is sent, and InferLab MUST NOT read the AIPerf cache it populates. InferLab MUST NOT parse private Hugging Face, lm-eval, or AIPerf cache layouts or make one tool consume another tool’s derived cache representation.
A source selected by an InferLab release-qualified catalog MUST resolve to the catalog’s immutable identity, close every remote content dependency declared by that catalog, and satisfy every expected digest before preparation succeeds. Failure to acquire, bind, or verify that complete closure MUST fail preparation. It MUST NOT fall back to a mutable alias or opaque execution.
For task-owned or operator-owned remote content, preparation MUST resolve a mutable selector to an immutable identity when the owning interface provides one and MUST bind the consumer to that identity when supported. The execution MAY be classified as reproducible only when the complete remote content closure actually consumed is immutably identified and bound. An immutable repository revision is insufficient when a loader obtains additional unbound remote content. When complete immutable binding is unavailable, execution MAY proceed only through the opaque non-reproducible path with a specific reason.
When the owning resolver can enumerate and bind the complete regular-file closure of task-owned or operator-owned local content, InferLab MUST snapshot those files into an immutable prepared location before serving launch or the first manual-Bench request and MUST make the consumer read that snapshot. The acquired-source identity MUST preserve an ordered list of normalized source-root-relative paths and the SHA-256 digest of each file; InferLab MUST NOT invent an aggregate directory digest with semantics the owning resolver did not provide. When the resolver cannot enumerate or bind the complete closure, the owning consumer MAY read the original local source only through the opaque non-reproducible path; evidence MUST preserve the unresolved original path and missing-closure reason and MUST NOT claim an immutable prepared snapshot. Local paths remain subject to the unredacted local-record policy of RFC-0005:C-EVIDENCE.
Eval and Bench preparation in one execution MUST receive identical explicit HF_HOME, HF_HUB_CACHE, HF_DATASETS_CACHE, and XDG_CACHE_HOME bindings when those values are operator-selected. Unset-path derivation MUST be delegated to each applicable release-pinned Hugging Face runtime rather than reimplemented by InferLab. Evidence MUST preserve every effective source and dataset cache store reported by the acquisition runtime for each attempt. A verified immutable byte-cache hit MUST avoid downloading those source bytes, but metadata access MAY still occur and MUST remain distinguishable from byte acquisition.
Hugging Face source caches, Hugging Face dataset caches, lm-eval and AIPerf derived caches, and InferLab’s digest-addressed release cache MUST remain independently owned physical stores. Per-store outcomes MUST distinguish full hit, miss, partial reuse, and unavailable observation, and MUST distinguish remote metadata access from source-byte download or reuse. Successfully acquired cache content is recoverable machine-local state and MUST NOT be rolled back solely because a later preparation, serving, Eval, or Bench step fails.
Dry-run MUST inspect the effective selection, already-local source state, and existing materializations without external access, download, cache mutation, or persisted materialization. It MAY compute deterministic materialization facts purely from locally available inputs when doing so has no external or persisted side effect, preserving the existing dry-run obligations of RFC-0004:C-BENCH-REQUEST-SOURCES. It MUST otherwise report planned attempts and unavailable identities or outcomes rather than claim successful resolution, acquisition, verification, reproducibility, or materialization.
Rationale: Eval and Bench retain different source and materialization semantics, but both need acquisition failure and reproducibility evidence settled before recipe-owned serving resources are occupied. One source-preparation boundary supplies that invariant without replacing the distinct Eval-case, Bench-wide population, and AgentX-case materialization owners.
Since: v0.27.0
[RFC-0004:C-BENCH-ARTIFACT-LEVEL] Serving Bench Artifact Level (Normative)
Section titled “[RFC-0004:C-BENCH-ARTIFACT-LEVEL] Serving Bench Artifact Level (Normative) ”A serving-Bench definition MAY declare artifact_level as performance or diagnostic. Omission MUST resolve to diagnostic. The effective artifact level MUST be preserved in canonical merged workspace output, dry-run, execution, and record evidence whether it was declared or defaulted.
At the diagnostic level, InferLab MUST request the measurement runtime’s full per-request raw request/response export for the case. At the performance level, InferLab MUST NOT request that raw export and MUST still obtain the runtime’s normalized per-request records and summary export. The artifact level MUST NOT alter the effective request population, load shape, warmup policy, request-body fragment, endpoint selection, server-metric collection, or normalized metric semantics of the case.
Warmup success and failure validation and population-identity reconciliation MUST be evaluated from the normalized per-request records, which carry the native benchmark-phase marker for the warmup and profiling phases, and MUST NOT require the raw request/response export under either artifact level.
A performance case with a session_source or agentic_source remains a valid case. Evidence dimensions that depend on the raw request/response export degrade under RFC-0005:C-BENCH-LINEAR-SESSION-EVIDENCE and RFC-0005:C-BENCH-AGENTIC-TRACE-EVIDENCE; the degradation MUST be recorded and MUST NOT fail the case by itself.
Rationale: Full raw request/response retention is a diagnostic artifact with material host and export cost on long-output workloads, while normalized records already carry the phase and identity markers that case validation needs. Making raw capture an explicit level keeps ordinary performance runs lean without weakening diagnostic capability, and degradation recording keeps session and agentic evidence honest when the raw artifact is absent.
Since: v0.29.0