Skip to content

Specification · RFC

RFC-0005: Evidence And Operator Journal Contract

Version: 0.48.0 | Status: normative | Phase: spec Owners: @Zihua Wu

References: RFC-0004


[RFC-0005:C-SUMMARY] Summary (Informative)

Section titled “[RFC-0005:C-SUMMARY] Summary (Informative) ”

This RFC governs durable evidence: readable file-first records whose immutable resolved object is shared by dry-run and execution, process-keyed runtime evidence, unredacted local values, and the operator scratchpad journal on the same time axis. Server and image workflows are defined by RFC-0003 and RFC-0007. RFC-0001 holds the specification overview and authority map.

Since: v0.1.0


[RFC-0005:C-EVIDENCE] Records And Evidence (Normative)

Section titled “[RFC-0005:C-EVIDENCE] Records And Evidence (Normative) ”

InferLab MUST create a durable local execution record after resolution is frozen and before the first state-affecting external command of a non-dry-run server, recipe, manual Bench, or image workflow, and of a recorded run under RFC-0002:C-ADHOC-EXECUTION.

Record directory identifiers MUST be readable and use these shapes when the corresponding fact exists:

  • <UTC>-serve-<server>[-<case>]-<pid>;
  • <UTC>-recipe-<recipe>[-<case>]-<pid>;
  • <UTC>-bench-<bench>-<pid>;
  • <UTC>-image-<image>-<pid>;
  • <UTC>-run-<stack>-<pid>.

A server record directory MUST NOT include one role name because one record may own several roles. Per-process log filenames MUST include the process identity.

Every server or recipe record MUST contain one immutable resolved object produced by RFC-0003:C-RESOLUTION. Dry-run MUST present the same resolved shape without claiming execution. Runtime code MUST NOT mutate or reconstruct that object after freezing it.

A direct serve start record MUST preserve a null recipe identity. A recipe record MUST preserve recipe, workload suite, server, and selected case identities. A server with base behavior and no selected case MUST preserve a null case identity.

The resolved server evidence MUST preserve producing InferLab version, record schema, workspace and dirty-source identity, stack and realization identity, model and served name, topology, selected case and selection source, typed effective behavior, ordered behavior declarations, explicit overrides, placement and its selection source, integration and protocol identities, planning and rendering request and response digests, Engine and frontend component links, independent Gateway and P/D Router backend identities, frontend component bindings, rendering sources, P/D routing policy, server-control capabilities, public endpoint including the optional integration-declared server-metrics path and public-or-named-port binding, the exact control-plane-resolved server-metrics URL when present, and cleanup plan when those facts apply.

The resolved server topology MUST preserve every model-serving role hierarchically as role, replica, then rank and MUST preserve one closed frontend section when Gateway is present. That section MUST contain the Gateway component record, the P/D Router component record when applicable, the derived frontend processes, and each component-to-process binding. Each concrete allocation and lifecycle outcome MUST appear once; a fused frontend MUST reference its one process from both component records rather than duplicate it.

Each resolved model-serving role MUST preserve role identity, declared and effective replica counts, declared and complete effective settings, declared and complete effective parallelism, declared devices with its source when present, whole-replica device count, readiness and endpoint contracts, and profiling capability or preparation.

Each resolved frontend component record MUST preserve component identity, selected backend identity, implementation identity and observable version, complete effective settings, rendering source, readiness, links, controls, endpoint semantics, and the process identity that realizes it. P/D Router absence MUST be explicit when it does not apply.

The ordered behavior declarations MUST be the sole provenance for server, selected-case, and invocation inputs. Each declaration MUST identify its source and preserve its common and role-scoped inputs. Resolved effective fields MUST NOT duplicate that information in per-field source values or source maps; integration defaults and adjustments MUST remain observable from the declared and effective values.

Each resolved model-serving replica MUST preserve its role-local identity, routing or serving-entry semantics, and ordered ranks.

Each resolved model-serving rank MUST preserve process identity, rank index and count, machine, ordered devices, model locator and whether it came from the machine-specific map or fallback, endpoint, allocated ports, runtime-cache inputs and path, launch context, dependencies, exact command and environment, launch files, readiness, and container identity when those facts apply.

Each resolved frontend process MUST preserve process identity, canonical process role, closed component binding, machine, empty devices, absent model locator, public or internal endpoint semantics, allocated ports, runtime-cache inputs and path, launch context, dependencies, exact command and environment, launch files, readiness, and container identity when those facts apply. A resolved discovery process MUST preserve the same facts as a frontend process, with process role discovery, component binding [discovery], and its assigned data directory.

Runtime evidence MUST be stored separately from resolved allocation and keyed by process identity. It MUST preserve logs, durable runtime handles, spawn and readiness outcomes, hardware observations, control actions, failures, termination, and cleanup. It MUST NOT repeat machine, device, locator, endpoint, command, or dependency facts already authoritative in the resolved allocation.

A long-running server record MUST preserve its lifecycle state until stop or recovery finalizes it. Server status, logs, stop, and recovery MUST be executable from the record without reloading workspace or local configuration.

For every machine that hosts a serving process, execution evidence MUST preserve the hardware actually assigned at launch: each assigned device’s model name, memory capacity, and device UUID, plus driver version. The probe MUST run through the same launch path as serving processes. A failed probe MUST fail before serving process launch naming machine and error.

The same probe MUST read the compute applications each assigned device reports. When any assigned device reports a compute application, launch MUST fail before any serving process spawns, whichever user or tool owns that application and whether or not its process still lives, naming the machine and, for each such application, the device, process ID, process name, and reported memory. A launch that waits for devices under RFC-0003:C-RUNTIME-WORKFLOWS waits instead of failing. A machine that hosts only zero-device processes has no assigned device to check.

Profiling evidence MUST additionally preserve capture targets and their role/replica/rank bindings, profiler identities, storage, capture plans, semantic windows, range mappings, control actions, collection finalization, report references, verification, and failures. A control action, capture-window or server-control, that declares a success predicate under RFC-0006:C-INTEGRATIONS MUST preserve the predicate and the observed value alongside its response status.

For every attempted framework-range HTTP control action, profiling execution evidence MUST preserve the control process identity, operation, declared method and path, effective URL, request-body presence and complete object contents when present, received response status or categorized serialization, transport, deadline, or invalid-response failure, timing, and success outcome.

Eval evidence MUST additionally preserve execution kind, effective request, result metrics, raw artifacts, failure, and skip reason. Built-in OpenAI smoke evidence MUST identify its kind, its effective vision mode selection, and InferLab version and MUST NOT claim that lm-eval or a Bench runtime participated. lm-eval evidence MUST preserve host platform, toolchain manifest and lock, runtime and runner identities, invocation, logs, and process outcome.

When a recipe applies measurement overrides, each measurement child record MUST preserve the declared definition, complete effective definition, and ordered override provenance.

Bench evidence MUST preserve host platform, toolchain manifest and lock, AIPerf and runner identities, adaptive and normalization policy identities, exact cases, the effective artifact level governed by RFC-0004:C-BENCH-ARTIFACT-LEVEL, resolved AIPerf configuration, native invocation, load-unit and transport-request counts, normalized metrics, produced raw artifacts, logs, process and cleanup outcomes, and failures. When server metrics are requested, Bench evidence MUST always preserve the declared selection. After server-metrics capability resolution succeeds, it MUST preserve the logical metrics path, logical public-or-named-port binding, and exact resolved URL. After the measurement client is launched, it MUST preserve the native server-metrics configuration, invocation, and process outcome. Every produced native artifact MUST be preserved. A required artifact that is not produced MUST be represented by the case failure and MUST NOT be fabricated as an artifact reference. When SPEED-Bench acceptance computation applies and is attempted, evidence MUST preserve each acceptance value and the source it was taken from: per-request records, the run summary, or the scraped server metrics. Workload record schema version 23 introduces output_stop evidence, AIPerf public-dataset source evidence, and the SPEED-Bench acceptance value-and-source evidence, and removes the SPEED-Bench report invocation and CSV evidence; an implementation MUST NOT encode them under an earlier workload record version.

A workload record MUST carry exactly one tagged evidence variant, eval or bench, as its sole workload-kind authority. The Eval variant MUST contain only Eval cases and Eval-only result evidence; the Bench variant MUST contain only Bench cases, exactly one of request_source evidence governed by RFC-0005:C-BENCH-REQUEST-SOURCE-EVIDENCE, session_source evidence governed by RFC-0005:C-BENCH-LINEAR-SESSION-EVIDENCE, or agentic_source evidence governed by RFC-0005:C-BENCH-AGENTIC-TRACE-EVIDENCE. The optional adaptive summary MAY appear only for request-source adaptive cases and MUST be absent from session-backed and agentic-source Bench evidence. A case MUST keep process, timing, paths, native invocation, produced raw artifacts, and terminal error as common evidence while storing Eval gate, trial, task-metric, and native Eval outcome fields only in Eval evidence, and load, request-count, normalization, population-slice, prefix-cache-reset, and SLO fields only in Bench evidence.

A failed or skipped case MUST omit a normalization schema and other result-derived fields unless an accepted runner result established those values. An empty placeholder MUST NOT claim that a result schema, request count, or normalized result existed.

Workload record schema version 7 introduces this tagged evidence shape. Typed workload-record readers MUST reject earlier schema versions rather than infer case kind from legacy optional-field combinations. Workload record schema version 8 introduces the linear-session source-evidence alternative; an implementation MUST NOT encode it under version 7. Workload record schema version 12 introduces the agentic-source evidence alternative; an implementation MUST NOT encode it under an earlier version. Workload record schema version 20 introduces the optional image-decoration evidence member on random request-source evidence under RFC-0004:C-BENCH-REQUEST-SOURCES and the effective vision-mode selection on built-in OpenAI smoke evidence under RFC-0004:C-MEASUREMENTS; an implementation MUST NOT encode either under an earlier version. Server record schema version 5 and workload record schema version 9 introduce the logical server-metrics port binding and exact resolved URL representation. Server record schema version 8 introduces the optional synthetic-acceptance evidence member under RFC-0003:C-SERVE-SYNTHETIC-ACCEPTANCE; an implementation MUST NOT encode it under an earlier server record version. Server record schema version 9 introduces the auxiliary-model evidence member under RFC-0003:C-SERVE-AUXILIARY-MODELS — the declared kind-to-model mapping and each effective auxiliary weight locator with its kind and locator provenance; an implementation MUST NOT encode it under an earlier server record version. Server record schema version 10 introduces the optional model fallback-locator evidence member — the weight binding’s shared locator, recorded so controller-side measurements can resolve it from the record alone; an implementation MUST NOT encode it under an earlier server record version. Server record schema version 11 introduces the optional device-residual evidence member — the per-device outcome (freed, residual held with bytes, or probe unavailable) of the post-cleanup compute-memory probe on the record’s assigned devices; an implementation MUST NOT encode it under an earlier server record version. Server record schema version 12 records, beside those per-device outcomes, the settle window within which a device still holding memory was probed again before cleanup was judged; it also drops the per-process copies of the server readiness budget and attempt timeout, whose single resolved authority is the server; an implementation MUST NOT encode either change under an earlier server record version. Typed readers MUST reject any unsupported record version rather than infer the new endpoint representation from a path-only record or project session data into request-source fields. Summary-only readers MAY continue reading the legacy top-level kind, definition_id, and status fields without interpreting legacy cases; InferLab MUST NOT rewrite or silently coerce legacy mixed case records into the current typed shape. Server record schema version 13 introduces the discovery process with its data directory and the registry-membership readiness evidence with its last observed missing targets and unexpected entries under RFC-0006:C-INTEGRATIONS; an implementation MUST NOT encode them under an earlier server record version. Server record schema version 14 introduces the ownership marker on process-group runtime handles and the marked-process sweep in each cleanup outcome under RFC-0003:C-RUNTIME-WORKFLOWS — the marker, the marked processes found, the termination and kill actions, any marked process that remained, and whether the sweep completed with its failure context; it also records, in the resolved placement, the unavailable pool ports and pool devices observed on each machine that received a process, each with its holder, under RFC-0002:C-LOCAL-PLACEMENT, and, exactly when a launch waiting for devices under RFC-0003:C-RUNTIME-WORKFLOWS found occupied devices, how long it waited and the occupants it first found, including for a wait that ended by interruption or probe failure; an implementation MUST NOT encode any of them under an earlier server record version. Server record schema version 14, recipe record schema version 5, and workload record schema version 24 name, beside the workspace identity, each local workspace definition file composed into the run under RFC-0002:C-WORKSPACE-AUTHORITY, as its workspace-relative path and the SHA-256 digest of its bytes, in path order, the member being absent when none composed; a record naming one or more MUST NOT claim that the workspace revision alone reproduces the run; an implementation MUST NOT encode them under an earlier version of its record. Server record schema version 15 and workload record schema version 25 record each cleanup completion wait under RFC-0009:C-COMPLETION-WAITS — process and process-group exit after signalling, container disappearance after removal, and device-memory release after cleanup — with its stall window, attention ceiling, attempt bound, the progress signals used and unavailable, failed observations, the time of last observed progress, what remained, and how it ended, in place of the fixed post-kill grace, container-removal deadline, remote cleanup deadline, and device-residual settle window; a container removal also records why its removal request did not itself complete; an implementation MUST NOT encode them under an earlier version of its record. Server record schema version 16 and recipe record schema version 6 record each model-serving role’s declared devices with its source under RFC-0003:C-SERVE-PARALLELISM each layer’s declared gateway_settings under RFC-0006:C-INTEGRATIONS, the server’s declared KV store, KV store plan, and store process, and each serve-state probe attempt made during readiness with its per-process answers, under RFC-0003:C-SERVE-KV-STORE; an implementation MUST NOT encode it under an earlier version of its record. Workload record schema version 26 records, for a server with a serve-state probe, each Bench case’s per-process probe answers at the start and at the end of its profiling window and whether the case lost its store; an implementation MUST NOT encode them under an earlier workload record version. Server record schema version 17 and recipe record schema version 7 record, for a role without an effective parallelism shape under RFC-0003:C-SERVE-PARALLELISM, its device count and each Engine’s data-parallel rank count as observed at readiness; for a KV store whose access is observed, each process’s reported access and publication bound in every serve-state probe answer; and for a filesystem KV store, each Engine’s bound tier directory under RFC-0003:C-SERVE-KV-STORE. Workload record schema version 27 records, for a Bench that separates reuse by namespace under RFC-0004:C-BENCH-CACHE-STATE, the namespace nonce in the resolved plan, each case’s warmup and profiling namespaces in its population preparation request, and each case’s own population, and records no reset for its cases; it also records serve-state probe answers with each process’s reported access and publication bound, and each Bench plan’s publication wait and cache-owning targets as applied from the facts the server observed at readiness under RFC-0004:C-BENCH-CACHE-STATE; an implementation MUST NOT encode these under an earlier version of its record. Server record schema version 18, recipe record schema version 8, and workload record schema version 28 record the adapter protocol 16 plans they resolve, whose KV-transfer mechanism is a name under RFC-0003:C-SERVE-TOPOLOGY; an implementation MUST NOT encode them under an earlier version of its record. A record finalized by serve stop or recipe stop under RFC-0003:C-RUNTIME-WORKFLOWS while its controller was gone MUST record that an operator stopped it and whether --force overrode a held controller lock. Workload record schema version 21 introduces per-target prefix-cache reset evidence under RFC-0004:C-BENCH-CACHE-STATE and the success predicate with its observed value on prefix-cache reset and capture-window actions; an implementation MUST NOT encode them under an earlier workload record version. Workload record schema version 22 records every attempt of a public prefix-cache reset, including declined attempts retried under RFC-0004:C-BENCH-CACHE-STATE, and admits a boolean expected value in a recorded success predicate; an implementation MUST NOT encode either under an earlier workload record version.

Residual compute memory is memory on an assigned device held by a process that no longer exists on the launch machine, or exists only as a zombie. Memory held by a live process is not the stopped server’s residual; a process that exists on the launch machine MUST count as live even when the probing user’s process table hides it. Held memory the device does not attribute to an identified process MUST make that device’s outcome probe unavailable rather than residual held.

A run record MUST preserve the selected stack and its locked environment identity, the workspace revision, dirty marker, and source-content digest, the exact argv, the provided InferLab context and device projection as the environment InferLab set, the server record and its lifecycle state at start and at command exit when --serve applies, the model identifier and resolved locator when --model applies, the parent record when nested, the command’s stdout and stderr, start and end times, exit status, and a terminal status of succeeded, failed, or interrupted. It MUST state that the context was provided and MUST NOT claim that the command used it. It MUST NOT record the operator’s inherited environment beyond the variables InferLab set. It MUST NOT claim outputs, results, or qualification. When a recorded run is interrupted by a signal, it MUST terminate, within the cleanup grace, the process group it started and the process groups of the command’s live descendants, and MUST finalize the record as interrupted; the cleanup evidence MUST preserve its trigger, the termination grace applied, the signaled groups, the wait until the process tree exited under RFC-0009:C-COMPLETION-WAITS, elapsed time, and verification, and the record MUST state that processes that left that process tree and containers started through a daemon are not cleaned up by InferLab.

A manual Bench record MUST preserve the target server record, its producer version and serving snapshot, and the invoking InferLab version and measurement workspace snapshot that supplied the Bench and overrides.

Image records MUST preserve the immutable resolved image plan, build inputs, native builder commands, content closure, package and assembly evidence, immutable image identity, inspection and export outcomes, stack check results, validation references and outcomes, partial failures, and cleanup under RFC-0007:C-IMAGE-BUILD.

Dry-run and records MUST preserve actual effective model locators, machine hosts, endpoints and ports, device assignments, command arguments, environment values, and local paths when those facts apply.

InferLab MUST NOT perform value-level redaction or maintain a separate shareable record projection. Operators are responsible for access control, retention, and sharing of local records.

A dirty-workspace record MUST preserve baseline revision, dirty marker, effective source-content digest, and non-reproducible status. It MUST NOT claim that the baseline revision alone reproduces execution.

A finalized record MUST remain inspectable without the original workspace, original machines, or rerunning the operation. InferLab records need not remain compatible with record schemas or directory layouts predating this contract.

Rationale: Process cleanup verifies the server’s own processes gone — its process group and every process carrying its ownership marker — so live memory is not attributed to the stopped server; a process outside that ownership, such as one that cleared its environment, is not attributed either. The immutable resolved object is the authoritative statement of intended execution, while process-keyed evidence reports what happened. The model role → replica → rank hierarchy keeps heterogeneous Engines readable; the closed frontend binding records Gateway and P/D Router semantics without duplicating a fused process allocation. Readable record identifiers answer which named workflow ran without pretending a multi-component record belongs to one role. Refusing occupied devices at launch keeps a framework from sizing its memory against another process’s allocation or measuring while sharing the device, which would leave a record that silently misstates the conditions it measured; the device’s own process list is the one authority that covers every occupant, including ones InferLab did not start.

Since: v0.1.0

[RFC-0005:C-SCRATCHPAD-JOURNAL] Operator Scratchpad Journal (Normative)

Section titled “[RFC-0005:C-SCRATCHPAD-JOURNAL] Operator Scratchpad Journal (Normative) ”

InferLab MUST provide an operator scratchpad journal: one workspace-local append-only stream of entries per workspace, stored as machine state under the workspace state directory.

A journal entry MUST identify its creation timestamp, its author, its free text, an optional topic label, and zero or more referenced execution records. Entries MUST be appended in creation order, and the scratchpad surface MUST NOT edit or delete an appended entry.

Entry timestamps MUST use the same UTC format as execution record identifiers so entries and records order on one time axis.

Appending MUST require no prior setup: the journal and any topic exist by first use, a topic carries no lifecycle state, and no workspace-level mode routes entries — an entry’s topic is stated on the entry itself or absent. Entries without a topic form the common stream.

Appending an entry MUST validate that each referenced execution record exists and MUST NOT mutate a referenced record.

The journal rendering MUST present entries chronologically with a one-line summary for each referenced record and MUST support restricting the view to one topic. A default view MAY limit itself to recent entries, but it MUST state how many earlier entries are omitted, and a complete view MUST be available.

Scratchpad state MUST be excluded from workspace source identity and MUST NOT become an input to resolution, execution, or record content.

Scratchpad state MUST remain inspectable as plain files without the InferLab binary.

Rationale: Journal writes compete with the experiment itself, so the write path carries no ceremony and no mode. A sticky active-thread designation is only convenient when narratives do not interleave — exactly when routing is unneeded — and silently misroutes append-only entries when several efforts (for example, multiple models validated side by side in one workspace) do interleave; an explicit per-entry topic is the only routing that cannot land an entry in the wrong narrative, and an absent topic costs nothing. Reading is rare but resume-oriented, so the default view leads with the recent tail. Entries share the record-identifier time axis so records and narrative interleave into one history without a second authority: records own execution facts, the journal owns the narrative, and neither feeds resolution. Scratchpad state stays out of source identity for the same reason records do, and one line per entry under a short exclusive file lock — the timestamp taken inside the lock — keeps concurrent entries whole and in time-axis order on any filesystem, including network mounts where bare O_APPEND guarantees neither; readers take the shared lock, so a partially written line is never observed.

Since: v0.1.0

[RFC-0005:C-LM-EVAL-EVIDENCE] lm-eval Execution Evidence (Normative)

Section titled “[RFC-0005:C-LM-EVAL-EVIDENCE] lm-eval Execution Evidence (Normative) ”

lm-eval evidence MUST preserve the declared and resolved task-source kind; the built-in task name, bundled task name and bundled task-closure digest, or the workspace-relative and resolved local YAML path; the contained YAML include closure when one exists; the one resolved task identity; the effective task configuration; and the effective output type, dataset selections, evaluation and few-shot splits, prompts, filters, metrics, and scoring directions when those values apply.

Evidence for an InferLab-bundled task MUST preserve the task-definition identity, bundled task-closure digest, prompt-asset digest, dataset-asset digest when separate from the prompt asset, and scorer identity. Those identities MUST match the invoking release’s completed toolchain installation.

lm-eval evidence MUST preserve the actual resolved tokenizer locator, tokenizer backend, and text-request setting. These local record values MUST remain unredacted under the access-control and sharing rules of RFC-0005:C-EVIDENCE.

When the endpoint conformance probe is required, evidence MUST preserve its effective request and timeout, elapsed time, transport or HTTP outcome, response status when available, raw response artifact when a response was received, each token-array and tokenizer-alignment check, and the typed supported, unsupported, or inconclusive conclusion. A failed probe MUST preserve the specific failure class and diagnostic context.

Each started lm-eval attempt MUST preserve the exact native command, exit or timeout outcome, stdout and stderr artifacts, raw output directory, discovered result files, and partial artifacts even when execution or normalization fails.

Every lm-eval attempt MUST enable lm-eval’s per-sample logging, whatever its trial count, so a completed Eval keeps each sample’s result. InferLab MUST write that log as a projection of lm-eval’s per-sample records: each record MUST keep every field lm-eval emits except the sample document and each request’s context, which the recorded task identity and native document identity reproduce. The projection MUST replace each request context with an explicit null at its position and MUST keep the document, prompt, and target hashes, so a reproduced document or prompt can be verified. Evidence MUST show that per-sample logging was enabled, MUST identify the artifact as that projection, and MUST NOT describe it, or synthesized failure evidence, as unaltered lm-eval output. For repeated-result normalization, evidence MUST retain the sample identity, responses, filters, and scores that the engine emitted.

A single-trial lm-eval attempt MUST preserve generation outcomes as they happen, because lm-eval writes its native sample log only when evaluation completes. Before issuing a task’s generation requests, InferLab MUST record one entry per generation request instance carrying its task name, native document identity, and the SHA-256 digest of its request context. For every HTTP attempt of a generation request, InferLab MUST record one entry carrying the context digests that request carries, its send and completion times, its HTTP status or transport failure, every returned choice without token log-probabilities, and the endpoint’s reported token usage or its explicit absence. Each entry MUST be written when its event occurs, so the instance entries and the entries of completed attempts remain available when the lm-eval process exits unsuccessfully, times out, or is interrupted. These entries MUST NOT duplicate request prompt text, MUST NOT carry a task score, and MUST NOT derive a token count the endpoint did not report; a question’s score remains in the projected sample log, joined through its native document identity.

Repeated Eval evidence MUST preserve requested_trials and the stable identity and effective request seed planned for every requested trial. An unissued trial MUST retain its planned identity and seed but MUST NOT acquire a fabricated endpoint request, response, score, or outcome.

For every issued trial, InferLab MUST incrementally preserve one flat endpoint-outcome entry carrying the stable trial identity and resolved sample identity. That entry MUST preserve the exact effective request, complete generated response or explicit absence, endpoint finish reason or explicit absence, effective generation-token limit, completion-token count and its source, binary task score or explicit absence, passing or non-passing conclusion when completed, and failure evidence when no completed classification exists. The evidence MUST remain available when the lm-eval process exits unsuccessfully, times out, or is interrupted.

When a native response or score exists for an issued trial, evidence MUST identify its corresponding sample record without altering the fields the projection keeps. Task-specific answer extraction, normalization, and outcome labels MUST remain task-owned evidence and MUST NOT be reconstructed from task names by the InferLab control plane.

The completion-token count source MUST be server-usage when the endpoint supplies a valid completion-token count. Otherwise InferLab MUST derive the count with the resolved tokenizer and record source resolved-tokenizer when generated text exists. When no generated text exists, the completion-token count MUST be zero and its source MUST be none.

The maximum-token-hit conclusion MUST be true only when finish reason is literal length. It MUST be false when finish reason is literal stop. It MUST be absent when the finish reason is absent or has another value because InferLab does not infer backend-specific aliases.

Repeated Eval aggregate evidence MUST preserve issued_trials, unissued_trials, completed_trials, request_failure_trials, passed_trials, and pass_rate or its explicit absence. The aggregate counts and pass rate MUST be reconstructible from the planned trials and flat endpoint-outcome entries. Evidence MUST preserve the observed metric-gate conclusion separately from an overall timeout, interruption, or other Eval terminal failure.

A normalized primary metric MUST preserve the exact resolved task identity, declared metric and optional filter, selected native metric key, finite value, resolved scoring direction, resolved prompt authority, effective threshold, comparison operation, and gate conclusion. When a model-keyed threshold is effective (RFC-0004:C-LM-EVAL), the Eval plan MUST name the model whose entry selected it. The prompt authority MUST travel with the metric itself rather than beside it, because the same task scored under another authority is a different measurement. A repeated pass-rate metric MUST additionally preserve its binary per-trial scoring contract and denominator counts. Successful evidence MUST retain the complete raw lm-eval result in addition to the normalized metric map.

The public backend support matrix MUST report loglikelihood status independently for every listed direct Engine endpoint and Gateway-backed public endpoint. A Gateway-backed claim MUST identify its Gateway backend and its P/D Router backend or explicit absence. A qualified claim MUST identify the published integration revision, route identity, topology, and qualified hardware baseline demonstrated by its real execution record. An unsupported or unqualified entry MUST state that bounded status without generalizing it to untested revisions or routes. Public documentation MUST NOT publish credentials, private hosts, private model locators, local filesystem paths, or private downstream revisions from the unredacted local record.

Rationale: The local record must be sufficient to explain which task semantics, tokenizer, endpoint behavior, trial requests, and metric comparison produced a gate result without rerunning the workspace. Bundled task and asset identities prove which release-fixed task closure executed without placing a concrete task catalog or scorer in this RFC. Projected sample logs retain lm-eval’s response, filter, and score for every sample while omitting the document and prompt content that the task identity reproduces and that can otherwise dominate the record; flat endpoint outcomes preserve partial execution and endpoint metadata without pretending that lm-eval emitted evidence it did not. Recording the token-count source avoids presenting tokenizer-derived counts as server observations. Public support claims need an exact capability, route, revision, topology, and qualification boundary but do not inherit the record’s intentionally local sensitive values.

Since: v0.6.0

[RFC-0005:C-INFERENCE-REQUEST-EVIDENCE] Inference Request Evidence (Normative)

Section titled “[RFC-0005:C-INFERENCE-REQUEST-EVIDENCE] Inference Request Evidence (Normative) ”

A measurement attempt is one lm-eval invocation, one serving-Bench case, or one built-in smoke invocation. A transport request is one HTTP request emitted within that attempt. Evidence MUST NOT represent a multi-request measurement attempt as one fabricated complete HTTP request body.

Every started Eval or Bench attempt MUST preserve the selected named route, effective public URL, definition-declared request_body, task or client request defaults that it replaces, invocation-scoped request overrides, and the complete effective non-structural request fragment after deterministic resolution, including backend-owned template controls when declared.

Evidence MUST preserve the exact measurement-client invocation and native configuration or request template that applies the effective fragment to transport requests. Framework-specific request members MUST remain under their real JSON names and MUST NOT be translated into an InferLab-owned sampling or reasoning profile.

When a definition fragment or invocation override replaces a task- or client-supplied non-reserved default, evidence MUST preserve the earlier and replacement values and their authority sources.

Each lm-eval attempt MUST identify its one resolved task, output type, request family, and selected route. Multi-task experiments MUST preserve the separate Eval attempt and request fragment for each recipe-selected task.

Every serving-Bench attempt MUST preserve the resolved prompt kind, derived request representation and route, rendering authority, resolved model tokenizer locator and backend, and installed Transformers version. Synthetic population, local rendering, prefix geometry, cache preparation, backend prompt-token reconciliation, and observed cache-read evidence MUST follow RFC-0005:C-BENCH-PROMPT-CACHE-EVIDENCE. Dataset-backed entries MUST preserve their structured messages and pre-template content counts under RFC-0005:C-BENCH-REQUEST-SOURCE-EVIDENCE. Linear-session turn evidence MUST follow RFC-0005:C-BENCH-LINEAR-SESSION-EVIDENCE. A successful profiling result MUST separately preserve the backend-observed prompt-token distribution sourced from AIPerf input_sequence_length; locally planned counts MUST NOT replace that observation.

When lm-eval or AIPerf emits a native per-sample or per-request artifact, InferLab MUST preserve it unchanged and retain its native request identity. Materialized independent-request identities MUST reconcile each native warmup or profiling request to its assigned population entry and case slice so that a synthetic entry’s planning result can be compared with that request’s backend-observed prompt tokens. Requirements for a specific bundled task sample and endpoint artifacts remain governed by RFC-0005:C-LM-EVAL-EVIDENCE. This clause does not require InferLab to intercept and duplicate every transport body when the release-pinned engine does not expose one.

A request rejected before client launch because it conflicts with an InferLab-owned structural member MUST preserve the conflicting member path and validation failure. A backend rejection observed by the client MUST preserve the engine HTTP status and response diagnostic when available and MUST NOT be reclassified as an unsupported field without backend evidence.

The built-in smoke is one transport request and MUST continue to preserve its complete effective URL and body under RFC-0004:C-MEASUREMENTS.

Rationale: The reproducible experiment input is the resolved non-structural request fragment plus the task or load-generator prompt, message, request-shape, and rendering authority. One task per Eval makes request-family and route evidence unambiguous, while derived serving-Bench routes keep prompt semantics authoritative. Keeping the request layers explicit avoids claiming that a many-sample Eval or Bench case had one request body. Native artifacts remain the authority for dynamic request content they expose.

Since: v0.8.0

[RFC-0005:C-BENCH-SLO-EVIDENCE] Serving Bench SLO Evidence (Normative)

Section titled “[RFC-0005:C-BENCH-SLO-EVIDENCE] Serving Bench SLO Evidence (Normative) ”

The complete effective Bench definition in resolved evidence MUST preserve every aggregate constraint, request-level bound, minimum good-request ratio, adaptive initial rate, search-step bound, minimum rate resolution, and adaptive policy identity governed by RFC-0004:C-BENCH-SLO.

Each static case and adaptive probe MUST own one case-level SLO evaluation as the sole authority for that case’s SLO outcome.

Each aggregate-constraint evaluation MUST preserve its stable metric name, declared bound and direction, finite observed value when available, and passed, failed, or unavailable outcome.

Each request-level evaluation MUST preserve good-request count, good-request ratio, goodput, profiling duration, duration source identity, the native request-identity and resolved-count reconciliation outcome, the optional native aggregate-count consistency outcome, and its ratio-gate outcome.

The case’s existing completed-request and failed-request fields MUST remain the sole record authority for those counts and MUST NOT be duplicated inside its request-level SLO evaluation.

A case-level SLO evaluation MUST preserve its AND-composed conclusion separately from the case’s measurement status.

A static Bench record with SLOs MUST preserve its overall pass conclusion as the AND of successful execution and case-level SLO satisfaction across every resolved case.

An adaptive Bench summary MUST preserve policy identity, selected observed rate or its explicit absence, whether a higher observed above-region boundary brackets that selection, normal termination reason, and ordered case identifiers for every executed probe.

The adaptive summary MUST NOT duplicate the per-probe metrics or SLO evaluations owned by its referenced case records.

A normally exhausted search budget MUST be distinguishable from reaching the configured rate resolution.

An adaptive record with no selected rate MUST preserve that absence without claiming that no feasible rate exists outside the executed probes.

Measurement failure, timeout, interruption, process outcome, and cleanup evidence MUST remain governed by their existing record authorities and MUST NOT be duplicated as adaptive-summary termination states.

Rationale: One case-level evaluation makes every SLO decision auditable beside the measurements that produced it, while the adaptive summary remains a compact ordered index of the search rather than a second copy of case evidence.

Since: v0.10.0

[RFC-0005:C-BENCH-REQUEST-SOURCE-EVIDENCE] Serving Bench Request Source Evidence (Normative)

Section titled “[RFC-0005:C-BENCH-REQUEST-SOURCE-EVIDENCE] Serving Bench Request Source Evidence (Normative) ”

Serving-Bench dry-run and execution evidence MUST preserve the declared and effective request-source variant, source-specific shape policy, seed, required population count, and resolved tokenizer locator.

random source evidence MUST preserve each input- and output-token selector as either its exact fixed value or its inclusive_uniform kind and inclusive bounds. Prompt-kind, shared-prefix-geometry, cache-preparation, and observed cache evidence MUST follow RFC-0005:C-BENCH-PROMPT-CACHE-EVIDENCE. Omission of each optional declaration MUST remain explicit.

Weighted-random-mixture evidence MUST preserve the ordered exact input- and output-token shapes, each declared integer weight, and the total weight from which the effective categorical sampling probabilities are reconstructed.

Every random and weighted-random-mixture source MUST preserve one frozen generated population artifact and its digest. Each entry MUST retain its stable sequence index, selected input-token target, selected output-token target, prompt-kind-specific materialization evidence governed by RFC-0005:C-BENCH-PROMPT-CACHE-EVIDENCE, and the exact sequential case slice that consumed it. Population summaries MUST preserve prompt-kind-specific exact and fallback classifications where applicable without reconstructing the population from AIPerf output.

Random, weighted-random-mixture, dataset, and session-source evidence MUST preserve the declared and effective output_stop, with omission explicit, and each frozen entry MUST retain the stop controls its request carries; replay evidence MUST record output_stop as not applicable. Dataset-source evidence MUST preserve its catalog name and profile when applicable, immutable upstream identity, or for an AIPerf public-dataset source the AIPerf release and public-dataset name, expected content digest, resolved cache path, cache state, maximum input-token bound, fixed output-token override or target-derived policy, closed split and filter policy when applicable, and versioned materialization identity.

When dataset acquisition is attempted, evidence MUST preserve whether bytes were downloaded or reused, observed byte count, observed digest or its absence, and the terminal acquisition outcome; for an AIPerf public-dataset source, it MUST additionally preserve whether the rows were reused or materialized by AIPerf, the location of AIPerf’s cache, and AIPerf’s failure message when materialization fails. Progress diagnostics MUST remain separate from machine-readable output.

A dataset materialized population MUST be preserved as a record artifact with its content digest. Each entry MUST retain stable source-sample identity, structured source input messages, effective request representation, effective input-token count, and effective output-token limit. An entry whose catalog materialization uses a held-out assistant target MUST additionally retain that target’s identity and kept and removed message counts. An entry with a fixed output limit and no source target MUST explicitly record target absence and MUST NOT fabricate a target identity.

Each speed_bench population entry MUST retain its upstream question identity, resolved profile, unchanged first user-turn content, count of later source turns not replayed, structured one-message input, fixed requested output-token limit, and tokenizer-observed pre-template content length.

Dataset population evidence MUST preserve requested, admitted, and ineligible candidate counts plus ineligibility counts by stable reason. It MUST preserve reconstructible effective input- and output-token distributions and whether a fixed output override applied.

Every dataset-backed static case and adaptive probe MUST reference the one frozen population digest and the exact sequential slice assigned to its warmup and profiling phases. Those references MUST make reuse across adaptive probes and prefix relationships across differently sized static cases explicit without duplicating the population artifact.

When dry-run observes a dataset cache miss or lacks another input required for materialization, it MUST preserve the unavailable population status and planned acquisition facts. It MUST NOT claim a population digest, token distribution, or admitted count that was not computed.

AIPerf native per-request artifacts MUST remain unchanged evidence under RFC-0005:C-INFERENCE-REQUEST-EVIDENCE. Every materialized independent-request identity, whether synthetic or dataset-backed, MUST reconcile to its assigned warmup or profiling population entry and case slice.

For a replay request source, the record MUST preserve the replayed file’s observed SHA-256, the declared expected digest when present, the workspace-relative path as declared, and the entry count; the record MUST NOT claim provenance beyond the replayed file content.

When a random source declares a corpus, population evidence MUST preserve the corpus declared path, its declared and observed digests, and the resolved tokenizer identity, and each entry MUST retain its corpus slice offset and length alongside its selected token targets.

When a random source declares images under RFC-0004:C-BENCH-REQUEST-SOURCES, dry-run and execution evidence MUST preserve the effective image count and fixed width and height, the source form (synthetic noise or an operator directory), and for a directory source the declared path, the declared and observed enumeration digests, and the effective sampling policy. Evidence MUST present the image decoration as a request-time policy and MUST NOT claim a frozen image sequence or per-request image identities.

Rationale: Synthetic source evidence must distinguish selected shapes, prompt authority, shared-prefix geometry, and seeded weighted mixtures without turning the record into a copy of the AIPerf configuration language. Dataset identity alone cannot explain the actual serving load because tokenizer choice, structured-message content, input bounds, output policy, seed, and case size determine the emitted requests. Preserving one population artifact and case slices makes that load auditable without copying it into every adaptive probe or misrepresenting source conversations as live sessions.

Since: v0.11.0

[RFC-0005:C-BENCH-LINEAR-SESSION-EVIDENCE] Serving Bench Linear Session Evidence (Normative)

Section titled “[RFC-0005:C-BENCH-LINEAR-SESSION-EVIDENCE] Serving Bench Linear Session Evidence (Normative) ”

Serving-Bench evidence for session_source MUST preserve the selected source boundary, declared and effective source, inter-turn-delay, and session-count fields, seed, catalog and profile identity, immutable upstream identity and digest, cache and acquisition state, tokenizer identity, structured-message route, and materialization identity. The common dataset-source and dry-run rules of RFC-0005:C-BENCH-REQUEST-SOURCE-EVIDENCE apply unless this clause assigns session-specific evidence.

The frozen population MUST be one content-addressed artifact. Each entry MUST have one stable template identity and ordered source turns containing the unchanged user message, source inter-turn delay, effective inter-turn delay, effective output-token limit, and fixed-override or target-derived provenance. The source snapshot and template identity remain authoritative for held-out target content. Population evidence MUST preserve requested, admitted, and ineligible template counts, stable ineligibility reasons, first-turn pre-template content-token counts, reconstructible turn-count, source-delay, effective-delay, and output-limit distributions, and the population-wide TPOT applicability.

Every static case MUST reference the population digest and the ordered template identities assigned to its warmup and profiling slices. Each slice MUST preserve planned session count and planned transport-request count. Prefix reuse across cases and the disjoint warmup and profiling slices MUST be reconstructible from those references.

Dry-run MUST preserve determinable source, delay-control, population, and slice facts but MUST NOT fabricate a population digest when materialization inputs are unavailable or any runtime session, live response, transport, or outcome fact.

Each admitted runtime session MUST preserve its case, phase, x_correlation_id runtime identity, template identity, planned and attempted turn counts, and final outcome of succeeded or failed. A failed session MUST preserve its first failure classification and diagnostic, the failing turn when known, and the number of planned later turns that were not attempted. Cancellation, interruption, missing terminal evidence, and a session still active when the case ends are session failures rather than additional session outcome variants.

Each attempted turn MUST preserve its runtime session identity, zero-based turn index, phase, and phase-local native transport-request identity, plus the measured pre-template content-token count when the raw request/response artifact is produced. When a terminal backend response reports prompt-token usage, the turn MUST additionally preserve that observed post-template prompt-token count; a successful turn MUST NOT substitute its pre-template content count for missing backend usage. When the raw request/response artifact is produced, a dependent turn on the valid path through the first failure MUST additionally preserve the preceding native live-response identity and preserve or reference the preceding terminal-response receipt time, effective inter-turn delay, and native request-start timestamp sufficiently to demonstrate whether transport began after the delay elapsed. A native dependent turn emitted after the first failure MAY lack that live-response identity and complete pairwise history; evidence MUST identify it as post-failure continuation rather than fabricate the missing response. When the raw request/response artifact is produced, it MUST remain authoritative for the exact structured messages, effective request, backend response, and timing, and the turn record MUST reference that artifact instead of duplicating its request or response body.

At the performance artifact level governed by RFC-0004:C-BENCH-ARTIFACT-LEVEL, the raw request/response artifact is not produced. Turn dimensions derived from it — the measured pre-template content-token count, the max_input_tokens bound violation check, and the preceding native live-response identity and terminal-response receipt-time pairwise history — MUST be recorded as unavailable due to the artifact level rather than fabricated or treated as reconciliation failures. Session identity, phase assignment, turn order, inter-turn-delay, and planned-versus-attempted count reconciliation MUST still be evaluated from the normalized per-request records, which carry the native benchmark-phase marker for both phases.

A later turn whose native structured-message request exceeds max_input_tokens MUST preserve the measured bound violation, native request, and backend outcome and MUST fail the session even when transport completed successfully. A native turn emitted after the session’s first failed turn MUST remain attached to that failed session and MUST NOT restore success or be hidden from attempted-request counts. Evidence MUST NOT fabricate missing turns, responses, or transport records.

Native x_correlation_id MUST equal the InferLab runtime session identity. Together with turn_index, it MUST reconcile every native request to one session turn. Native session_num remains unique only within its benchmark phase. The pair of phase and session_num is the case-level transport-request identity and MUST NOT be relabeled as a session identity.

Each case summary MUST preserve planned, started, succeeded, and failed session counts and planned, attempted, completed, and failed transport-request counts. Started sessions MUST equal succeeded plus failed sessions after finalization. Session counts MUST reconcile to the frozen slice and session records; request counts MUST reconcile to native records and identify planned turns that were not attempted. An attempted request without a terminal native record MUST fail the case. Warmup MUST use the same summary separately and MUST NOT contribute to profiling metrics. A native warmup phase marker and its observed prompt tokens MUST remain visible in warmup turn and raw-request evidence without being attributed to the frozen source template.

A successful session-backed case MUST preserve affirmative population-slice, session, dependency-order, inter-turn-delay, native-request, and count reconciliation. Request-level SLO evidence remains governed by RFC-0005:C-BENCH-SLO-EVIDENCE and uses transport-request metrics only.

Rationale: The frozen template owns source content and effective delay, x_correlation_id owns runtime session identity, and native AIPerf artifacts own transmitted structured messages, live responses, and transport timing. Referencing those authorities records the dependency chain without copying cumulative histories or introducing a client-side chat renderer. Failed native continuation is retained as evidence but cannot convert an invalid session into a benchmark result.

Since: v0.15.0

[RFC-0005:C-BENCH-PROMPT-CACHE-EVIDENCE] Serving Bench Prompt And Prefix-Cache Evidence (Normative)

Section titled “[RFC-0005:C-BENCH-PROMPT-CACHE-EVIDENCE] Serving Bench Prompt And Prefix-Cache Evidence (Normative) ”

A serving Bench record MUST preserve the declared prompt kind, the resolved prompt kind, the derived request family and route, and the authority that rendered the final prompt. These values MUST come from the resolved execution object used by dry-run and execution; record production MUST NOT infer them again from transport observations.

For flat, the record MUST preserve every selected input-token target, the locally measured final prompt-token count, and either the frozen final prompt population or a content digest plus a durable reference from which that exact population can be recovered.

For rendered_chat, the record MUST additionally preserve the structured source messages, whether the tokenizer default or an explicitly supplied chat template was selected, the effective template content, effective template kwargs, template digest, generation-marker policy, and the resulting frozen flat prompt or its recoverable content digest. The locally measured final prompt-token count is the planned input-token count.

For every measured flat or rendered_chat request, the record MUST preserve the backend-reported prompt-token count and its reconciliation with the planned count. A missing count or a mismatch invalidates that request result as required by RFC-0004:C-BENCH-PROMPT-AUTHORITY; the record MUST preserve the observed value and the validation failure.

For server_chat, the record MUST preserve the structured messages sent to the server and the backend-reported prompt-token count. When a synthetic source uses local content-length projection, the record MUST also preserve the selected content-token target, the local projected final prompt-token count when one was available, whether projection was exact, and any fallback reason. That projection is planning evidence only; the backend-reported count remains authoritative for the measured final prompt.

When prefix_sharing is declared, the record MUST preserve whether the operator supplied shared_prefix_tokens or shared_prefix_ratio, its declared value, and each population entry’s resolved input-token count, resolved shared-prefix-token count, and resolved unique-suffix-token count. It MUST preserve the canonical shared-token-stream digest and sufficient frozen population evidence to reproduce the exact nested prefixes. The record MUST NOT replace the declared geometry with an inferred cache-hit percentage.

When shared_system_content is declared, the record MUST preserve its declaration form and value, the resolved system-content and user-content token counts for every population entry, and the canonical system-content digest. It MUST state that these counts describe pre-template message content and do not claim an exact final prompt prefix.

The record MUST preserve the resolved cache start policy. uncontrolled MUST be explicit in the record even when selected by omission. For a cold or primed case that resets, it MUST preserve every reset attempt, including each declined attempt retried under RFC-0004:C-BENCH-CACHE-STATE, with its route and method, response status, success-predicate outcome when a predicate is declared, elapsed duration, and outcome, together with the aggregate reset outcome and its position relative to runtime warmup, conditioning, and profiling. When the reset capability is per-target under RFC-0004:C-BENCH-CACHE-STATE, each attempt MUST also identify its model-serving process and resolved URL.

For primed, the record MUST preserve, for each conditioning request, the target replica where the request was fanned out behind a frontend, the target data-parallel rank, derived route, model, prompt kind, effective template identity where applicable, effective non-structural request-body members, prompt content or recoverable digest, locally measured prompt-token count, backend-reported prompt-token count and cache-read usage when available, response status, elapsed duration, and outcome. Conditioning fan-out evidence MUST preserve every targeted replica and rank and every per-target failure. It MUST preserve the maximum resolved shared-prefix-token count that selected the conditioning prompt and MUST identify the request as preparation excluded from measured population counts and distributions.

Phase-transition evidence MUST make the actual ordering observable: runtime warmup completion and drain, reset completion when the case resets, conditioning completion when applicable, and release of the first measured request. A case that separates reuse by namespace under RFC-0004:C-BENCH-CACHE-STATE records no reset, and its population slice evidence names the case’s own population. A dry-run MUST render the planned ordering and resolved values but MUST NOT claim runtime outcomes.

For every measured request for which RFC-0004:C-BENCH-CACHE-STATE requires prompt-cache-read usage, the record MUST preserve backend-reported prompt tokens, prompt-cache-read tokens, derived uncached prompt tokens, and derived per-request cache-read ratio. Aggregate distributions and the weighted aggregate cache-read ratio MUST be derived from these same per-request observations. A grouped population MUST preserve each entry’s prefix group and the number of entries per group, and each conditioning request’s group. Where RFC-0004:C-BENCH-CACHE-STATE requires reuse opportunity, each such observation MUST also preserve its prefix group, reusable tokens, and opportunity state, and the record MUST preserve the publication bound and the per-request send and first-token times that decided them; the opportunity metrics MUST be derived from these same observations. When the resolved endpoint declares the backend-defined omitted-zero representation, the record MUST preserve that capability and MUST record the normalized zero as the request observation. Otherwise, the record MUST preserve unavailable metrics and invalid arithmetic as explicit normalization failures rather than silently substituting zero.

A population entry whose resolved shared prefix equals its full input length MUST be recorded as full-prompt reuse. InferLab MUST NOT label that entry or workload decode_only solely from declared geometry; any stronger conclusion requires the observed uncached-prompt evidence to support it.

Native AIPerf evidence retained by RFC-0005:C-BENCH-RAW-EVIDENCE remains unchanged. Normalized prompt and cache metrics MUST reconcile to the retained native observations rather than to a separately reconstructed request population.

Rationale: prompt semantics, workload geometry, cache preparation, and observed reuse answer different reproducibility questions. Preserving each authority and the phase ordering prevents an apparent cache-hit percentage from concealing a changed prompt, an uncontrolled cache, or preparation traffic counted as measurement.

Since: v0.20.0

[RFC-0005:C-BENCH-AGENTIC-TRACE-EVIDENCE] Serving Bench Agentic Trace Replay Evidence (Normative)

Section titled “[RFC-0005:C-BENCH-AGENTIC-TRACE-EVIDENCE] Serving Bench Agentic Trace Replay Evidence (Normative) ”

Serving-Bench dry-run and execution evidence for agentic_source MUST preserve the selected source boundary, declared and effective dataset and profile, catalog and materialization identities, Bench seed, root-tree concurrency, profiling duration and its declared-or-defaulted provenance, served model, endpoint, timeout, and server-metric selection. It MUST preserve every closed replay-policy value contributed by the release profile without exposing those values as independent workspace controls.

Source evidence MUST preserve the upstream dataset repository and immutable revision, expected and observed source digests, upstream benchmark-script repository and revision reproduced by the profile, acquisition and cache outcome, release-pinned AIPerf identity, native AgentX scenario identity, loader identity, and source format. If an upstream acquisition API cannot report an immutable revision, execution MUST fail reproducibility validation rather than label a mutable alias as an immutable snapshot.

The immutable source snapshot and its digest MUST remain the authority for the frozen trace topology and source geometry. Evidence MUST preserve a durable content-addressed source reference from which each trace’s root stream, descendant streams, agent type when present, parent links, spawn and join dependencies, source request coordinates, source timestamps, token lengths, and cache-block geometry can be recovered. InferLab MUST NOT duplicate that topology into a second independently authoritative DAG representation, and MUST NOT require inputs.json from a Weka run for which the release-pinned AIPerf deliberately omits it.

Every case MUST preserve the deterministic replay inputs that AIPerf exports or receives, including seed, source snapshot, profile policy, and native run identity. When the raw request/response artifact is produced, every native request record MUST map its runtime conversation and transport identities to the available source trace, outer-request, inner-request, and source-kind coordinates. Agent depth and parent or root correlation identities MUST be preserved when AIPerf reports them. Evidence MUST NOT claim an observed per-lane sampled trajectory boundary, recycle identity, or scheduler state that is not exported by the release-pinned artifact contract.

Timing evidence MUST preserve the recorded source timestamps recoverable from the source snapshot, the effective source-timing and global-idle policies, and native request start and terminal timestamps. These facts MUST distinguish recorded trace timing from observed replay transport timing without presenting their difference as an exact exported timer adjustment. Evidence MUST preserve the release profile’s effective global and per-trace idle-gap caps, which AIPerf applies natively, and MUST state that no InferLab inter-turn delay distribution was applied. Internal per-adjustment global-idle shifts, dependency timer movements, and residual delays MAY be described only when a future qualified native artifact reports them.

Warmup evidence MUST separately preserve every emitted native warmup request record, its available source coordinates and runtime identities, terminal outcome, warmup request and error counts, the effective snapshot and cache-pressure policy, and the native warmup phase outcome. It MUST preserve whether profiling began only after successful warmup and drain. AIPerf-internal trajectory snapshots, streams that emitted no warmup request, live scheduler state at handoff, and residual next-request delay MUST NOT be fabricated when the release-pinned AIPerf does not export them. Dry-run MUST NOT claim any runtime warmup fact.

Each native transport record MUST preserve or reference its case, phase, runtime conversation and available parent or root identities, source trace and request coordinates, agent depth and source kind when present, request start and terminal timing, prompt and output token observations, terminal response, error or cancellation, context-overflow classification, and cache-bust marker and target when present. When the raw request/response artifact is produced, it remains authoritative for exact headers, structured payload, streamed response, timing, and diagnostic; normalized evidence MUST reference rather than rewrite it.

At the performance artifact level governed by RFC-0004:C-BENCH-ARTIFACT-LEVEL, the raw request/response artifact is not produced. Evidence dimensions derived from it — the per-record source trace, outer-request, inner-request, and source-kind coordinate mapping, cache-bust marker and target observations, and warmup source-coordinate records — MUST be recorded as unavailable due to the artifact level rather than fabricated or treated as reconciliation failures. Phase assignment, runtime conversation and transport identities, terminal outcomes, and count reconciliation MUST still be evaluated from the normalized per-request records and the aggregate artifact.

Each case summary MUST separately preserve profiling and warmup request counts; distinct observed source traces, runtime conversations, and transport requests; the complete native branch_stats counters for child spawn, completion, error, truncation and delay plus parent suspension, resume, child-caused failure and suppressed joins; ordinary failed-request counts and native threshold outcome; and context-overflow counts and rate outcome. Missing native dimensions, including root-tree completion and recycle counts, MUST remain explicitly unavailable rather than be inferred from transport records or concurrency.

Normalized transport latency, token, request-throughput, and optional server metrics MUST reconcile to the same accepted native profiling records and preserve their existing units. Branch summaries MUST remain separate and MUST NOT be presented as semantic task success, agent quality, root-tree throughput, or transport-request throughput. Warmup traffic and context-overflow records excluded by AIPerf from the normal profiling metric stream MUST remain visible in their own counts and MUST NOT be silently added back to latency distributions.

The case MUST preserve the native scenario locks and violations, submission-valid flag and reason codes, process exit, cancellation state, required-artifact inventory, normalization and exported-field reconciliation outcomes, and final InferLab case outcome. InferLab MUST preserve the native failed-request abort threshold and context-overflow validity decision as evidence and MUST NOT claim an independently computed acceptance verdict.

A successful agentic case MUST preserve affirmative source-identity and digest validation, effective replay policy, warmup phase outcome, native-request coordinate reconciliation when the raw request/response artifact is produced, complete native branch statistics, scenario validity, required artifacts, and normalized-metric reconciliation. A failed case MUST retain every partial source, warmup, raw-request, aggregate, scenario, and process artifact that was actually produced without fabricating absent results.

Rationale: The immutable source snapshot owns the frozen session tree, AIPerf owns materialization, replay scheduling, and raw transport facts, and InferLab owns the resolved policy, qualified public-artifact validation, and durable cross-artifact reconciliation. Preserving only exported facts makes a SemiAnalysis AgentX result auditable without private AIPerf instrumentation, a second DAG schema, or confusion between request performance and agent-task correctness.

Since: v0.21.0

[RFC-0005:C-MEASUREMENT-DATA-ASSET-EVIDENCE] Measurement Data Asset Evidence (Normative)

Section titled “[RFC-0005:C-MEASUREMENT-DATA-ASSET-EVIDENCE] Measurement Data Asset Evidence (Normative) ”

RFC-0004:C-MEASUREMENT-DATA-ASSETS introduces recipe-record schema version 3 and workload-record schema version 13. A recipe record at version 3 MUST own the preparation-attempt collection shared by its measurement children. A recipe-owned child workload at version 13 MUST reference the applicable recipe preparation-attempt identities and MUST own its materialization evidence at the scope required by its measurement contract. A standalone manual Bench workload at version 13 MUST own both its preparation-attempt collection and its materialization evidence.

Record listing and summary projections MUST retain every previously supported record version and MUST add recipe version 3 and workload version 13. A typed consumer that requests data-asset fields from an older record MUST treat them as unavailable because that schema predates the evidence, not as an empty successful preparation. Unsupported future schema versions MUST NOT be interpreted as a supported asset-evidence contract.

A preparation-attempt identity MUST be assigned from the pre-effect source-selection key before the first external preparation effect. The key MUST include the selection authority, acquisition provider and release-pinned acquisition-runtime identity, locator, requested revision or alias, subset, split, filter, data-file selection, local path, and release-catalog identity whenever the field is already selected by effective configuration or catalog data without that effect. Fields discovered only by the owning resolver after the attempt begins MUST be preserved in the resolved effective-selection and acquired-source evidence and MUST NOT retroactively change the attempt identity. One preparation owner MUST preserve exactly one entry for each distinct pre-effect selection key, and that entry MUST list every consumer. Equal keys MUST share the entry; unequal keys MUST remain separate even if they later resolve to the same effective selection, acquired bytes, or physical cache.

Each preparation entry MUST preserve its attempt identity, selecting definitions and consumers, selection authority, acquisition provider and runtime identity, source kind, declared selection, current state, and whether source readiness established a closed immutable source or an opaque non-reproducible source. Resolved facts MUST remain separate from the declared selection and MUST include the acquired-source identity and complete consumed-content closure when established. For a remote closure, evidence MUST preserve every resolved immutable revision or content identity and the explicit absence of any required identity. For a closed local source, it MUST preserve the original resolved root, prepared snapshot location, and ordered normalized relative path plus SHA-256 entry for every consumed regular file. For an opaque local source, it MUST instead preserve the original path and missing-closure reason without a fabricated snapshot identity.

Cache evidence MUST preserve every applicable effective store reported by the owning acquisition runtime, the store authority, and a per-store outcome of full hit, miss, partial reuse, or unavailable observation. It MUST separately preserve whether remote metadata access occurred and whether source bytes were downloaded or reused when those facts are observable. It MUST NOT collapse multiple Hugging Face stores into one path or present a physical cache path as source identity.

Each source-ready preparation entry MUST carry a reproducibility conclusion and basis. A reproducible conclusion MUST identify the complete immutable content closure actually bound to the consumer and, for a release-qualified source, successful expected-versus-observed verification. A non-reproducible conclusion MUST preserve the missing closure identity, unsupported immutable binding, mutable dependency, incomplete local file enumeration, deferred source access, or other specific reason. An immutable repository revision alone MUST NOT justify reproducibility when additional consumed remote content is outside that revision. An attempt that fails or is interrupted before source readiness MUST instead record that reproducibility was not established and MUST NOT synthesize either execution conclusion.

Materialization evidence MUST remain in the workload child that performs it and MUST reference the applicable preparation-attempt identity. Eval and AgentX case-owned materializations MUST be attached to their case. Independent-request and linear-session Bench-wide frozen-population materialization MUST be attached once to the Bench child before its first case, and every consuming case MUST reference the same materialization identity. It MUST preserve the effective lm-eval task and reported dataset fingerprint, AIPerf loader or file-format identity and derived-cache identity, or InferLab frozen-population identity when those facts apply. A derived identity MUST NOT be presented as the acquired-source identity.

For a recipe-owned workload, the source-preservation obligations of RFC-0005:C-LM-EVAL-EVIDENCE, RFC-0005:C-BENCH-REQUEST-SOURCE-EVIDENCE, RFC-0005:C-BENCH-LINEAR-SESSION-EVIDENCE, and RFC-0005:C-BENCH-AGENTIC-TRACE-EVIDENCE MUST be satisfied by a resolvable reference to the authoritative recipe preparation entry. A standalone workload MUST preserve those preparation facts directly. A compatibility projection copied into a child MUST be derived from the authoritative entry and MUST match it; it MUST NOT become a second authority. Existing task, population, topology, request, metric, and native-artifact authorities remain unchanged.

Preparation evidence MUST be durable incrementally. The owner MUST write the selection key, attempt identity, consumers, and planned state before the first external preparation effect. Within one attempt, it MUST durably record each externally observable phase result, including resolution and cache observations, before starting that attempt’s next acquisition or verification phase. Different attempts MAY proceed concurrently. The owner MUST durably record each attempt’s terminal success, failure, or interruption before returning control to the workflow. After interruption, the record MUST report only the last durably committed facts and MUST NOT synthesize an outcome for an effect whose result was not committed.

A recipe source-preparation failure MUST identify that no serving process was launched. A manual-Bench source-preparation failure MUST identify that its existing target server remained outside preparation ownership. A later materialization failure MUST reference successful or opaque source readiness and MUST retain its owning classification: Eval or AgentX case failure, or independent-request or linear-session pre-case Bench-child failure.

Dry-run output MUST use planned preparation projections rather than terminal preparation entries. It MUST distinguish existing local observations and pure deterministic materialization facts from planned external work and unavailable facts. It MUST NOT claim successful acquisition, verification, immutable binding, reproducibility, or persisted materialization merely because those actions are planned.

Rationale: A source selection exists before its final content identity is known, and one prepared source can feed measurement-owned representations at different scopes. Versioned parent ownership, stable attempt and materialization references, and incremental durability preserve that lifecycle without duplicating authority or confusing source bytes with consumer representations.

Since: v0.22.0