Specification · RFC
RFC-0003: Server, Recipe, And Runtime Execution Contract
Version: 0.31.0 | Status: normative | Phase: spec Owners: @Zihua Wu
1. Summary
Section titled “1. Summary”[RFC-0003:C-SUMMARY] Summary (Informative)
Section titled “[RFC-0003:C-SUMMARY] Summary (Informative) ”This RFC governs independently launchable servers, server-owned cases, recipes, and runtime execution: typed resolution before external effects, long-running and closed-loop lifecycles including image-backed launches, canonical Engine, Gateway, and P/D Router components, per-role parallelism, and fused P/D frontend binding. RFC-0001 holds the specification overview and authority map.
Since: v0.1.0
2. Specification
Section titled “2. Specification”[RFC-0003:C-RECIPE-COMPOSITION] Server, Case, And Recipe Composition (Normative)
Section titled “[RFC-0003:C-RECIPE-COMPOSITION] Server, Case, And Recipe Composition (Normative) ”A server MUST be an independently launchable workspace definition. Its stack, model, and topology fields MUST select exactly one stack, model, and supported topology; its common behavior and canonical model-serving roles supply the typed base inherited by its cases.
A single topology MUST use the canonical model-serving role serve. When its server base declares gateway_backend, resolution MUST additionally derive the canonical frontend process role gateway; when it omits gateway_backend, resolution MUST derive no frontend process role. A single server MUST NOT declare pd_router_backend.
A prefill_decode topology MUST use canonical model-serving roles prefill and decode. Its server base MUST declare independently selected gateway_backend and pd_router_backend values. Resolution MUST derive one canonical gateway process role whose closed component binding contains both Gateway and P/D Router.
Workspace configuration MUST reject roles.gateway, roles.pd_router, roles.router, and every other authored frontend role. It MUST NOT permit alternate names for canonical roles or a public role kind field; topology and backend presence determine role identity and cardinality.
A server MAY define identifier-keyed cases and an optional default_case. Each case MUST be a typed behavior patch over that server. A case MUST NOT change the selected stack, model, topology, canonical role identities, case set, placement, model locator, presence of either frontend component, or canonical frontend process-role cardinality. An empty case is valid.
A declared default_case MUST reference one owned case. A server with no cases MUST resolve its base behavior when case selection is omitted and MUST reject --case. A server with exactly one case MUST select that case when case selection is omitted. A server with more than one case and a valid default_case MUST select it when case selection is omitted; without one, such a server MUST reject an omitted case selection as ambiguous. Explicit case selection MUST name one case owned by the selected server.
A recipe MUST select exactly one server and one workload_suite. It MUST NOT own, copy, filter, reorder, or redefine the server’s cases. A recipe MAY declare case, which MUST reference one case of its server. A run of the recipe MUST select that case unless the invocation selects one explicitly with --case, which takes precedence; the resolved case MUST record whether the invocation, the recipe, the server’s default_case, or its sole case selected it.
A workload suite MUST name an ordered sequence of Eval and Bench measurements and MAY name one Eval as its gate. A gated Bench is skipped when the gate does not succeed (RFC-0004:C-MEASUREMENTS).
Every referenced model, stack, server, measurement, workload suite, and selected case MUST resolve within the selected workspace revision before the first external effect.
An ordinary operator MUST be able to select a server directly or select a recipe and, when applicable, one server case without manually composing referenced definitions.
A recipe MUST define bounded server-plus-measurement composition. The public contract MUST NOT require users to author arbitrary task nodes or a general workflow DAG.
Rationale: Server intent has an independent lifecycle and reuse boundary: the same server is started directly, consumed by multiple recipes, attached to a manual Bench, and referenced by image validation. Server-owned typed cases vary behavior without giving recipes a second authority. Fixed model-serving roles and one closed frontend process role let placement, records, and integrations share one vocabulary without turning topology into a general graph language. The server base fixes component presence and process cardinality while cases compare compatible Gateway and P/D Router backends independently.
Since: v0.1.0
[RFC-0003:C-RESOLUTION] Resolution Dry Run And Overrides (Normative)
Section titled “[RFC-0003:C-RESOLUTION] Resolution Dry Run And Overrides (Normative) ”Server resolution MUST use one ordered path: load the workspace; select the server, case, stack, and model; load the permitted local bindings and select placement; merge typed behavior patches; validate realization compatibility; invoke integration planning; allocate ranks, machine-specific model locators, ports, caches, and network facts; invoke final integration rendering; then freeze one immutable resolved execution object.
Behavior precedence MUST be server base, then the selected server case, then invocation-scoped overrides in command-line order. Resolution MUST preserve the ordered common and role declaration layers and the source of every effective behavior-affecting value.
The effective Gateway-backend and P/D-Router-backend identities MUST each be the corresponding backend identity after server-base, selected-case, and invocation-override precedence. They MUST remain separate facts even when they name the same provider or share one process.
For a single server, the base definition’s gateway_backend presence MUST determine whether resolution derives the canonical gateway process role. A case or invocation-scoped override MAY replace the effective Gateway-backend identity only when the server base declares gateway_backend. It MUST NOT introduce or remove the Gateway component or introduce a P/D Router component.
For a prefill_decode server, the base definition MUST declare both backend fields. A case or invocation-scoped override MAY replace either effective backend identity independently. It MUST NOT introduce or remove either frontend component or change the one derived frontend process role.
An invocation-scoped override MUST use repeatable --set <TOML-KEY-PATH>=<TOML-VALUE>. TOML quoted key segments MUST allow a literal dot in a key. Values MUST be parsed as TOML values. Repeated overrides apply left to right, and the last assignment to the same path wins.
A serve start override path MUST be rooted at the selected server through server. A recipe run override path MAY be rooted at server, evals.<eval-id>, or benches.<bench-id> under RFC-0004:C-RECIPE-MEASUREMENT-OVERRIDES. Framework setting paths MUST use server.settings.<key> for common values or server.roles.<role>.settings.<key> for role values. A core server field name MUST NOT be reinterpreted as a framework setting.
Cases and invocation overrides MUST use the same typed patch model. An override MAY change only typed server behavior fields and entries under an explicit framework settings map. It MUST NOT change server identity, stack, model, topology, canonical role identities, case definitions, case selection, placement, model locators, frontend-component presence or canonical frontend process-role cardinality.
Workspace server configuration MUST declare a positive framework-neutral readiness_timeout_seconds independently of framework settings. An integration response MUST NOT replace it. Ordinary launch readiness MUST be finite. Its owning budget MUST begin immediately after every resolved process has spawned and immediately before the first blocking process-status, HTTP-readiness, or target-registry attempt. A capture-armed launch waits without that owner budget under RFC-0004:C-WORKLOAD-PROFILING.
Workspace server configuration MUST also accept a positive framework-neutral readiness_attempt_timeout_seconds, defaulting to 30 seconds when undeclared. The value MUST bound each blocking process-status, HTTP readiness, and target-registry attempt during readiness; one HTTP attempt includes connection establishment and the complete response. For finite readiness, the effective attempt deadline MUST be the lesser of this value and the readiness owner’s remaining budget. Expiry of a blocking attempt MUST return control to the finite readiness loop and MUST NOT by itself terminate readiness. For capture-armed readiness, attempt expiry MUST return control to the unbounded readiness loop without making that loop terminal.
Server cases and invocation-scoped server overrides MUST be able to patch readiness_timeout_seconds through the typed server patch model.
Server cases and invocation-scoped server overrides MUST be able to patch readiness_attempt_timeout_seconds through that same typed server patch model. The frozen resolved server, dry-run, and record MUST preserve its effective value, and an integration response MUST NOT replace it.
A profiling-capable server MUST resolve positive capture_arm_deadline_seconds, capture_control_deadline_seconds, and capture_finalization_deadline_seconds facts. The arm and control budgets MUST default when undeclared to 60 seconds each. The finalization budget default MUST follow the resolved capture mechanism under RFC-0004:C-WORKLOAD-PROFILING: 300 seconds for managed collection and 3600 seconds for engine trace, whose stop dispatch, response consumption, and artifact flush wait share that one budget. Server cases and invocation-scoped server overrides MUST be able to patch all three through the typed server patch model. The frozen resolved server, dry-run, and record MUST preserve their effective values.
The selected realization MUST be validated against the resolved server before any state-affecting command. A local realization MUST use the server stack’s Pixi environment. A built image MUST realize the same stack. A declared external image MUST carry an immutable digest and claim an integration compatible with the server stack; it does not claim stack equality.
Local integration operations MUST execute in the selected stack’s Pixi environment. Image-backed integration operations MUST execute against the selected image realization. Integration operations and external probes used during resolution MUST be read-only.
The frozen resolved object MUST be the sole authority for dry-run output, state-affecting execution, cleanup, and record production. Execution MUST NOT reload configuration, reselect defaults, rerun placement choice, or reconstruct server, case, adapter, endpoint, command, or measurement facts after that object is frozen.
For an adaptive Bench, the frozen parent resolved object MUST own the complete effective definition, endpoint, immutable case inputs, SLOs, and adaptive policy.
Each outcome-dependent adaptive probe MUST be derived only from that frozen parent and the recorded prior case-level SLO evaluations, and MUST itself be frozen as one immutable child case execution object before its first external effect.
Adaptive child-case derivation MUST NOT reload workspace configuration, reapply overrides, or reconstruct a fact already frozen in the parent.
Dry-run MUST perform the same reference resolution, case selection, typed patching, placement compatibility, determinable resource and toolchain validation, integration planning and rendering, rank allocation, locator resolution, endpoint allocation, cache allocation, and command construction required before real execution. It MUST NOT launch a server, execute a measurement, acquire persistent server-operation ownership, execute realization checks, or create a record that claims execution occurred.
Manual Bench resolution MUST combine one named Bench definition from the discovered workspace root, one explicit running server record stored under that root, the immutable endpoint, model, and server-control capabilities in that record, the Bench toolchain owned by the invoking InferLab release, and invocation-scoped Bench overrides.
A manual Bench MAY use a newer or dirty measurement workspace snapshot than the serving snapshot. It MUST reject a target record whose record schema or preserved server capabilities are incompatible with the invoking release.
A manual Bench override MUST use repeatable --set <path=value>, patch the selected typed Bench definition in command-line order, and be fully validated before case expansion. It MUST NOT change Bench identity or kind, target server record, endpoint, model, serving integration, or local bindings.
A Bench that requests prefix-cache reset MUST fail preflight when the target server record does not preserve a compatible reset capability.
For a static Bench matrix, dry-run MUST report every exact expanded case in execution order and the resolved aggregate and request-level SLO inputs that apply to those cases.
For an adaptive Bench, dry-run MUST report the initial probes, resolved aggregate and request-level SLOs, max_search_steps, configured minimum rate resolution, and adaptive policy identity.
Each adaptive next case MUST be resolved from recorded prior case-level SLO evaluations according to that reported policy before it starts.
Rationale: One resolution path prevents dry-run, execution, and evidence from drifting. TOML-path overrides retain rapid experiment iteration without creating persistent local behavior or an untyped shell surface. Fixing frontend-component presence in the server base keeps placement stable while independently patchable Gateway and P/D Router backend identities remain explicit experimental inputs. Freezing after both integration phases means every later external action consumes an already validated, fully concrete execution rather than rediscovering configuration under changing files or defaults.
When both a base layer and an overriding layer declare the framework settings entry extra_args, the two arrays MUST compose by per-flag group last-wins merge rather than wholesale replacement. Each layer’s array MUST be segmented into flag groups: a ---prefixed token opens a new group named by the flag text before any =, and following tokens without a -- prefix belong to that group; a value token preceding any flag MUST fail workspace validation with a typed error. An overriding group MUST replace the same-named earlier group in that group’s original position; groups named only by the overriding layer MUST append in override order; groups absent from the overriding layer MUST be inherited unchanged, so cases are additive over the server base and no removal path exists. A bare -- token and every token after it MUST form one verbatim passthrough block that an overriding layer replaces as a whole. Every other settings array MUST keep wholesale replacement. The composed array remains subject to integration validation of InferLab-owned options, and dry-run, execution, and records MUST preserve the final composed contents. The bare -- sentinel is a composition marker only: an integration MUST NOT emit it into the rendered engine command, and MUST append the post-sentinel tokens verbatim after the managed command tail so engine last-wins parsing applies the deliberate override. Last-wins parsing cannot retract a managed store-true flag, so when a post-sentinel token names a flag from a managed flag group the integration documents as verbatim-replaceable, the integration MUST suppress its managed rendering of that group entirely and MUST record the suppression with the rendered command; the verbatim block then owns the group’s spelling.
Since: v0.1.0
[RFC-0003:C-RUNTIME-WORKFLOWS] Runtime Workflows (Normative)
Section titled “[RFC-0003:C-RUNTIME-WORKFLOWS] Runtime Workflows (Normative) ”InferLab MUST support a long-running server workflow in which an operator starts one resolved server behavior, observes its status and logs, and explicitly stops it.
InferLab MUST support a closed-loop recipe workflow in which InferLab resolves the recipe’s server and effective case selection, starts it, waits for readiness, executes the workload suite, and stops the server.
Both workflows MUST use the same server and case resolution, validation, integration planning and rendering, launch, readiness, runtime-handle, and cleanup behavior.
A successfully started long-running server record MUST remain running until an explicit stop or recovery action finalizes it. Its resolved recipe identity MUST be null. A recipe record MUST preserve its recipe identity and selected server and case.
InferLab is an experiment harness, not a production supervisor. It MUST NOT automatically restart experiment servers or claim availability guarantees.
During a live start or closed-loop command, InferLab MUST attempt cleanup after handled launch, readiness, measurement, interruption, or cancellation failure. A closed-loop recipe run MUST also attempt cleanup after success. Recovery after the InferLab process or host terminates before a runtime handle is durable MAY require operator cleanup, and the record MUST NOT claim verified cleanup.
The public runtime CLI MUST provide:
serve start <server>, accepting optional--case,--placement, mutually exclusive--imageand--external-image, repeatable--set,--local, and--dry-run;serve status <record>,serve logs <record>, andserve stop <record>, accepting--force;recipe stop <record>, accepting--force;recipe run <recipe>, accepting the same server realization selectors plus--capture;bench <bench> --serve <record>, accepting repeatable--set,--capture, and--dry-run;--detachonserve start,recipe run, andbench, andjob interrupt <job>;--wait-for-devicesonserve startandrecipe run.
--detach MUST be rejected together with --dry-run. A detached invocation MUST launch, as a job of its workspace under RFC-0012:C-ACTIONS, the same executable with the invocation’s argument vector less --detach, run from the invocation’s working directory. A job that cannot start MUST fail the invocation with that error and leave no job directory. Once the job has started, the invocation MUST name its job identifier and job directory in its diagnostics and MUST wait for one of two outcomes. When the job’s command has created its first top-level record — the server record for serve start, the recipe record for recipe run, the Bench record for bench — the invocation MUST print as its machine-readable output an object with job_id, job_dir, stdout, stderr, record_id, and record_dir, and MUST exit successfully. When the job ends without creating that record, the invocation MUST exit unsuccessfully, MUST report how the job ended — its exit status or terminating signal — and MUST include the job’s stderr and job directory in its diagnostics. The wait MUST be governed as unbounded and terminable by the job’s end and by operator interruption under RFC-0009:C-OPERATION-BUDGETS; an interrupted wait MUST exit unsuccessfully, naming the job directory, and MUST NOT signal the job. job interrupt <job> MUST deliver to a job of the workspace the interrupt a terminal Ctrl+C delivers, MUST first validate the job’s recorded producer identity as RFC-0012:C-ACTIONS requires of a console cancel, MUST NOT escalate to a kill, and MUST fail naming the reason when nothing was signalled. Like serve status, job interrupt MUST NOT load shared workspace configuration or local bindings.
With --wait-for-devices, a launch whose assigned devices report compute applications under RFC-0005:C-EVIDENCE MUST NOT fail for that reason. It MUST instead probe every assigned machine again at an interval until no assigned device reports a compute application, and MUST then continue the launch. While it waits, it MUST report on diagnostics, at each probe, the occupants with the fields RFC-0005:C-EVIDENCE requires of the occupied-device failure. A probe failure MUST still fail the launch. The wait MUST be governed as unbounded and terminable by operator interruption under RFC-0009:C-OPERATION-BUDGETS; an interruption during the wait MUST fail the launch as interrupted before any serving process spawns. The wait orders nothing: launches waiting on the same devices MAY all proceed once the devices free. Dry-run records the selection; it neither runs the launch-time device check nor waits.
The process that creates a recipe record MUST hold that record’s controller lock until the recipe is finalized, and the process that starts a server MUST hold the server record’s controller lock from the record’s creation until the server runs or its start has failed. Each controller MUST record its host, process ID, start time, and command beside the lock, and MUST honor a stop request placed in a record whose lock it holds, within its stop-request polling interval, by following its operator-interruption path. serve stop of a server record whose status is starting, and recipe stop of a recipe record whose status is running, MUST act on the record’s controller lock. When the lock is free, the controller is gone: the command MUST stop what the record started — for a recipe, its server when that server’s record is not terminal, through serve stop — and finalize the record as failed and stopped by the operator. When the lock is held, the command MUST report the holder, place a stop request, and wait until the controller releases the lock, then report the record as the controller finalized it; the wait MUST be governed as unbounded and terminable by operator interruption under RFC-0009:C-OPERATION-BUDGETS, and interrupting it MUST leave the request standing. With --force, the command MUST report the holder and proceed as though the lock were free, for a controller whose host is gone while its lock persists. recipe stop of a terminal recipe record MUST change nothing and report it. These locks rely on the workspace file system’s advisory locking across the hosts that share it. serve stop of a running server is unchanged.
serve start MUST select the named server directly. recipe run MUST select the server named by the recipe. Both MUST resolve exactly one effective server behavior and zero or one case identity under RFC-0003:C-RECIPE-COMPOSITION.
--local MUST be accepted only by serve start, recipe run, and image build among these governed runtime workflows. Server status, logs, and stop MUST resolve lifecycle facts solely from the named record and MUST NOT load shared workspace configuration or local bindings.
SSH execution MUST remain noninteractive while honoring the selected target’s OpenSSH configuration as the authority for connection timeout and keepalive policy. InferLab MUST NOT inject fixed connection-timeout or keepalive values that override that target configuration. An SSH child invoked inside an InferLab operation or cleanup deadline MUST remain supervised by that owner’s remaining bound. The SSH client child is a host tool, not a member of the serving stack: InferLab MUST strip the dynamic-linker override variables LD_LIBRARY_PATH and LD_PRELOAD from the SSH subprocess environment so the workspace’s Pixi or stack activation environment cannot corrupt the host SSH client, and MUST NOT rely on PATH indirection or wrapper scripts for that isolation.
After the resolved object is frozen and a durable record exists, real launch MUST execute stack realization checks and machine hardware probes and materialize launch files. When the topology contains a discovery process, launch MUST spawn it first and MUST wait until its declared readiness succeeds before spawning any model-serving process. That wait consumes the server’s single readiness budget, which later readiness evaluation continues rather than restarts; a server prepared with capture targets waits without that bound under RFC-0004:C-WORKLOAD-PROFILING. A discovery readiness failure MUST fail the launch and roll back before any model-serving process spawns. Launch MUST then spawn model-serving ranks in canonical order: serve, or prefill then decode, with replica and rank indices ascending. When present, the one gateway process MUST spawn after all model-serving ranks. A P/D frontend MUST bind Gateway and P/D Router to that process and spawn it exactly once. Readiness evaluation of model-serving and gateway processes MUST begin only after every resolved process has spawned. Rollback and cleanup MUST proceed in reverse spawn order, so a discovery process is stopped after every process that depends on it.
A Gateway-backed topology MUST NOT be reported ready until every required model-serving backend, its P/D Router when present, and its public Gateway endpoint are ready. A handled partial-start failure MUST attempt rollback of every process already started. Until the server is reported ready, the exit of any started process — including one whose own readiness already succeeded — MUST fail the start promptly rather than after the readiness budget, and the failure MUST name the exited process.
serve start and recipe run MAY select a successful image build record. The selected host-platform assembly substitutes for the local stack realization through the same containerized substitution used by image validation (RFC-0007:C-IMAGE-BUILD); all other server resolution, measurement, evidence, and cleanup semantics remain unchanged.
Image-backed resolution MUST reject profiling capture until this specification defines an in-container profiler contract, because a host profiler wrapping the container client does not observe the server process.
Built-image and external-image integration planning and rendering MUST execute inside the selected image through its one-shot JSON contract, using the workspace adapter environment’s packages mounted read-only: the selected integration package and every InferLab package it requires. Lowering MUST follow the invoking InferLab and the workspace’s adapter pins, never adapter packages baked into an image, so an image built or published for one InferLab protocol version launches under a later one; the image supplies the engine and its runtime. It MUST NOT require a locally installed stack environment. A failed or timed-out adapter invocation MUST NOT leave its container running.
Local bindings MAY declare positive adapter.timeout_seconds and adapter.image_timeout_seconds values for local-process and image-backed adapter invocations respectively. Omission MUST resolve to 30 seconds for a local-process invocation and 120 seconds for an image-backed invocation. The local-process deadline MUST begin immediately before spawning the adapter process in the selected stack environment. The image-backed deadline MUST begin immediately before requesting creation of the adapter container. Each deadline MUST cover the complete one-shot adapter response and process outcome, and adapter timing evidence MUST preserve its configured budget, elapsed time, and terminal cause. Adapter process or container cleanup MUST remain separately governed by RFC-0009:C-CLEANUP-GRACE. Local bindings MAY also declare, only for a host whose container runtime rejects device-less container creation, one workaround device. The adapter container MUST request no device when none is declared and no capability beyond that workaround.
A built-image selection MUST be rejected before any state-affecting command or integration invocation when the record’s image stack differs from the server stack, when it has no successful assembly for the launching host platform, or when the selected placement is not entirely local to the machine whose builder storage holds the image. InferLab defines no distribution of a built image between machines.
A built-image server record MUST preserve the immutable image identity, image build record, image workspace revision, and invoking workspace revision so qualification drift remains observable.
A containerized launch MUST receive every resolver- and integration-set process environment variable explicitly, with its resolved value verbatim, and MUST NOT infer that set by comparison with the ambient environment. Those variables apply only inside the container: the container client process MUST run under its launch machine’s own environment and declared pass-through names, and MUST NOT receive the container’s variables.
Each external_images definition MUST contain a digest-pinned reference and an integration claim, MAY declare entrypoint as replace (the default) or image, and MUST NOT claim a stack. A declaration without an immutable digest, with an invalid integration identifier, with any additional field, or with a colliding identifier MUST be rejected at workspace load.
An external image and built-image selection MUST be mutually exclusive. External-image resolution MUST validate the claimed integration against the server stack’s integration, not claim stack equality. It MUST verify by read-only probe that the digest exists in builder storage on the controller and every launch machine, MUST NOT pull it, and MUST fail a missing image naming the machine and exact pull command.
An external-image placement MAY span machines. A remote containerized launch MUST NOT require a remote workspace realization: the image replaces the serving stack environment and the remote workspace path is only a launch directory. Machine-scoped user, launch, environment pass-through, device, capability, and cache facts MUST resolve on each launch machine and enter the record.
Each declared environment pass-through value MUST be resolved on its launch machine; an absent value MUST be reported naming that machine. On a remote launch, the value MUST expand on that machine and MUST NOT enter the resolved plan, recorded command, or launch-script text. A local launch records its effective environment under the standing unredacted-record posture.
The containerized substitution MUST lower each launch machine’s declared container hardware facts (RFC-0007:C-IMAGE-BUILD) into every server container: host devices, unlimited pinned-memory ulimit, capabilities, and the cpuset CPUs and memory nodes of the machine binding’s declared NUMA nodes (RFC-0002:C-LOCAL-PLACEMENT). It MUST NOT request privileged mode.
Containerized processes launched locally or over SSH MUST carry a resolver-assigned container name as a runtime handle. Cleanup verification MUST confirm removal on the launch machine; an already absent container is confirmation. A launch failure after a container may exist MUST attempt removal using that durable name. Killing only the container client MUST NOT be reported as verified server cleanup.
Every server process InferLab launches as a process group, locally or over SSH, MUST carry the environment variable INFERLAB_SERVER_PROCESS set to its ownership marker: the server record identifier and the process’s identifier in the resolved server topology, joined by /. A marker MUST match only as the exact whole value. InferLab MUST set the variable after every declared and passed-through variable, so no other value replaces it, and the process’s runtime handle MUST record the marker. As part of every cleanup of such a process, InferLab MUST, on its launch machine, send SIGTERM to each live process the launching user can inspect whose initial environment sets INFERLAB_SERVER_PROCESS to exactly the marker that runtime handle records; wait within the same termination grace as process-group cleanup; send SIGKILL only to processes that still match when it is sent; and verify, as a completion wait under RFC-0009:C-COMPLETION-WAITS, that no matching process remains. Each enumeration and signalling attempt of the sweep, including over SSH, MUST have a finite attempt bound under RFC-0009:C-COMPLETION-WAITS. Cleanup MUST NOT be reported verified while a matching process remains or the sweep cannot be completed. A process whose initial environment lacks the marker, such as one executed with a cleared environment, or that runs as another user, is outside this ownership; a containerized process is covered by container removal. A runtime handle that records no marker predates this rule and is cleaned up by its process group alone.
A built image MUST run through its activation entrypoint. An external image whose entrypoint is replace MUST run its serving processes through an explicit command override and assumes nothing about the image’s own entrypoint. An external image whose entrypoint is image MUST run its serving processes without an entrypoint override, passing the rendered command as the arguments of the image’s own entrypoint. InferLab’s own adapter and framework-version probes in an external image MUST use an explicit command override in either case. External-image adapter lowering MUST use the workspace-pinned integration packages from the committed framework-free Pixi environment named adapter, made available to the adapter operation without treating that environment as a serving stack or requiring the serving framework locally. Resolution MUST observe and record the framework version in the external image and reject an image in which the claimed framework is not observable.
An external-image launch is not qualified by the workspace. Its record MUST preserve a distinct realization state, make no stack-check claim, and carry the declaration identifier, digest, integration claim, and observed framework version. Manual Bench admission MUST otherwise treat local, built-image, and external-image managed servers identically.
serve start, recipe run, and bench MUST support dry-run under RFC-0003:C-RESOLUTION.
A manual bench MUST target one explicit managed server record, MUST NOT select an implicit latest server, and MUST reject a record not in running state or whose typed runtime handles do not identify live processes. It MUST NOT require a separate endpoint-readiness probe; endpoint request failure remains Bench failure evidence.
No more than one manual Bench may execute against one managed server at a time. Bench acceptance and serve stop acceptance for that server MUST be mutually exclusive. Every handled Bench terminal outcome MUST finalize its record, terminate owned measurement process groups after timeout, interruption, or cancellation, and release server-operation ownership.
A measurement client MUST have a durable process-group handle with leader start-time identity before its first experiment effect. A later run in the same workspace MUST terminate an identity-matching survivor of an unclean exit and MUST NOT signal a process group whose identity no longer matches.
Stale operation ownership MUST NOT permanently block a later Bench or stop. Recovery MUST verify that no live owner remains and preserve its outcome when an operation record exists.
This specification does not define separate serve-scoped Eval or Bench subcommands.
Rationale: Detached invocations reuse the console’s jobs, so one job directory contract serves both entry points and an hours-long run outlives the session that started it; returning once the first record exists hands the caller the identity every later lifecycle command takes. A process group is the launch command’s ownership, not the serving tree’s: a launch tool can start the serving tree outside that group — a managed Nsight Systems collection starts its target in a new session — beyond any group signal, while every descendant that keeps its inherited environment carries the marker, so the marker names what the server owns wherever it runs. The two server workflows differ only in who owns the server after readiness; direct server selection no longer requires a synthetic recipe. Record-only lifecycle commands remain operable from frozen evidence even if configuration changes. Deterministic spawn and reverse cleanup order make heterogeneous multi-process failures diagnosable. Built and external images reuse the same resolved server contract while keeping their distinct distribution and qualification claims honest.
Since: v0.1.0
[RFC-0003:C-SERVE-TOPOLOGY] Serve Topology And Frontend Binding (Normative)
Section titled “[RFC-0003:C-SERVE-TOPOLOGY] Serve Topology And Frontend Binding (Normative) ”InferLab MUST support two public server topologies: single and prefill_decode. They are product workflows, not arbitrary role or component graphs.
A resolved topology MUST distinguish logical serving components from the concrete processes that realize them. Model-serving roles MUST remain hierarchical as role, replica, then rank. Every model-serving process MUST belong to exactly one role, one replica, and one rank. Frontend processes MUST use the closed component bindings defined by this clause and MUST NOT introduce arbitrary component sets.
A single topology MUST contain one or more serve replicas and MUST NOT select a KV-transfer mechanism. A single topology without a Gateway MUST contain exactly one serve replica; a routed single topology MAY contain several, and its integration MUST reject a replica count it does not support during planning.
A single server whose base definition omits gateway_backend MUST expose its serve replica’s rank-zero endpoint directly. It MUST have no Gateway, P/D Router, or frontend process.
A single server whose base definition declares gateway_backend MUST contain one logical Gateway and exactly one derived gateway process with component binding [gateway]. That process endpoint MUST be the public workload endpoint. The topology MUST have no P/D Router.
A prefill_decode topology MUST contain independent positive replica counts for prefill and decode, one logical Gateway, and one logical P/D Router. It MUST select gateway_backend, pd_router_backend, and KV transfer independently. The KV-transfer mechanism (kv_transfer) is a name, not a closed set: a non-empty string of lowercase ASCII letters, digits, -, and _. The control plane MUST reject any other spelling when it loads the workspace; each integration MUST accept only the mechanism names it supports and reject every other name during resolution before launching a process, and the declared name MUST be recorded unchanged.
A prefill_decode topology MUST contain exactly one derived gateway process whose closed component binding is [gateway, pd_router]. The two logical plans MUST share that process identity, placement, runtime handle, launch command, logs, exit status, and cleanup outcome without aliasing their backend identities or effective semantics.
Each model-serving role’s declared replica count MUST equal its resolved replica count. Replica identity MUST be unique within a role. Each replica MUST contain one or more ranks whose indices are unique and contiguous from zero.
Each model-serving replica MUST have exactly one rank-zero serving entry point. Nonzero ranks are headless participants and MUST NOT be treated as independently routable backends. Gateway dispatch, P/D routing, KV transfer, bootstrap, and side-channel targeting MUST use rank-zero endpoints unless a backend-specific clause explicitly defines a non-routing internal dependency.
Every derived gateway frontend process MUST have process-only identity with no replica or rank coordinates, have empty effective parallelism, demand zero devices, and carry no model locator. Workspace server configuration MUST NOT declare it directly.
A Gateway-backed server whose selected Gateway backend declares a discovery requirement under RFC-0006:C-INTEGRATIONS MUST contain one logical discovery service and exactly one derived discovery process with closed component binding [discovery]. A server whose selected Gateway backend declares no discovery requirement MUST have no discovery process.
The derived discovery process MUST have process-only identity with no replica or rank coordinates, empty effective parallelism, zero devices, and no model locator. It MUST be placed on the machine of the gateway process. It is not a frontend process. Workspace server configuration MUST NOT declare it directly.
The discovery service MUST NOT be a public workload endpoint and MUST NOT be the target of Gateway dispatch, P/D routing, KV-transfer, bootstrap, or side-channel links.
A single model-serving replica MAY span machines through explicit local rank placement. Different roles and replicas MAY have different device counts, rank counts, machine sets, ports, settings, and parallelism. The gateway process and, when present, the discovery process each have exactly one placement.
Directed component links MUST distinguish public Gateway dispatch, P/D request routing, KV transfer, bootstrap, side-channel, and discovery semantics. When a discovery process exists, every model-serving process and the gateway process MUST each have exactly one discovery link to it. A routed single topology MUST link Gateway to every serve replica’s rank-zero target. A prefill_decode topology MUST preserve the logical Gateway-to-P/D-Router handoff within the fused process and MUST link P/D Router to the complete prefill and decode rank-zero target sets. A link between prefill and decode sets MUST NOT imply one-to-one replica pairing; every actual target and routing policy MUST be explicit in the resolved result.
The selected integration MUST plan canonical model-serving roles and return separate Engine-role, Gateway, and P/D Router results when they apply. It MUST preserve requested replica counts and both backend identities; return complete effective behavior, whole-replica device counts, compatible co-rendering requirements, rendering sources, directed links, ports and endpoint kinds, readiness, profiler controls, server-control actions, and render inputs; and MUST NOT choose concrete model rank decomposition, machines, devices, port values, model locators, or runtime storage.
The control plane MUST apply RFC-0002:C-LOCAL-PLACEMENT, derive the closed frontend process binding from the topology, allocate ranks, machines, devices, ports, model locators, cache roots, and network facts, then supply each integration-rendered concrete allocation to final integration rendering.
The selected integration MUST render exactly one process for every supplied allocation and no additional process. It MUST preserve each model allocation’s process, role, replica, and rank identity and each frontend allocation’s process role and closed component binding. A control-plane-rendered frontend allocation MUST NOT be supplied to integration rendering.
The control plane MUST validate model-role identities, replica cardinality, rank continuity, frontend component presence, independent backend identities, the closed component-to-process binding, rendering sources, links, resource totals, launch dependencies, endpoint authority, and public endpoint uniqueness before the first launch.
A resolved topology MUST expose exactly one public workload endpoint: the sole serve rank-zero endpoint for direct single serving or the Gateway endpoint for every Gateway-backed topology. Single and prefill/decode topologies MUST use the same runtime-handle, rollback, readiness, status, logs, stop, recovery, and cleanup semantics.
The vLLM integration MUST support prefill_decode over Mooncake and NIXL with independent positive prefill and decode replica counts and the backend-pair obligations in RFC-0003:C-GATEWAY-PD-ROUTER.
The required logical-component, role, replica, rank, frontend-binding, allocation, and rendering shapes MUST use the current adapter protocol version under RFC-0006:C-INTEGRATIONS. An adapter advertising an earlier protocol version MUST NOT emit or consume them.
Rationale: Gateway, P/D Router, and Engine describe responsibilities with independent backend authority; replicas describe independently routable Engine instances; ranks describe model-serving processes. The two closed frontend bindings express direct Gateway and fused P/D frontend processes without creating a general deployment graph. Independent backend selections and plans preserve the semantic boundary even when one process delivers both components.
Since: v0.1.0
[RFC-0003:C-SERVE-PARALLELISM] Component-Aware Serve Parallelism (Normative)
Section titled “[RFC-0003:C-SERVE-PARALLELISM] Component-Aware Serve Parallelism (Normative) ”A server, its canonical model-serving roles, its cases, and invocation-scoped overrides MAY declare component-aware parallelism patches and a devices count.
The component-aware shape MUST expose these optional positive-integer components: outer.tensor_parallel_size, outer.pipeline_parallel_size, attention.tensor_parallel_size, attention.data_parallel_size, attention.context_parallel_size, experts.tensor_parallel_size, experts.data_parallel_size, experts.expert_parallel_size, and experts.dense_tensor_parallel_size.
A server-level parallelism shape is only an inheritance patch for model-serving roles. For each role, resolution MUST merge server common values, server role values, selected-case common and role values, and invocation-scoped values under RFC-0003:C-RESOLUTION. The resulting per-role request is the declared role shape.
An operator MUST be able to override one component without repeating unaffected components.
devices is a positive integer that MAY be declared as a server common or role value, a case common or role value, or an invocation-scoped value. It resolves per role under RFC-0003:C-RESOLUTION by the same layering as each parallelism component and independently of them. A common value applies to every model-serving role, including both prefill and decode. Declaring devices MUST NOT clear or change any inherited parallelism component. devices applies only to model-serving roles; derived gateway and discovery processes keep zero devices under RFC-0003:C-SERVE-TOPOLOGY.
When a role’s resolved request declares devices, that value MUST be the sole authority for the role’s whole-replica device count. The selected integration MUST either fill the omitted components into a complete effective shape whose device count under its own reading equals devices — pure tensor parallelism of width devices when no component is declared — or reject the declared shape as unsupported or inconsistent with devices before launch. The control plane MUST reject a plan whose returned whole-replica device count for that role differs from devices. A placement failure for such a role MUST name devices and the layer that declared it. When devices is not declared, the integration MUST derive the whole-replica device count from the role’s complete effective shape. An integration whose Engines own their layout MAY return only the role’s device count and no effective shape, provided it rejects declared components and requires devices; its reading in the backend support matrix is that components are rejected.
The parallelism components are a mapping onto framework options. Each integration owns its reading of them and MUST state that reading in the backend support matrix. The same declared shape MAY imply different device counts under different integrations; declaring devices makes the device count independent of that reading. The control plane takes the attention data-parallel rank count from attention.data_parallel_size, or, for a role without an effective shape, from the count each of its Engines reports at readiness. The components are deprecated in favor of devices together with framework-owned layout settings; they remain accepted, without a diagnostic, until a later revision removes them.
For every model-serving role that has an effective shape, the selected integration MUST either preserve every explicitly declared component or reject the combination as unsupported before launch. It MUST fill every omitted component needed to return one complete positive-integer effective shape for that role.
The sole effective parallelism authority MUST be each role’s complete effective shape, or, for a role without one, its device count together with its Engines’ observed data-parallel rank counts. The integration protocol and resolved execution MUST NOT return or reconstruct a server-wide effective parallelism shape.
Every derived gateway frontend process role MUST have an empty effective shape and zero device count. A P/D frontend has one gateway process shape for both logical components.
Prefill and decode MAY resolve to different tensor, pipeline, data, context, expert, and dense tensor parallelism, different whole-replica device count, different replica counts, and different rank decompositions.
Dry-run output and records MUST preserve common and role declarations, the source of each declared component and of devices, and each role’s complete effective shape, or, for a role without one, its device count and its Engines’ observed data-parallel rank counts.
Placement MUST consume each role’s whole-replica device count accepted under this clause.
The configuration and protocol MUST NOT expose framework-native flags. The component names are shared across integrations, but a component’s effect on the device count is the integration’s reading.
These required per-role shapes MUST use the current adapter protocol version under RFC-0006:C-INTEGRATIONS. An adapter advertising an earlier protocol version MUST NOT emit or consume them.
Rationale: Common values remove repetition but do not create a second effective authority. Per-role effective shapes are necessary because prefill and decode can use different card counts and decomposition, while the integration remains responsible for each framework’s algebra and unsupported combinations. Placement needs only a device count, while parallel layouts keep becoming finer and per-component and one vocabulary has already been read differently by different integrations; devices separates the placement fact from that vocabulary without breaking existing declarations, and leaving inherited components untouched keeps resolution a plain layered merge.
Since: v0.1.0
[RFC-0003:C-EXPERIMENTAL-PROXY] Gateway And P/D Router Contract (Normative)
Section titled “[RFC-0003:C-EXPERIMENTAL-PROXY] Gateway And P/D Router Contract (Normative) ”Status: superseded Superseded by: RFC-0003:C-GATEWAY-PD-ROUTER
Superseded by RFC-0003:C-GATEWAY-PD-ROUTER.
Since: v0.1.0
[RFC-0003:C-SGLANG-PREFILL-DECODE] SGLang Prefill Decode Routing (Normative)
Section titled “[RFC-0003:C-SGLANG-PREFILL-DECODE] SGLang Prefill Decode Routing (Normative) ”The SGLang integration MUST support Mooncake and NIXL as selectable KV-transfer mechanisms for the prefill/decode topology defined by RFC-0003:C-SERVE-TOPOLOGY.
The SGLang integration MUST support independent positive replica counts for the prefill and decode roles.
For both KV-transfer mechanisms, the SGLang integration MUST declare one control-plane-allocated bootstrap endpoint for every prefill replica.
The SGLang prefill/decode topology MUST support the independently selected backend pairs (builtin, builtin) and (sglang-router, sglang-router). The first tuple member is gateway_backend and the second is pd_router_backend. Each pair MUST support both Mooncake and NIXL.
The SGLang integration MUST reject every unlisted Gateway/P/D Router pair during resolution before launching a process. The two backend axes MUST remain separately represented even when the selected identifiers are equal.
Selecting (builtin, builtin) MUST return separate Gateway and P/D Router results with render_source = control_plane, implementation kind sglang, and one fused process binding. It MUST use the InferLab-rendered proxy and MUST NOT require a SGLang Model Gateway package in the serving workspace.
Selecting (sglang-router, sglang-router) MUST return separate Gateway and P/D Router results with render_source = integration and one fused SGLang Model Gateway process from the selected stack or image realization under RFC-0003:C-GATEWAY-PD-ROUTER.
The two supported pairs MUST NOT alias one another in resolution, rendered commands, or record evidence. Gateway and P/D Router identities MUST remain separately attributable within either pair.
Both P/D Router backends MUST target only the rank-zero serving entry point of each selected prefill and decode replica.
Initial server readiness with either pair MUST wait for every configured prefill replica, every configured decode replica, the P/D Router readiness requirements, and the Gateway public endpoint to become ready.
An integration-rendered SGLang Model Gateway MUST use its worker-aware readiness endpoint, which is unsuccessful until at least one prefill and one decode worker are healthy, rather than its process-liveness endpoint.
Before reporting an SGLang Model Gateway topology ready, InferLab MUST verify through the P/D Router worker registry that every configured rank-zero prefill and decode target is registered with its expected role, every prefill target has its expected bootstrap port, and every target is healthy.
For each public completions or chat-completions request, the InferLab-rendered P/D Router MUST select one configured prefill replica and one configured decode replica.
The InferLab-rendered Gateway MUST accept the scalar-prompt completion request shape consumed by InferLab and MUST reject a prompt-array completion request before P/D dispatch.
For a chat-completions request, the InferLab-rendered Gateway and P/D Router MUST preserve the original structured message request and selected /v1/chat/completions path for both roles. They MUST NOT translate messages into a scalar prompt, apply completion-only prompt validation, or dispatch either phase to the completions path.
For either route, the fused implementation MUST preserve every request-body member it does not own. The P/D Router MUST inject only the selected prefill bootstrap host, bootstrap port, and one request-unique bootstrap room into the request delivered to both selected replicas.
The P/D Router MUST initiate both role requests without waiting for a prefill generation response and MUST drain the prefill response.
When the request sets stream to false or omits it, Gateway MUST return the complete decode response.
When the request sets stream to true, Gateway MUST relay the decode response incrementally while preserving its status and content type.
A prefill or decode failure observed before public response headers are sent MUST produce an HTTP status outside 200 through 299.
A prefill or decode failure observed after response streaming begins MUST terminate the public stream as failed rather than complete it successfully.
Both Gateway backends MUST expose the exact completions and chat-completions paths governed by RFC-0003:C-OPENAI-WORKLOAD-ROUTES.
Both Gateway backends MUST expose POST /flush_cache as the prefix-cache reset action through the public endpoint.
The corresponding P/D Router backend MUST attempt the public prefix-cache reset action on every configured prefill and decode replica.
Gateway MUST return HTTP 200 only when every target succeeds and HTTP 206 when one or more targets fail.
InferLab MUST record HTTP 206 from the public prefix-cache reset action as reset failure evidence.
Both supported pairs MUST otherwise use the common backend independence, realization, lifecycle, readiness, rollback, cleanup, and evidence requirements in RFC-0003:C-GATEWAY-PD-ROUTER.
Rationale: Gateway and P/D Router are separate experimental facts even though both currently supported SGLang providers fuse them into one process. The built-in implementation and SGLang Model Gateway remain complementary operator choices: a dependency-light experiment baseline and the upstream provider. SGLang uses the prefill bootstrap rendezvous for both Mooncake and NIXL, so transport selection remains independent. Registry-backed readiness proves the P/D Router target set, while Gateway remains authoritative for public routes, streaming, and cache-control responses.
Since: v0.5.0
[RFC-0003:C-TENSORRT-LLM-PREFILL-DECODE] TensorRT-LLM Prefill Decode Routing (Normative)
Section titled “[RFC-0003:C-TENSORRT-LLM-PREFILL-DECODE] TensorRT-LLM Prefill Decode Routing (Normative) ”The TensorRT-LLM integration MUST support NIXL as the selectable KV-transfer mechanism for the prefill/decode topology defined by RFC-0003:C-SERVE-TOPOLOGY.
The TensorRT-LLM integration MUST reject every other KV-transfer mechanism for this topology before launching a process.
The TensorRT-LLM integration MUST support independent positive replica counts for the prefill and decode roles.
A TensorRT-LLM prefill/decode topology MUST support the independently selected backend pairs (builtin, builtin) and (trtllm-disaggregated, trtllm-disaggregated). The first tuple member is gateway_backend and the second is pd_router_backend.
The integration MUST reject every unlisted Gateway/P/D Router pair during resolution before launching a process. The two backend axes MUST remain separately represented even when the selected identifiers are equal.
The two supported pairs MUST NOT alias one another in resolution, rendered commands, or record evidence. Gateway and P/D Router identities MUST remain separately attributable within either pair.
Both pairs MUST declare one logical Gateway-to-P/D-Router handoff, P/D Router links to the prefill and decode roles, and one NIXL KV-transfer link from prefill to decode.
The P/D Router backends MUST NOT invent a control-plane bootstrap or side-channel endpoint when the selected TensorRT-LLM NIXL protocol exchanges its connection metadata in the context response.
Both P/D Router backends MUST target only the rank-zero serving entry point of each selected prefill and decode replica.
Each prefill and decode Engine worker MUST receive an integration-rendered launch file under RFC-0006:C-LAUNCH-FILES.
The effective Engine worker configuration MUST select the NIXL cache-transceiver backend and the Python cache-transceiver runtime.
The effective prefill-worker configuration MUST disable the overlap scheduler.
The effective KV-cache configuration of every prefill and decode worker MUST disable block reuse while the Gateway advertises no prefix-cache reset action.
An open TensorRT-LLM settings escape hatch MUST NOT override the selected cache-transceiver backend, cache-transceiver runtime, required prefill overlap setting, or required block-reuse setting.
Selecting (builtin, builtin) MUST return separate Gateway and P/D Router results with render_source = control_plane, implementation kind trtllm, and one fused process binding. It MUST use the InferLab-rendered TensorRT-LLM frontend and MUST NOT require the native TensorRT-LLM disaggregated server.
For each public completions or chat-completions request, the InferLab-rendered P/D Router MUST select one configured prefill replica and one configured decode replica using independent static round-robin selection.
The InferLab-rendered Gateway MUST accept the scalar-string-prompt completion request shape consumed by InferLab.
The Gateway MUST reject prompt arrays and a requested completion count other than one before dispatching a completions request to P/D Router.
The Gateway MUST accept the structured message request shape for chat completions and MUST reject a requested chat-completion count other than one. It MUST preserve messages, chat-template arguments, reasoning controls, tools, and every other request-body member it does not own. It MUST NOT translate a chat request into a completions request.
The P/D Router MUST assign each dispatched request a request-unique integer greater than or equal to 1 << 42 and MUST use that same disagg_request_id for its context and generation phases.
The P/D Router MUST issue the selected prefill replica a non-streaming context-only request with context-first scheduling semantics on the same selected completions or chat-completions path as the public request.
The P/D Router MUST reject a context response that has no first choice.
When the first context-response choice has finish reason length or not_finished, the P/D Router MUST reject the response unless it contains prompt-token identifiers for the selected request family, top-level usage, and a first-choice disaggregated-parameter object whose ctx_request_id is non-null, whose disagg_request_id equals the identifier assigned to the context request, and whose first_gen_tokens is present.
For a completions request, those prompt-token identifiers MUST be a prompt-token array. For a chat-completions request, they MUST be a prompt_token_ids_b64 string when supplied by the context response, otherwise a prompt-token array.
When the first context-response choice has a finish reason other than length or not_finished, the P/D Router MUST NOT dispatch a decode request.
For a non-streaming public request that completes during the context phase, the P/D Router MUST remove internal disaggregated parameters and Gateway MUST return the complete context response.
For a streaming public request that completes during the context phase, the P/D Router MUST remove internal disaggregated parameters and Gateway MUST convert the complete route-specific context response into one or more valid completions or chat-completions server-sent events that preserve its generated content, choice identity, and finish reason, then emit the terminal data: [DONE] event without dispatching a decode request.
When a completions generation handoff is required, the P/D Router MUST replace the original prompt with the context response’s prompt-token array.
When a chat-completions generation handoff is required, the P/D Router MUST preserve the original structured messages and set the generation request’s prompt_token_ids_b64 from the context response when present, otherwise its prompt_token_ids array. It MUST dispatch the generation phase to the selected decode replica at /v1/chat/completions.
The generation request MUST copy every member of the returned disaggregated-parameter object that the P/D Router does not explicitly replace.
The generation request MUST replace only request_type, schedule_style, and ctx_usage in that object, setting them respectively to generation-only, context-first, and the context response’s top-level usage.
When the public request is non-streaming, Gateway MUST return the complete decode response.
When the public request is streaming, the P/D Router MUST keep the context request non-streaming and Gateway MUST relay the decode response incrementally while preserving its status and content type.
A context or decode failure observed before public response headers are sent MUST produce an HTTP status outside 200 through 299.
A decode failure observed after public response streaming begins MUST terminate the public stream as failed rather than complete it successfully.
Selecting (trtllm-disaggregated, trtllm-disaggregated) MUST return separate Gateway and P/D Router results with render_source = integration and one fused component binding. The selected stack or image realization MUST render one native TensorRT-LLM disaggregated server under RFC-0003:C-GATEWAY-PD-ROUTER.
The native P/D Router MUST receive a static context-first round-robin configuration containing every configured rank-zero prefill and decode target through an integration-rendered launch file under RFC-0006:C-LAUNCH-FILES.
Initial server readiness with either pair MUST wait for every configured prefill replica, every configured decode replica, the P/D Router readiness requirements, and the Gateway public endpoint to become ready.
Both Gateway backends MUST expose the exact completions and chat-completions paths governed by RFC-0003:C-OPENAI-WORKLOAD-ROUTES.
Neither Gateway backend MUST advertise a prefix-cache reset action for this workflow while the selected TensorRT-LLM serving interface exposes no action that can reset every configured prefill and decode worker.
Frontend evidence for (builtin, builtin) MUST record separate Gateway and P/D Router identities and versions for the InferLab-rendered frontend.
Frontend evidence for (trtllm-disaggregated, trtllm-disaggregated) MUST record separate backend identities, their shared fused process binding, and the TensorRT-LLM integration adapter version.
Both supported pairs MUST otherwise use the common backend independence, realization, lifecycle, readiness, rollback, cleanup, and evidence requirements in RFC-0003:C-GATEWAY-PD-ROUTER.
Rationale: TensorRT-LLM context-first disaggregation is a two-stage P/D Router protocol for both standard OpenAI request forms, while Gateway retains the public route, validation, response, and streaming contract. The context response carries prompt tokens, the first generated token, usage, and opaque connection metadata needed to construct the generation-only request. The built-in fused frontend is therefore a bounded protocol adapter rather than the concurrent bootstrap implementation used by SGLang. The native disaggregated server remains separately selectable on both backend axes so operators can compare an InferLab-rendered baseline with upstream orchestration while records retain the Gateway/P/D Router semantic split. NIXL carries its dynamic endpoint in response metadata, so requiring a bootstrap or side-channel port would record and allocate a topology that does not exist. Disabling block reuse provides repeatable measurement isolation while no composed reset action exists.
Since: v0.6.0
[RFC-0003:C-TOKENSPEED-PREFILL-DECODE] TokenSpeed Prefill Decode Routing (Normative)
Section titled “[RFC-0003:C-TOKENSPEED-PREFILL-DECODE] TokenSpeed Prefill Decode Routing (Normative) ”The TokenSpeed integration MUST support the prefill/decode topology defined by RFC-0003:C-SERVE-TOPOLOGY.
The topology’s Gateway backend, P/D Router backend, and KV-transfer mechanism MUST remain independently selected and independently recorded.
The TokenSpeed integration MUST accept Mooncake as the KV-transfer mechanism for this topology and MUST reject every other mechanism during resolution before launching a process.
The TokenSpeed integration MUST support independent positive replica counts for the prefill and decode roles.
A TokenSpeed prefill/decode topology MUST accept the independently selected backend pair (smg, smg). The first tuple member is gateway_backend and the second is pd_router_backend. A selection naming the former backend tokenspeed-smg MUST be rejected during resolution with an instruction to select smg.
The integration MUST reject every other Gateway/P/D Router pair during resolution before launching a process in this RFC version. The two selected backend facts MUST remain distinct even though both currently name SMG.
The topology MUST declare one logical Gateway-to-P/D-Router handoff, P/D Router links to the prefill and decode roles, one Mooncake KV-transfer link from prefill to decode, and one control-plane-allocated bootstrap endpoint for every prefill replica.
P/D routing, KV transfer, and bootstrap targeting MUST use only the rank-zero serving entry point of each selected prefill and decode replica.
Each rendered prefill Engine worker MUST expose a TokenSpeed gRPC serving endpoint and MUST receive the prefill disaggregation role, the Mooncake transfer backend, and its allocated bootstrap endpoint.
Each rendered decode Engine worker MUST expose a TokenSpeed gRPC serving endpoint and MUST receive the decode disaggregation role and the Mooncake transfer backend.
An open TokenSpeed settings escape hatch MUST NOT override the resolved disaggregation role, KV-transfer backend, model, served endpoint, process topology, parallelism, or allocated bootstrap endpoint.
Selecting (smg, smg) MUST return separate Gateway and P/D Router results with render_source = integration and one fused component binding. The selected stack or image realization MUST render one native SMG process under RFC-0003:C-GATEWAY-PD-ROUTER.
The SMG P/D Router result MUST enable prefill/decode routing and receive a static target list containing every configured rank-zero prefill gRPC endpoint paired with its bootstrap endpoint and every configured rank-zero decode gRPC endpoint.
The SMG P/D Router MUST use independent round-robin prefill and decode policies. Its P/D Router settings MAY declare extra_args under the SMG rule of RFC-0003:C-SPECIALIZED-ENGINE.
Initial server readiness MUST wait for every configured prefill replica, every configured decode replica, the SMG P/D Router readiness requirements, and the SMG Gateway public endpoint to become ready.
Before reporting the topology ready, InferLab MUST verify through the SMG P/D Router worker registry that every configured rank-zero gRPC target is registered with its expected role, every prefill target has its expected bootstrap port, and every target is healthy.
The SMG Gateway MUST expose the exact completions and chat-completions paths governed by RFC-0003:C-OPENAI-WORKLOAD-ROUTES and MUST preserve the selected request form through its P/D Router handoff. The SMG mapping MUST preserve structured chat messages and every supported unowned request member in its worker request; a member the mapping cannot represent MUST produce an explicit non-2xx public rejection rather than be discarded.
The SMG Gateway MUST expose POST /flush_cache as its public prefix-cache reset action.
The SMG P/D Router MUST attempt that action on every configured prefill and decode worker. Gateway MUST report success only when every target succeeds.
Frontend evidence MUST preserve separate smg Gateway and P/D Router identities, their shared fused process binding, each rendering-source fact, the independent round-robin P/D policy, every rank-zero gRPC target and bootstrap endpoint, the public endpoint, and the version of the installed SMG Gateway distribution.
KV-transfer evidence and rendered Engine commands MUST preserve Mooncake independently from both frontend backend facts.
The topology MUST otherwise use the common backend independence, realization, lifecycle, readiness, rollback, cleanup, and evidence requirements in RFC-0003:C-GATEWAY-PD-ROUTER.
Rationale: SMG currently realizes Gateway and P/D Router in one process, while Mooncake owns KV movement between selected Engines. Treating those as three independent selected and recorded facts keeps the experiment model truthful and permits a later Gateway or P/D Router replacement without renaming SMG into an ownership category. TokenSpeed’s packaged gRPC servicer is the stable Engine boundary consumed by SMG, and registry-backed readiness proves the complete P/D target set while InferLab retains placement and lifecycle authority.
Since: v0.7.0
[RFC-0003:C-OPENAI-WORKLOAD-ROUTES] OpenAI Workload Routes (Normative)
Section titled “[RFC-0003:C-OPENAI-WORKLOAD-ROUTES] OpenAI Workload Routes (Normative) ”Every supported public server endpoint MUST expose one completions route and one chat-completions route.
The completions route MUST be /v1/completions. The chat-completions route MUST be /v1/chat/completions.
The integration endpoint contract MUST declare both named paths. Resolution, dry-run output, server records, and measurement client inputs MUST preserve those declared path values and MUST NOT reconstruct either path from the other.
A direct single topology MUST expose both routes through the rank-zero endpoint of its sole serve replica. Every topology with a Gateway MUST expose both routes only through that Gateway. A P/D Router MUST NOT be a public endpoint authority.
Every Gateway backend that InferLab reports as supporting a topology MUST preserve both paths through its selected Engine or P/D Router handoff without translating a chat request into a completions request or a completions request into a chat request.
Before internal handoff, Gateway MUST preserve each request-body member it does not own or reject the public request with a non-2xx response that retains the backend diagnostic. An integration MUST NOT remove a member to compensate for the selected Gateway or P/D Router. A Gateway backend that silently discards an unowned member MUST NOT be reported as supporting or qualified for the affected request workflow.
A P/D Router MUST preserve the selected route and every unowned request member across prefill and decode dispatch unless its backend-specific handoff contract explicitly replaces that member. It MUST report a handoff incompatibility to Gateway as a failure and MUST NOT silently translate or discard the request.
An InferLab-rendered Gateway MUST preserve every public request member except a member that its specified handoff protocol explicitly replaces. It MUST return the selected route’s framework response shape and streaming protocol.
An InferLab-rendered Gateway MUST NOT interpret, normalize, remove, or rename sampling, logprob, reasoning, chat-template, tool, or other framework request members that its handoff protocol does not own.
A framework and topology MUST be reported as qualified for a route only when a real public-route execution record demonstrates that exact route, model, integration revision, Gateway backend, P/D Router backend when present, frontend component binding when present, KV-transfer selection when present, and topology. Source availability or qualification of the other route MUST NOT establish that claim.
Rationale: Generative measurements need the model server’s real chat encoding and reasoning controls, while likelihood scoring and the lowest-level smoke retain completions semantics. Two closed, named routes cover those concrete workflows without introducing an arbitrary endpoint registry. Gateway owns the public protocol boundary; P/D Router owns only the internal phase handoff. Preserving both the route and unowned body members keeps that separation observable across direct and fused serving.
Since: v0.9.1
[RFC-0003:C-GATEWAY-PD-ROUTER] Gateway And P/D Router Contract (Normative)
Section titled “[RFC-0003:C-GATEWAY-PD-ROUTER] Gateway And P/D Router Contract (Normative) ”Gateway and P/D Router MUST be separate logical components. Gateway owns the public workload boundary when present. P/D Router owns prefill/decode target selection and phase orchestration. Neither component owns placement, concrete endpoint allocation, process lifecycle, cleanup, or authoritative records; InferLab retains those authorities.
Integration planning MUST return separate gateway and pd_router results whenever the selected topology requires them. It MUST NOT return a routing-ownership union or the protocol-v6 direct, inferlab_builtin, or integration_native ownership variants.
Each returned frontend component result MUST preserve its independently selected backend identity, implementation identity and observable version, complete effective settings, readiness requirements, controls, links, endpoint requirements, render inputs, and one render_source of control_plane or integration. render_source means only which bounded implementation renders the accepted concrete allocation into a launch command. It MUST NOT be interpreted as process-lifecycle, placement, endpoint, cleanup, or evidence ownership.
A direct single server MUST omit both frontend component results and expose the sole serve replica’s rank-zero endpoint.
A routed single server MUST return exactly one Gateway result matching the effective gateway_backend, MUST omit P/D Router, and MUST bind Gateway to one derived zero-device gateway process with component binding [gateway]. Gateway MUST target only the serve replica’s rank-zero endpoint.
InferLab MUST NOT report a Gateway backend as supporting single solely from this generic contract. Routed-single support MUST have a backend-specific contract for Engine target compatibility, rendering source, readiness, both public-route semantics, and server-control behavior before it can be reported as supported or qualified.
A prefill_decode server MUST return one Gateway result matching the effective gateway_backend and one P/D Router result matching the effective pd_router_backend. Those backend selections and the KV-transfer mechanism MUST remain independently selected, validated, and recorded.
The selected Gateway/P/D Router backend pair MUST declare fused compatibility. An unsupported integration, topology, backend identity, backend pair, or KV-transfer mechanism MUST fail resolution before any process launches. Topology alone MUST NOT silently select either backend.
Both P/D frontend component results MUST declare the same rendering source and compatible co-rendering requirements. The control plane MUST derive one zero-device gateway allocation with component binding [gateway, pd_router]. The selected renderer MUST return one process and one command for that allocation. Gateway and P/D Router MUST share the process identity and lifecycle evidence while retaining separate logical backend identities, settings, policies, readiness obligations, controls, and versions.
For render_source = control_plane, the component result MUST name one closed InferLab implementation kind supported by the selected backend and topology. For the currently specified built-in P/D implementations, the closed kinds are vllm_mooncake, vllm_nixl, sglang, and trtllm. InferLab MUST render the corresponding frontend allocation and MUST NOT supply it to render_serve.
For render_source = integration, the selected integration MUST render the allocation from the chosen stack or image realization. The control plane MUST place it through the same placement authority as every other process and MUST supply the concrete allocation to render_serve. An integration-rendered frontend process MAY run remotely.
One stack realization, image, or integration package MAY supply Engine, Gateway, and P/D Router implementations. Co-delivery MUST NOT merge their logical identities. A fused process MAY implement two independently named backends only when the selected stack or image realization declares and validates that exact pair.
An integration MUST NOT advertise a Gateway backend unless it can validate settings, target compatibility, public endpoint semantics, readiness, public controls, implementation identity, and rendering requirements. It MUST NOT advertise a P/D Router backend unless it can validate settings, declare complete prefill and decode target semantics and policy, declare bootstrap or registry requirements, identify its internal handoff contract, and co-render it with a compatible Gateway backend.
Every Gateway and P/D Router target set MUST use only rank-zero serving entry points. Component links MUST NOT imply replica pairing; the planned policy and complete target sets are authoritative.
InferLab MUST allocate every concrete endpoint and retain lifecycle authority for every frontend process regardless of rendering source. Commands, environment, logs, readiness, durable handles, failures, rollback, cleanup, and component-to-process bindings MUST have the same evidence guarantees as model-serving processes.
A Gateway-backed topology MUST NOT report readiness until every model-serving backend required by the selected policy, the P/D Router when present, and the public Gateway endpoint are ready. Eval and Bench MUST attach only to the public workload endpoint.
When a public completions or chat-completions request sets stream to true, Gateway MUST make each complete route-native streaming event available to the public client without waiting for a later event or upstream body completion. This requirement applies independently of HTTP transport chunk boundaries. Gateway MUST preserve the successful upstream response status and content type.
An upstream failure observed before public response headers are sent MUST produce an HTTP status outside 200 through 299. An upstream failure observed after public streaming begins MUST terminate the public stream as failed rather than complete it successfully.
InferLab-provided baseline frontends MAY use static routing over configured ready backends. They MUST NOT be required to provide automatic restart, degraded-backend recovery, backend eviction, minimum-ready capacity, or production availability guarantees.
The vLLM integration MUST support the independently recorded backend pairs (builtin, builtin) and (vllm-router, vllm-router) over both Mooncake and NIXL, and (dynamo, dynamo) over NIXL under RFC-0003:C-DYNAMO. The first tuple member is gateway_backend and the second is pd_router_backend. The builtin pair MUST use render_source = control_plane and the corresponding closed vLLM implementation kind. The vllm-router and dynamo pairs MUST use render_source = integration. The integration MUST reject every unlisted cross-pair during resolution before launching a process in this RFC version.
The builtin vLLM P/D Router MUST attempt the prefix-cache reset on every configured prefill and decode replica and MUST count a replica’s reset as succeeded only when the replica returns a 2xx response whose body reports success true. It MUST report the aggregate in a success member: HTTP 200 with success true when every replica succeeded; HTTP 200 with success false when every failed replica declined with a 2xx response reporting success false; and HTTP 206 when any replica failed otherwise. The prefix-cache reset action of that pair’s public endpoint and of a direct vLLM Engine endpoint MUST declare the success predicate /success equal to boolean true.
Frontend evidence MUST preserve the effective Gateway backend, P/D Router backend with explicit absence when it does not apply, both implementation identities and versions, each rendering source, effective Gateway public behavior, effective P/D routing policy, KV-transfer selection, every target, internal link and endpoint, component-to-process binding, public endpoint, commands, and integration version when integration rendering applies. A P/D frontend process MUST be recorded once and referenced by both logical component records; evidence MUST NOT duplicate its allocation or lifecycle outcome.
Rationale: Protocol v7 names the real semantic boundaries instead of calling a rendering decision ownership. Independent Gateway and P/D Router selections make public API behavior and phase orchestration separately replaceable experimental facts. The fused P/D binding matches every currently demonstrated provider without promoting arbitrary deployment graphs. InferLab remains the only runtime authority, and KV movement remains a third independent mechanism. Incremental public delivery is part of Gateway correctness because client-observed TTFT and token spacing cease to describe generation when a frontend buffers the stream to completion.
Since: v0.11.0
[RFC-0003:C-SERVE-CONTEXT-PARALLELISM] Serve Context Parallelism Semantics (Normative)
Section titled “[RFC-0003:C-SERVE-CONTEXT-PARALLELISM] Serve Context Parallelism Semantics (Normative) ”A declared attention.context_parallel_size expresses one framework-neutral context-parallel degree for attention under RFC-0003:C-SERVE-PARALLELISM.
For a role that declares devices under RFC-0003:C-SERVE-PARALLELISM, the role’s whole-replica device count is devices in every topology, and the integration MUST lower the declared degree only to mechanisms whose resulting device count equals devices, or reject the role before launch.
For a role without devices in a single topology, context parallelism MUST NOT increase any role’s whole-replica device count: the integration MUST lower the declared degree only to context-parallel mechanisms that subdivide the role’s existing device set. For a role without devices in a prefill_decode topology, the prefill role MAY lower the declared degree to a prefill context-parallel mechanism that multiplies that role’s whole-replica device count, and the integration MUST then derive that role’s device count from the framework’s algebra; the decode role’s context parallelism MUST NOT increase its whole-replica device count.
When one declared degree activates several concrete framework mechanisms, the integration MUST render every activated mechanism into the role’s command so the record preserves them, and MUST NOT activate a device-count-multiplying mechanism outside the prefill role of a prefill_decode topology unless the role declares devices.
Applicability constraints that depend on model internals such as attention architecture or head counts, on hardware generation, or on the framework’s runtime kernel or backend selection are owned by the framework. The integration MUST NOT inspect model artifacts, hardware, or framework installations to pre-validate them; its own validation of a declared parallelism shape is limited to arithmetic over the declared components. A framework’s rejection of an unsupported combination surfaces as a recorded launch failure.
Rationale: One declared degree keeps the neutral surface minimal while engines split context parallelism into incompatible prefill and decode mechanisms. The device-count rule keeps single requests resource-predictable and confines device-multiplying prefill context parallelism to the prefill role it accelerates. Model-internal applicability moves with framework releases and fork behavior, so the framework, not the integration, owns it; the integration’s records preserve what was declared and rendered, and captured launch output preserves the framework’s own verdict.
Since: v0.14.0
[RFC-0003:C-SERVE-SYNTHETIC-ACCEPTANCE] Serve Synthetic Acceptance Overlay (Normative)
Section titled “[RFC-0003:C-SERVE-SYNTHETIC-ACCEPTANCE] Serve Synthetic Acceptance Overlay (Normative) ”A server definition and each of its server cases MAY declare a synthetic_acceptance member selecting controlled synthetic speculative-acceptance behavior. A case-level declaration MUST replace the server-level declaration wholesale. Invocation-scoped overrides MUST NOT patch the synthetic acceptance declaration. InferLab MUST NOT model the speculative decoding method, draft count, or method-specific framework parameters; those remain operator-owned framework settings under RFC-0003:C-RESOLUTION. The overlay MUST NOT consume or model the draft model; a draft weight artifact independent of the server’s model MAY be declared under RFC-0003:C-SERVE-AUXILIARY-MODELS, and that declaration alone governs the artifact’s resolution, carriage, and evidence.
synthetic_acceptance MUST take exactly one of two forms, and each form MUST carry exactly its named members; unknown members MUST be rejected. The explicit form MUST carry a finite acceptance_length number greater than or equal to one. The curve form MUST carry a workspace-relative path subject to the same validation as other workspace-referenced source paths, an expected_sha256 of exactly 64 lowercase hexadecimal characters digesting the complete curve file bytes, a non-empty model_key string, and an optional thinking_mode string that MUST be non-empty when present.
A curve file MUST be a YAML document mapping model keys to acceptance-length entries in exactly one of two shapes: a flat list of single-entry mappings from a positive integer draft length to a mean acceptance length, or a thinking-mode mapping from thinking-mode names to mappings of positive integer draft lengths to mean acceptance lengths. Every entry value MUST be a finite number. Any other document shape or a non-numeric or non-finite entry value MUST fail workspace validation. A declared thinking_mode against a model entry in the flat-list shape MUST fail workspace validation, because no mode applies to that entry.
The control plane MUST digest-verify the curve file before it is used. A missing or unreadable curve file or a digest mismatch MUST fail workspace resolution and MUST name the file and the mismatch. The control plane MUST NOT resolve the acceptance length from the curve against a declared draft count: the draft count’s sole authority is the operator’s framework speculative configuration.
The declared synthetic-acceptance configuration MUST be preserved in canonical merged workspace output. The declared configuration, the effective acceptance length returned by the accepted plan, and, for the curve form, the curve path, digest, model key, effective thinking mode, and the draft count determined from the operator configuration MUST be preserved in dry-run, execution, and serve record evidence. Serve-record projection remains governed by RFC-0005:C-EVIDENCE.
The declaration MUST cross to the selected integration through plan_serve and render_serve as governed by RFC-0006:C-INTEGRATIONS: the explicit acceptance length for the explicit form, or the digest-verified curve text with its digest, model key, and effective thinking mode for the curve form. An omitted thinking_mode MUST resolve to thinking_on when the model entry uses the thinking-mode shape. The integration MUST overlay the effective acceptance length onto the operator’s declared framework speculative configuration and MUST fail planning with a typed error when no speculative configuration exists to overlay. For the curve form, the integration MUST resolve the effective acceptance length from the supplied curve by model key, by thinking mode, and by the draft count determined from the operator configuration, and MUST fail planning with a typed error when that draft count is not determinable or when the curve lacks the keyed entry, naming the missing element. The integration MUST return the effective acceptance length in the plan response, and the control plane MUST reject a plan response that omits it or carries a non-finite value or a value below one. For the curve form the plan response MUST additionally carry the determined draft count, and for the explicit form it MUST NOT carry one. When prefill and decode roles would resolve different effective values, the integration MUST fail planning with a typed error, because the plan response carries one effective value. Framework-specific command and environment rendering remains integration-owned, and an integration that cannot apply the overlay MUST fail planning with a typed error.
An Eval measurement of any kind governed by RFC-0004:C-MEASUREMENTS, including the OpenAI-compatible smoke measurement, that is bound to a server whose resolved configuration carries synthetic acceptance MUST fail measurement planning, identifying that synthetic acceptance bypasses real draft-model verification.
Rationale: InferenceX AgentX submissions prescribe speculative decoding with golden-curve synthetic acceptance so that results measure the inference system rather than a tuned speculative head. The overlay keeps InferLab’s contract independent of the moving speculative-method vocabulary — the upstream benchmark itself separates a framework-agnostic acceptance-length driver from per-framework injectors that rewrite the operator’s existing recipe. The curve file stays the single authority for the acceptance value while the operator configuration stays the single authority for the draft count, so the keyed lookup happens where both inputs meet: inside the integration. The Eval exclusion mirrors the upstream EVAL_ONLY split, because synthetic acceptance bypasses real verification and invalidates generation-quality measurement.
Since: v0.16.0
[RFC-0003:C-SERVE-AUXILIARY-MODELS] Serve Auxiliary Model Weights (Normative)
Section titled “[RFC-0003:C-SERVE-AUXILIARY-MODELS] Serve Auxiliary Model Weights (Normative) ”A server definition MAY declare an auxiliary_models member: a mapping from auxiliary weight kind to a workspace model identifier. A kind outside the governed vocabulary MUST be rejected. The auxiliary weight kind vocabulary is governed by this RFC; this version defines exactly one kind, draft-model, a weight artifact independent of the server’s model that the operator’s framework speculative decoding configuration consumes as its draft. A new kind MUST NOT enter the vocabulary without a demonstrated workflow that requires it. The referenced model identifier MUST differ from the server’s own model. auxiliary_models MUST NOT appear on a server case, and invocation-scoped overrides MUST NOT patch it.
Each referenced model identifier MUST select a declared workspace model. The auxiliary model’s weights MUST resolve through the standing machine-local model weight bindings under RFC-0002:C-LOCAL-PLACEMENT exactly as the server’s model does, including the shared fallback locator, per-machine locators, and their provenance. Resolution MUST fail when an auxiliary model has no resolvable locator for a machine the server is placed on, naming the kind, the model identifier, and the machine.
The declaration MUST cross to the selected integration as governed by RFC-0006:C-INTEGRATIONS: planning receives the logical auxiliary identities with their kinds, and each model-rank rendering allocation receives the machine-resolved locator for each declared kind. The integration MUST splice each declared kind’s resolved locator into its framework-specific configuration spelling and MUST fail planning with a typed error when it does not consume a declared kind. Planning MUST fail with a typed error naming the kind when the operator’s framework configuration already spells a weight artifact that a declared kind covers, so the declaration remains the single authority for the artifact. The integration MUST NOT introduce an auxiliary weight artifact the declaration does not carry.
Canonical merged workspace output MUST preserve the declared auxiliary_models mapping. Dry-run, execution, and serve-record evidence MUST preserve the declared mapping with each auxiliary model’s identifier and served name, and each effective auxiliary weight locator with its kind and locator provenance; serve-record projection remains governed by RFC-0005:C-EVIDENCE. Auxiliary weights are model weights under RFC-0007:C-IMAGE-BUILD: portable outputs such as runtime images, image build contexts, and OCI artifacts MUST NOT contain auxiliary weight locators.
Rationale: Speculative decoding deployments such as DSpark load draft weights independent of the main model, and operators could previously express them only as literal paths inside framework escape-hatch arguments — invisible to per-machine resolution, validation, and evidence. Naming the artifact role rather than the speculative method keeps the contract independent of the moving method vocabulary, while the governed kind vocabulary keeps every consumed artifact auditable. The mechanism reuses the models registry and the model-weight bindings so no parallel resolution machinery exists.
Since: v0.17.0
[RFC-0003:C-SPECIALIZED-ENGINE] Specialized Engine Routed-Single Contract (Normative)
Section titled “[RFC-0003:C-SPECIALIZED-ENGINE] Specialized Engine Routed-Single Contract (Normative) ”The Specialized Engine integration MUST support the routed single topology of RFC-0003:C-SERVE-TOPOLOGY with gateway_backend = "smg", no P/D Router, no KV-transfer mechanism, and one or more serve replicas, and the prefill_decode topology with gateway_backend = "smg", pd_router_backend = "smg", a declared KV-transfer mechanism, and one or more prefill and decode replicas. The Specialized Engine integration MUST accept any KV-transfer mechanism name, because the Engine owns its transport; the Engine rejects a name it does not implement under the rule below. Every other selection MUST be rejected during resolution before launching a process. A prefill_decode server that requests profiling MUST be rejected during planning.
Each replica placed on one machine MUST be one rank process that owns all of its allocated devices and serves every one of its data-parallel ranks behind its one listen address. A role of this integration MUST declare its device count as devices under RFC-0003:C-SERVE-PARALLELISM and MUST NOT declare a parallelism component; a declared component, or a role without devices, MUST be rejected during planning with a diagnostic directing the layout to Engine arguments. The integration returns the role’s device count and no effective parallelism shape. The layout within a replica is the Engine’s own: the Engine receives its allocated devices through device visibility, takes any further layout from its Engine arguments, and MUST exit before it reports SERVING, naming the option, when that layout and its device count disagree or it does not support the layout. The Engine MUST report SERVING only once every one of its ranks can admit requests.
Replicas spanning machines. A serve replica placed through an explicit multi-rank binding under RFC-0002:C-LOCAL-PLACEMENT MUST run as one Engine process per rank binding, in rank order; rank zero is host 0 and the replica’s only endpoint, and the host index is the rank index. The integration MUST declare a rendezvous port that the control plane allocates only to rank zero of a multi-rank replica. Every rank MUST receive --nnodes with the replica’s rank count, --node-rank with its rank index, 0 for rank zero, and --dist-init-addr with rank zero’s rendezvous address, identical on every rank; rank zero binds that address and every other rank connects to it. Rank zero MUST also receive every contract argument of a one-machine Engine. Every other rank MUST receive --model with its own machine’s model locator, --served-model-name, and, with a KV tier, --kv-tier-backend and --kv-tier-store with its own machine’s tier location, and no --listen. Every rank receives its own devices through device visibility and the role’s Engine arguments. Each host’s Engine MUST exit before it reports SERVING, naming the option, when its layout’s world differs from its visible device count times the host count. Host 0 answers for the whole replica: it MUST report SERVING only after every member has joined and can admit requests; its data_parallel_size MUST be the replica’s count across all hosts; its FlushCache MUST clear every host’s prefix-cache and tier entries; and with a tier its reported tier state, access, and publication bound MUST cover every host. The Gateway, registry readiness, observed facts, and serve-state probe address rank zero only; the other ranks are ready while their processes run. A replica whose ranks hold different device counts, a prefill or decode replica placed across machines, and a server that requests profiling with a replica placed across machines MUST be rejected before launch, naming the cause.
The stack MAY declare the stack setting engine_command under RFC-0002:C-WORKSPACE-AUTHORITY, the leading argv the Engine implementation defines and every role of the server shares: a non-empty list whose first entry is a bare executable name, resolved on the selected stack environment’s PATH, and whose remaining entries are fixed arguments of that implementation. It defaults to ["inferlab-token-engine", "smg-worker"], and the returned effective stack settings MUST carry the value in force. An empty engine_command, a first entry containing a path separator, or an entry naming a contract-owned option MUST be rejected during planning.
Engine-provided KV store. Beside mooncake and filesystem, the integration MUST accept as a KV store backend under RFC-0003:C-SERVE-KV-STORE any name of lowercase ASCII letters, digits, -, and _, as a store the Engine implementation provides. Such a backend MUST take no store settings, and the stack MUST declare the stack setting kv_store_command: a non-empty list whose first entry is a bare executable name resolved on the selected stack environment’s PATH and whose remaining entries are fixed arguments of the store, validated as engine_command is. A server declaring such a backend on a stack without kv_store_command, or with store settings, MUST be rejected during planning. The KV store plan MUST name the declared backend and record it as the implementation with the integration’s version, declare one additional logical port metrics, declare readiness as GET /health on that port answering 200, and declare the store access observed. The store process MUST be rendered as kv_store_command, then --listen with its allocated endpoint and --metrics-port with its allocated metrics port, and no other option. The store MUST answer 200 on /health only once it accepts Engine connections, MUST exit when signalled to terminate, and MUST NOT require devices. Its lifecycle, placement, readiness, and failure are those of any store InferLab runs under RFC-0003:C-SERVE-KV-STORE.
Engine metrics port. The stack MAY declare the boolean stack setting engine_metrics, absent by default. When it is true, every Engine replica MUST declare one additional logical port metrics; rank zero MUST receive --metrics-port with its allocated metrics port as a contract argument, and an extra_args entry naming --metrics-port MUST be rejected during planning. The Engine serves its metrics on that port on the host of its listen address. When it is absent or false, no Engine receives --metrics-port and no metrics port is declared.
The integration MUST render each Engine as engine_command, then the role’s Engine arguments in their composed order under RFC-0003:C-RESOLUTION, then exactly these contract arguments, then any Engine arguments after a bare -- sentinel, and no other option: --listen with its allocated listen address, --model with its model locator, and --served-model-name with the served model name; in a prefill_decode topology, --disaggregation-mode with prefill or decode for the Engine’s role, --disaggregation-transfer-backend with the declared KV-transfer mechanism, and for a prefill Engine --disaggregation-bootstrap-port with the bootstrap port allocated to it; and, when the server declares a KV tier under RFC-0003:C-SERVE-KV-STORE, --kv-tier-backend with the declared backend and --kv-tier-store with the tier location: the store’s host:port for mooncake or an Engine-provided backend, or the directory bound for the Engine’s machine for filesystem. A single Engine receives no prefill/decode argument. The Engine’s allocated devices are its device visibility.
Engine arguments are the role’s extra_args and extra_env settings. They carry every Engine option that is not a contract argument — layout, memory, scheduling and admission limits, context length, speculative decoding, tier tuning, and transfer timeouts — and the effective contents MUST be recorded with the rendered command. An extra_args entry naming a contract argument MUST be rejected during planning, before or after the sentinel. An extra_env entry naming the device-visibility variable CUDA_VISIBLE_DEVICES MUST be rejected during planning. A typed setting this integration formerly accepted — default_max_output_tokens, max_num_batched_tokens, gpu_memory_utilization_percent, workspace_reserve_mib, prefix_cache_gpu_entries, prefix_cache_host_memory_percent, prefix_cache_ranks, kv_tier_capacity_bytes, kv_tier_snapshot_interval, and kv_tier_timeout_seconds, and the stack setting kv_tier_publication_seconds — MUST be rejected during planning with a diagnostic directing it to the Engine’s own option or report.
The Engine MUST reject a mode, transfer mechanism, or tier backend it does not support by exiting before it reports SERVING, naming the option. A prefill Engine MUST accept bootstrap connections on its bootstrap port at its listen host before it reports SERVING; a decode Engine learns its counterpart from each request. A decode Engine MUST admit a request that finishes on its prefill counterpart’s first token, because a primed start conditions with a single output token under RFC-0004:C-BENCH-CACHE-STATE.
Observed serving facts. Each Engine MUST report its number of data-parallel ranks as the positive integer data_parallel_size in its GetServerInfo server_args, the value the Gateway exposes for every worker in its registry. Initial readiness MUST read it for every Engine from the Gateway’s worker registry and MUST NOT complete while an Engine’s value is absent or not a positive integer; a server whose readiness budget expires so fails naming the Engine and the field. An Engine given a KV tier MAY report SERVING before it has reached the tier and MUST keep serving when it loses the tier. Its GetServerInfo scheduler_info MUST then carry kv_tier_state — "available" once the Engine has completed a tier operation, "unavailable" before that and whenever the tier is lost; kv_tier_store — the location it uses; kv_tier_access — read_write or write under RFC-0003:C-SERVE-KV-STORE; and kv_tier_publication_ms — a non-negative integer bound, zero when the Engine publishes on completion. The integration MUST declare a serve-state probe under RFC-0003:C-SERVE-KV-STORE that reads them from the rank-zero process of every Engine of every model-serving role and declares the store access observed. An Engine is engaged only when its state is available and its location equals the tier location rendered for it. Every Engine of one role MUST report the same access; readiness MUST NOT complete otherwise, and its failure names the disagreeing Engines. Its FlushCache MUST clear every tier entry it could read. The Gateway’s POST /flush_cache reset succeeds only when its reported flushed-worker count equals the number of Engines.
Worker protocol. At its listen address the Engine MUST serve the gRPC service tokenspeed.grpc.scheduler.TokenSpeedScheduler as defined by the SMG Gateway distribution installed in the selected stack, and the standard gRPC health checking service, which MUST report SERVING only once the Engine can admit requests. The Engine MUST implement Generate as a server stream, HealthCheck, Abort, GetModelInfo, GetServerInfo, GetLoads, and FlushCache. An Engine that is to be profiled MUST also implement StartProfile and StopProfile, which the Gateway’s profiling-window actions invoke. Any other RPC of the service MAY answer UNIMPLEMENTED; a Gateway capability that depends on such an RPC is then unavailable for that Engine and MUST NOT be claimed by the integration.
Generate MUST require tokenized input. Prompt text in the request is transport metadata and MUST NOT reach the model core, which returns token IDs without tokenizing or detokenizing. A sampling or request field the Engine does not support MUST be rejected with INVALID_ARGUMENT rather than ignored or reinterpreted. The completion MUST carry the prompt, completion, and cached token counts, reporting zero cached tokens explicitly. The Engine MAY emit stream chunks only after generation completes, so the contract makes no low-latency streaming claim.
Protocol identity and compatibility. The SMG Gateway distribution defines the worker protocol; InferLab defines no protocol version of its own and performs no version negotiation. Protobuf field additions are compatible in both directions. An Engine built against another release of the published protocol MUST remain wire-compatible with the Gateway release the stack installs; the Gateway’s worker registration is the compatibility check.
Frontend. The integration MUST plan one SMG Gateway through the SMG lowering it shares with every integration that runs SMG, targeting every replica’s gRPC listen address, with retries and the circuit breaker disabled. Its routing policy is the typed Gateway setting policy, which MUST accept the SMG Gateway’s least_load, power_of_two, round_robin, random, cache_aware, prefix_hash, and consistent_hashing policies and defaults to least_load. The Gateway MUST expose the exact completions and chat-completions paths governed by RFC-0003:C-OPENAI-WORKLOAD-ROUTES, Prometheus metrics at /metrics on its allocated metrics port, and POST /flush_cache as its prefix-cache reset action. The SMG Gateway offers no prefix-cache conditioning fan-out, so the integration MUST NOT declare one; a primed start conditions through the ordinary serving flow under RFC-0004:C-BENCH-CACHE-STATE. Initial readiness MUST wait for every Engine process and for the Gateway’s worker registry to list every replica’s listen address as healthy, because the Gateway’s /readiness path reports ready once any one worker is healthy. In a prefill_decode topology the integration MUST plan one fused SMG Gateway and P/D Router through the same shared lowering, targeting every prefill replica’s listen address with its bootstrap port and every decode replica’s listen address. Its routing policies are the typed P/D Router settings prefill_policy and decode_policy, each accepting the policies listed above and defaulting to round_robin. The typed Gateway settings and P/D Router settings MAY also declare extra_args, SMG options of the operator’s own that the shared SMG lowering appends after every option it renders and that the effective settings record verbatim; an entry naming an SMG option InferLab renders — its listen address and port, Prometheus port, worker-startup timeout, model and tokenizer paths, worker URLs, prefill/decode wiring, routing policies, and retry and circuit-breaker switches — or a bare -- MUST be rejected during planning, naming it. Because SMG runs power_of_two only over two or more workers, a power_of_two Gateway or P/D Router policy over fewer than two workers in its pool MUST be rejected during planning. Initial readiness MUST then wait for the worker registry to list every prefill replica as a healthy prefill worker with its allocated bootstrap port and every decode replica as a healthy decode worker.
Evidence. Frontend evidence MUST record the backend smg, the implementation smg, the installed SMG Gateway distribution version, the effective routing policies and failure-handling settings, and the worker protocol as the service name tokenspeed.grpc.scheduler.TokenSpeedScheduler. Engine evidence MUST record the rendered command, including engine_command and the role’s Engine arguments as declared and as rendered, the workspace source identity that identifies the concrete Engine implementation, each Engine’s observed data_parallel_size, and, with a tier, each Engine’s observed tier facts.
Rationale: A Specialized Engine is a hardware-by-model token worker that reuses SMG for every public HTTP concern. Naming the published service, rather than an InferLab-invented label, keeps one authority for the wire contract — the SMG distribution — while the required RPC subset states exactly what InferLab’s frontend relies on, so a minimal Engine is conformant without implementing tokenizer download, KV events, or engine-side profiling. The contract fixes the arguments an Engine accepts, not its executable name, so each implementation may ship under its own name; the default keeps stacks written for the former fixed command working. InferLab renders only what it allocates — addresses, model, roles, devices, and the tier location. Layout, limits, and tier tuning belong to the Engine, and the facts the control plane derives measurements from are observed from the Engine, so a record never states a value InferLab inferred on the Engine’s behalf.
Since: v0.21.0
[RFC-0003:C-DYNAMO] Dynamo Gateway And Prefill Decode Serving (Normative)
Section titled “[RFC-0003:C-DYNAMO] Dynamo Gateway And Prefill Decode Serving (Normative) ”The vLLM integration MUST support Dynamo as Gateway backend dynamo for a routed single topology and as the independently selected backend pair (dynamo, dynamo) for a prefill_decode topology. The first tuple member is gateway_backend and the second is pd_router_backend.
A Dynamo prefill_decode topology MUST accept NIXL as its KV-transfer mechanism and independent positive prefill and decode replica counts. The integration MUST reject, during resolution and before launching a process, every other KV-transfer mechanism, every pair that names dynamo for only one member, and every model-serving replica whose rank_count is not one. Different roles and replicas MAY be placed on different machines.
Both frontend results MUST declare render_source = integration. The selected stack or image realization MUST render one Dynamo frontend process realizing the Gateway, and for prefill_decode also the P/D Router, under RFC-0003:C-GATEWAY-PD-ROUTER. The Dynamo Gateway result MUST declare a discovery requirement under RFC-0006:C-INTEGRATIONS, and the selected realization MUST render the discovery process from its own etcd installation on the discovery process’s allocated listeners and data directory.
Each rendered model-serving process MUST run the framework’s Dynamo worker and MUST receive the discovery endpoint, the namespace shared by every Dynamo process of the server, its allocated listener on the TCP request plane, and its allocated HTTP control listener with health tracking enabled for its generation endpoint. A prefill or decode process MUST also receive its disaggregation role, the NIXL connector with the producer role for prefill or the consumer role for decode, and its allocated KV side-channel listener. The Dynamo frontend MUST receive the discovery endpoint, the server’s namespace, the TCP request plane, its allocated public endpoint, its allocated response-stream port, and its effective routing mode. Every listener InferLab allocates to a Dynamo or discovery process MUST be rendered on the owning machine’s network address, except the frontend’s response-stream listener, which MUST receive only its allocated port. Because Dynamo and etcd listen only on IP addresses, the integration MUST reject during resolution a Dynamo or discovery process whose allocated host is not an IP address. The worker’s event publishers, whose ephemeral ports on all interfaces Dynamo provides no setting to choose, are outside InferLab allocation.
The integration MUST reject, before any process launches, an allocated control-listener port that the installed Dynamo cannot represent, naming the port and the limit.
For a Dynamo worker, the replica entry endpoint for HTTP server-control and capture-window actions MUST be its HTTP control listener. Gateway targets and registry membership MUST use its request-plane listener, and the Dynamo P/D Router result MUST declare the tcp target endpoint scheme.
An open settings escape hatch MUST NOT set, including after a verbatim-passthrough sentinel, Dynamo’s discovery, namespace, endpoint identity, request-plane, event-plane, or listener options, the environment form of any of them, a variable that renames a Dynamo process or adds a listener to it, or Dynamo’s reasoning parser. While Dynamo is selected, the effective reasoning-parser setting MUST render as Dynamo’s reasoning parser, because the Dynamo frontend, not the framework, answers requests and post-processes responses. For the same reason, framework tool-call parsing, automatic tool choice, and prompt-token details, and the framework’s own API-server listener and reasoning-parser options, MUST be rejected during resolution, whether set as typed settings or in an escape hatch, including after a verbatim-passthrough sentinel; the Dynamo frontend reports cache-read usage itself.
Initial server readiness MUST require all of: the discovery service’s health; every model-serving process’s control health with its generation endpoint reported ready; and Gateway readiness, which MUST be reported only when the frontend instance registry lists every model-serving process under its role with the request-plane address InferLab allocated to it, and the frontend model list contains the served model name. A registered instance whose address InferLab did not allocate MUST NOT satisfy any target.
The Dynamo Gateway MUST satisfy RFC-0003:C-OPENAI-WORKLOAD-ROUTES and MUST declare backend cache-read usage.
The Dynamo topology MUST provide its prefix-cache reset capability per target under RFC-0006:C-INTEGRATIONS: every model-serving role MUST declare a reset action, and InferLab MUST invoke it under RFC-0004:C-BENCH-CACHE-STATE. Each reset action and each capture-window action MUST declare the success predicate that distinguishes a reported failure from success within a 2xx response.
Frontend evidence MUST preserve separate dynamo Gateway and, when present, P/D Router identities, their shared process binding, each rendering-source fact, the effective routing mode, the installed Dynamo distribution version, every model-serving target with its request-plane address, the server namespace, the discovery process, the public endpoint, and NIXL KV-transfer evidence independently of both frontend facts.
The topology MUST otherwise use the common backend independence, realization, lifecycle, readiness, rollback, cleanup, and evidence requirements in RFC-0003:C-GATEWAY-PD-ROUTER.
Rationale: Dynamo realizes Gateway and prefill/decode orchestration in one frontend process and replaces each Engine with a worker that registers into its discovery service, so the Gateway choice determines the Engine launch while InferLab keeps placement, allocation, lifecycle, and records. A dedicated discovery process per server keeps registrations, namespaces, and cleanup inside one recorded run on one or several machines. Registry membership against allocated addresses proves that every planned replica serves the public endpoint before measurement starts, and no unallocated instance is counted.
Since: v0.22.0
[RFC-0003:C-SERVE-KV-STORE] Server External KV Store (Normative)
Section titled “[RFC-0003:C-SERVE-KV-STORE] Server External KV Store (Normative) ”A server definition and each of its server cases MAY declare kv_store, one external KV store shared by the server’s model-serving processes, with a backend identity and a settings table typed by that backend. A case MAY patch the settings but MUST NOT add or remove the store or change its backend; an invocation-scoped override MAY patch only the settings. The backends are mooncake, a store process InferLab runs, and filesystem, one directory on a file system that the server’s machines mount. The filesystem backend takes no settings. An integration MAY also accept Engine-provided backends, each a store process InferLab runs from a command the selected stack declares; such an integration names the backends it accepts and the store’s command-line and readiness convention in its own clause.
The store applies to a single or prefill_decode topology. The selected integration MUST either consume the declared backend in its model-serving role and return a KV store plan — backend and implementation identity, observed implementation version, complete effective settings, logical ports beyond the endpoint, readiness, and render inputs — or reject the declaration during planning. A filesystem KV store plan declares the store externally operated, with empty effective settings and no ports; its lowering lives in the shared package that owns external KV store backends. Store memory capacity is an Engine setting of each integration, not a store setting, because consuming frameworks size their contributions differently.
For a store InferLab runs — mooncake or an Engine-provided backend — the control plane MUST derive exactly one kv_store process. The process MUST have process-only identity with no replica or rank coordinates, zero devices, and no model locator. It MUST be placed on the Gateway’s machine when the server has a Gateway and otherwise on the machine of the first serve replica’s rank zero. The control plane MUST allocate its endpoint and declared ports, MUST supply every model-serving allocation with the store’s resolved endpoint, which MUST be reachable from every machine of the server, MUST start the store before any model-serving process and wait for its readiness, and MUST stop it after every model-serving and frontend process. The store process exiting while the server runs MUST fail the server; it MUST NOT be restarted.
For a filesystem store, each machine that runs a model-serving process MUST bind its tier directory as kv_tier_directory in its local machine binding under RFC-0002:C-LOCAL-PLACEMENT, as an absolute path, and resolution MUST fail naming any such machine that binds none. Every model-serving allocation MUST receive its own machine’s bound directory unchanged as its tier location, and the control plane runs no store process. The directory is not an exclusive allocation: servers on other machines, in other workspaces, or of other teams may share it, and keeping it undisturbed during a measurement is the responsibility of the developer running it. The control plane MUST NOT create, inspect, maintain, or remove the directory or its contents; the file system, its mounts, and any reclaimer of the directory are infrastructure outside InferLab.
A prefix-cache reset that an integration declares for a server with a store MUST also clear every store entry the server’s model-serving processes could read, so a cold start reads nothing from the store; an integration that cannot clear them MUST declare no prefix-cache reset for that server. A Bench that separates reuse by namespace under RFC-0004:C-BENCH-CACHE-STATE issues no reset, so it runs on such a server too; only the starts that reset depend on this rule. The KV store plan MUST either declare the store access of each model-serving role whose processes use the store, or declare that access observed, in which case the integration’s serve-state probe reports each process’s access and publication bound at readiness, the plan declares neither, and every model-serving role is treated as using the store. The access is read_write when each process reads matching prefixes from the store and writes the prefixes it computes to the store so that they are readable no later than the plan’s publication bound after the request completes, or write when it only writes. A role a declaring plan omits makes no claim, and its processes are treated as not using the store. The plan MAY declare that publication bound, a fact of how the consuming implementation publishes to the store; absent, it is zero. The server’s prefix cache is shared when every cache-owning role under RFC-0004:C-BENCH-CACHE-STATE — serve, or prefill on a prefill_decode server — is declared or observed read_write; such a server has exactly one cache-owning target. An observed access is authoritative for the server instance that reported it, and the server’s publication bound is then the largest bound any of its processes reported. Store access describes how roles use one store owned by the server; transfer between prefill and decode remains the declared KV-transfer mechanism.
When an Engine keeps serving without its store, readiness alone does not show that the store engaged. The integration MAY then declare a serve-state probe under RFC-0006:C-INTEGRATIONS that reports, for each rank-zero process of every model-serving role that uses the store, declared or observed, whether its store is engaged. With such a probe, the server MUST NOT become ready until every reported process is engaged, within the server’s readiness budget; every Bench case MUST record each process’s reported state at the start and at the end of its profiling window, and a process not engaged at either boundary MUST fail the case as having lost its store.
The integration MUST render the store process and each model-serving process’s connection to the store. The store command and each framework’s connection options belong to the integration and the shared package that owns the backend, not to the control plane.
Dry-run output and records MUST preserve the declared store per declaration layer, the returned KV store plan, the store process’s placement, endpoint, rendered command, and readiness evidence, and its cleanup outcome; for a filesystem store, each Engine’s bound directory and the access and publication bound each Engine reported.
Rationale: A shared tier changes what a measurement means, so its process, address, and settings belong in the same resolved authority and record as the Engines that use it rather than in an unrecorded side process. Placing it beside the Gateway, or beside the only replica of a direct server, keeps its address reachable without a new placement surface. Capacity stays with each Engine because one key name means per-rank memory in one framework and per-instance memory in another. A shared file-system tier is storage infrastructure like model weights: binding it per machine records where it is without making InferLab its operator, and since sharing is its purpose, guarding it against other users is left to whoever runs the measurement.
Since: v0.30.0