Decision record · ADR
ADR-0071: Limit the Specialized Engine contract to allocation facts and observe engine-owned serving facts
Status: accepted | Date: 2026-11-05
References: RFC-0003, RFC-0004, ADR-0068, ADR-0069, RFC-0005, RFC-0006, RFC-0009
Context
Section titled “Context”Two Specialized Engines now run through the one specialized-engine integration: a C++ engine and DeepInfer on SM120, a Python engine maintained by another team. Both implement the SMG worker protocol, readiness, FlushCache, and cached-token reporting as the contract requires, and that part of the contract holds. Their command lines diverge from it.
The contract renders attention tensor, data, context, and expert parallelism derived from the declared devices. The C++ engine selects its real layout with an engine-owned recipe name passed through extra_args (--parallelism dep4) and only checks that tensor times data parallelism equals the recipe’s world, so its records show attention tensor parallelism 4 and data parallelism 1 for a data-parallel layout, and the control plane, which reads data parallelism to count cache-owning targets, sees one rank. DeepInfer requires tensor and context parallelism of one and expert parallelism equal to data parallelism, which the derived arithmetic can express only because DeepInfer’s layout happens to match it; each new layout either Engine adds needs a matching change to InferLab’s derivation.
The contract’s typed tuning options fare worse. Neither Engine accepts the prefix-cache sizing group or the workspace reserve; DeepInfer accepts some of them only with a disabled value. max_num_batched_tokens means a per-step budget in one Engine and a per-rank prefill budget in the other, so one recorded value names two different limits. Context length, sequence limits, and speculative decoding, which both Engines configure, are absent and travel through extra_args.
The KV store moved under the contract. DeepInfer replaced its Mooncake tier with a tier on a shared file system: --kv-tier-store is now an existing directory that workers never create, a separate reclaimer process per store directory bounds its size, and the worker capacity option is gone. DeepInfer, which owns that tier, states that the file system, its mount on each node, and the reclaimer are storage infrastructure provisioned outside InferLab like the model weights directory, that the directory is a deployment input every sharing worker receives unchanged, and that InferLab should not start, stop, configure, or check any of them. The integration still renders a Mooncake address and capacity, so the declared tier servers no longer start. The KV store plan also requires the integration to declare each role’s store access and the publication bound, facts of how a particular Engine uses the tier that a generic integration cannot know.
The operator confirmed the governing principle for this redesign: InferLab needs to know only the information it requires to allocate resources.
Decision
Section titled “Decision”We will limit the Specialized Engine contract to the facts InferLab allocates and observe the serving facts each Engine owns.
The contract renders only allocations: the Engine command, listen address, model locator, served model name, the allocated devices through device visibility, the prefill/decode role with its transfer mechanism and bootstrap port, and the KV tier backend and location. Layout within the Engine, memory, scheduling and admission limits, context length, speculative decoding, and tier tuning are Engine arguments, recorded as declared and as rendered. An Engine argument or environment entry that names a contract argument or an allocation-carrying variable is rejected, including after the override sentinel, so no declaration can redirect an allocation. The integration no longer renders tensor, context, or expert parallelism or any typed tuning option, and a server of this integration declares devices, not parallelism components.
Facts the control plane uses but does not allocate are observed rather than declared. The data-parallel rank count is read at readiness from the Gateway’s worker registry, which takes it from each Engine’s GetServerInfo server_args, the field both Engines already report and SMG already reads; an Engine that does not report a positive count fails readiness. With a tier, each Engine reports its tier state, location, access, and publication bound through the integration’s serve-state probe. The control plane records these facts and derives the cache-owning target set and the primed-start publication wait from them; a primed start whose observed target set the frontend cannot condition fails after readiness and before preparation.
A KV tier is a resource the server is allocated, not necessarily a process InferLab runs. A filesystem backend joins mooncake: each machine binds its tier directory in its local bindings, as it binds model weights, and InferLab passes that directory unchanged to each Engine on the machine. The directory is not an exclusive allocation: a tier is infrastructure that servers on other machines, in other workspaces, or of other teams may share by design, and InferLab cannot observe its holders. Whether anything else uses or resets a tier during a measurement is controlled by the developer running it, as with any other shared infrastructure; InferLab neither guards nor detects it. InferLab does not create, inspect, maintain, or remove the directory or its contents; the file system, its mounts, and any reclaimer are infrastructure. A cold start relies on the existing rule that the prefix-cache reset invalidates every tier entry, and readiness on every Engine reporting the tier available at its bound directory. The Engine receives the backend with the location and rejects one it does not support before reporting SERVING.
Consequences
Section titled “Consequences”Positive: records state the layout the Engine reports rather than one InferLab inferred, so the C++ engine’s layout misreport and its single-rank cache-target count disappear. No contract value means different things in different Engines. An Engine adopts a new tuning option, layout, or speculative method without an InferLab change, and the integration keeps no Engine-specific branch. A shared file-system tier needs no InferLab process or lifecycle, and the backend is available to other integrations.
Negative: both Engines must change their worker entrypoints: the C++ engine derives its world from devices and its recipe instead of a rendered tensor parallelism, DeepInfer takes its layout from the visible devices and Engine arguments and accepts the tier backend, and a tier Engine reports its access and publication bound. Tuning mistakes are no longer caught at planning; they surface when the Engine rejects its arguments, before loading the model when it parses them first. The change breaks every existing specialized-engine workspace in one release, with no compatibility period. Planning can no longer reject a primed start that the observed target set disallows; that rejection moves to after readiness, so the operator pays one model load to learn it. Concurrent servers may share one tier, and a reset or write by another user of the tier during a window is neither prevented nor detected; keeping a tier quiet during a measurement is the developer’s responsibility. A tier outlives its runs: an uncontrolled start may read prefixes an earlier run left, which the existing uncontrolled semantics already disclaim, and keeping the tier’s size bounded is the infrastructure’s job, not InferLab’s.
Neutral: the parallelism vocabulary deprecation of ADR-0068 completes early for this integration; the other integrations keep their readings. The gRPC worker protocol, frontend, readiness, cache control, and profiling clauses are unchanged.
Alternatives Considered
Section titled “Alternatives Considered”Limit the contract to allocation facts, pass everything else as Engine arguments, and observe the serving facts the control plane uses from each Engine (accepted)
Section titled “Limit the contract to allocation facts, pass everything else as Engine arguments, and observe the serving facts the control plane uses from each Engine (accepted)”- Pros: One authority per fact: InferLab for what it allocates, the Engine for what it does with it, Records carry observed layout and tier semantics, never inferred ones, New Engine options, layouts, and tier mechanisms need no InferLab change
- Cons: Both Engines change their worker entrypoints and report new GetServerInfo fields, Primed-start target checks move from planning to readiness, costing one model load on a rejected layout
Keep the current contract and let each Engine conform to its rendered layout and typed options (rejected)
Section titled “Keep the current contract and let each Engine conform to its rendered layout and typed options (rejected)”- Pros: No InferLab change, Typed options are validated before launch
- Cons: The contract’s derived layout cannot describe recipe-based layouts, so records stay wrong for recipe-based Engines, Typed options already mean different limits in the two Engines, and neither accepts the prefix-cache group, InferLab would have to define and chase every tuning concept both Engines grow
- Rejected because: The contract owns facts it does not allocate and cannot describe correctly, so conformance would bend the Engines to an InferLab vocabulary without making the evidence true
Grow the contract with per-Engine variants or a capability declaration that selects option spellings and features (rejected)
Section titled “Grow the contract with per-Engine variants or a capability declaration that selects option spellings and features (rejected)”- Pros: One integration package, Each Engine could keep its own spellings
- Cons: Engine-specific knowledge enters the integration, which the integration ownership rule forbids, Every Engine change becomes an InferLab release, A capability handshake duplicates what an Engine already reports by rejecting an option before loading the model
- Rejected because: It moves Engine-owned facts into InferLab and adds a negotiation layer that launch-time rejection already provides
Give each Engine its own integration package, as vLLM and SGLang have, over a shared SMG and lifecycle library (rejected)
Section titled “Give each Engine its own integration package, as vLLM and SGLang have, over a shared SMG and lifecycle library (rejected)”- Pros: Each package can type its Engine’s options, Engines evolve independently
- Cons: Each Engine team maintains and releases Python lowering in InferLab’s monorepo, Both Engines already ship a worker entrypoint written for this contract, so a package would add a second translation layer, Facts InferLab does not allocate would still be declared rather than observed
- Rejected because: The translation already lives in each Engine’s worker entrypoint; a package per Engine adds release coupling without removing any fact InferLab should not own
Keep a declared data-parallel rank count beside devices and verify it against the Engine’s report at readiness; keep access and publication bound as stack settings the Engine implementation declares (rejected)
Section titled “Keep a declared data-parallel rank count beside devices and verify it against the Engine’s report at readiness; keep access and publication bound as stack settings the Engine implementation declares (rejected)”- Pros: Planning still rejects a primed start the declared layout cannot condition, without a model load, Smallest change to the existing declaration surface
- Cons: The workspace restates a fact the Engine owns, so two authorities exist and the declaration must change whenever the Engine’s layout recipe does, Stack-declared access and publication bound describe one Engine release; an Engine that changes how it uses the tier silently invalidates them until the workspace is edited
- Rejected because: It keeps InferLab declaring facts it does not allocate; the operator chose observation and accepted the one model load a disallowed primed layout costs
InferLab creates one tier directory per server instance under a locally bound root, verifies it from every machine, runs the Engine’s reclaimer as a managed maintenance process, and retires the directory at cleanup (rejected)
Section titled “InferLab creates one tier directory per server instance under a locally bound root, verifies it from every machine, runs the Engine’s reclaimer as a managed maintenance process, and retires the directory at cleanup (rejected)”- Pros: Each server starts from an empty tier and is isolated from concurrent servers by construction, A run’s tier lifecycle appears in its record
- Cons: The tier’s owner operates the file system and its reclaimer as infrastructure and asked InferLab not to start, configure, or check them, A reclaimer bounds every file under its store directory, so an infrastructure reclaimer on the root and InferLab-created subdirectories would manage the same files, Directory creation, cross-machine verification, a maintenance process, and retirement add lifecycle that resource allocation does not need; cold start already follows from the reset invalidating the tier
- Rejected because: Accepted first and reopened on 2026-11-05 after the tier’s owner stated that the file system and reclaimer are infrastructure outside InferLab; an exclusive directory allocation gives the isolation it offered without the lifecycle
Hold the tier directory as an exclusive allocation under the occupancy rules for devices and ports (rejected)
Section titled “Hold the tier directory as an exclusive allocation under the occupancy rules for devices and ports (rejected)”- Pros: Catches two servers of one workspace bound to the same directory before launch
- Cons: Occupancy is observed per machine and per workspace, so servers on other nodes, in other workspaces, or of other teams sharing the directory pass unseen, A tier has no machine-observable occupancy signal, and a lock inside the shared file system would have InferLab touch infrastructure it must not, Blocks sharing that the tier’s owner intends
- Rejected because: It promises an exclusivity InferLab cannot observe and blocks sharing the tier’s owner intends; keeping a shared tier quiet during a measurement is the developer’s to control
Have each tier Engine report a tier generation and fail a case whose generation changes during its profiling window without its own reset (rejected)
Section titled “Have each tier Engine report a tier generation and fail a case whose generation changes during its profiling window without its own reset (rejected)”- Pros: Detects an external reset from any machine, workspace, or team
- Cons: Adds an Engine reporting obligation and a failure class for a condition the developer running the measurement already controls
- Rejected because: The operator assigned control of external use of a shared tier to the developer running the measurement (2026-11-05)