Decision record · ADR
ADR-0022: Separate Gateway, Prefill/Decode Router, And Engine Semantics
Status: accepted | Date: 2026-07-18
References: ADR-0014, RFC-0003:C-SERVE-TOPOLOGY, RFC-0003:C-GATEWAY-PD-ROUTER, RFC-0003:C-OPENAI-WORKLOAD-ROUTES, RFC-0006:C-INTEGRATIONS, RFC-0002:C-LOCAL-PLACEMENT, RFC-0005:C-EVIDENCE
Context
Section titled “Context”InferLab is adding a concrete specialized-engine workflow in which a single engine may sit behind a reusable public frontend, while the first prefill/decode serving implementation may reuse TokenSpeed SMG. Existing integrations also use SGLang Model Gateway, the TensorRT-LLM disaggregated server, and InferLab’s bounded proxy implementations.
Problem Statement
Section titled “Problem Statement”Protocol version 6 overloads one canonical router concept with two different responsibilities:
- serving the public OpenAI-compatible endpoint and preserving public request, response, streaming, readiness, and server-control semantics; and
- selecting prefill and decode workers and coordinating the framework-specific handoff between them.
It also returns direct, inferlab_builtin, or integration_native as a routing-ownership result even though InferLab always retains placement, lifecycle, cleanup, and record authority. The result therefore describes a rendering source, not ownership, and obscures the boundary that a specialized engine should consume.
Because there are few downstream protocol consumers, the operator authorized a protocol-v7 break in favor of the clearest semantics and hierarchy. The concrete workflows that must remain expressible are direct single serving, single serving behind a public frontend, and prefill/decode serving whose public frontend and P/D routing behavior may be delivered by one process.
Constraints
Section titled “Constraints”Gateway, P/D Router, and Engine must remain separately attributable even when one provider, integration package, stack, image, or process delivers more than one component. InferLab remains the sole authority for composition, placement, concrete endpoints, process lifecycle, cleanup, and authoritative records.
The design must not introduce a universal component graph or require independently selected packages and environments before a demonstrated operator workflow needs them. Existing backend identities remain meaningful provider choices, and KV transfer remains independent from request routing.
Decision
Section titled “Decision”We will separate Gateway, Prefill/Decode Router, and Engine semantics and keep Gateway and P/D Router as independently selected backend facts, while using one fused frontend process for the currently demonstrated P/D workflows because public API behavior, phase orchestration, and model execution have different reasons to change.
- Gateway: when present, owns the public OpenAI-compatible endpoint, request validation and preservation at that boundary, public response and streaming behavior, client-visible readiness, and public server-control actions.
- P/D Router: when present, owns prefill/decode target selection, routing policy, worker health and registration requirements, bootstrap targeting, and phase orchestration. It does not own the public API contract.
- Engine: owns model execution. A direct single Engine also owns the public endpoint; otherwise its serving endpoints are consumed by Gateway or P/D Router.
- Control-plane composition: InferLab owns backend-pair validation, logical composition, placement, concrete endpoints, launch order, lifecycle, cleanup, and authoritative evidence.
Independent backend identity is a semantic and experimental boundary, not a requirement for independent packages, versions, stacks, images, or processes. One selected stack or image realization and one aggregate planning/rendering operation may deliver all three implementations, but must keep their effective facts separately attributable.
Protocol version 7 will replace the ambiguous protocol-v6 routing-ownership representation so rendering source is not presented as lifecycle ownership. The exact configuration, wire, topology, endpoint, placement, and evidence obligations belong to the referenced RFC clauses.
All currently demonstrated P/D providers fuse Gateway and P/D Router. A separately placed P/D Router remains deferred until a concrete operator workflow proves that process shape; this decision does not introduce a general component graph.
Consequences
Section titled “Consequences”Positive
Section titled “Positive”- Specialized Engines can use SMG or another frontend implementation without importing Gateway or P/D routing behavior into model execution.
- Gateway and P/D Router backends remain selected, identified, validated, and recorded separately even when one implementation process supplies both.
- Public API compatibility can evolve independently from P/D target selection and handoff policy.
- Current fused products remain practical without mislabeling rendering source as runtime ownership.
- Records can compare Gateway, P/D Router, KV transfer, and Engine behavior as separate experimental facts.
Negative
Section titled “Negative”- Protocol-v6 integrations and workspace configuration require a breaking migration. The mitigation is one protocol-v7 cut with no compatibility alias or partial v6 interpretation.
- A fused process has one failure and lifecycle boundary for two logical components. The mitigation is one process record referenced by two separately attributable component records.
- Independently selected backends create pair-compatibility validation. The mitigation is a closed support table rejected before launch rather than an open backend Cartesian product.
- Semantic separation can erode inside a co-delivered implementation. The mitigation is separate planning results, settings, readiness, controls, and evidence at the component boundary.
Neutral
Section titled “Neutral”- InferLab remains the lifecycle and evidence authority regardless of rendering source.
- KV-transfer selection remains independent from P/D request routing.
- Package or version independence is optional, not required by this decision.
- Split Gateway/P/D Router deployment remains deferred until a demonstrated workflow requires it.
- This decision does not create arbitrary role declarations, a universal component DAG, or a production-supervisor promise.
Alternatives Considered
Section titled “Alternatives Considered”Keep the protocol-v6 combined router concept and routing-ownership variants (rejected)
Section titled “Keep the protocol-v6 combined router concept and routing-ownership variants (rejected)”- Pros: Avoids an immediate wire and configuration migration
- Cons: Continues to conflate public API behavior with P/D orchestration, Keeps rendering source mislabeled as ownership
- Rejected because: The operator chose a protocol-v7 break specifically to establish the clearest semantic boundaries.
Rename the combined router role to Gateway while leaving P/D routing implicit (rejected)
Section titled “Rename the combined router role to Gateway while leaving P/D routing implicit (rejected)”- Pros: Improves the public-front process name
- Cons: Still hides P/D worker selection and handoff inside Gateway, Cannot attribute public API and P/D routing behavior independently
- Rejected because: A noun change alone does not establish the required ownership boundary.
Separate Gateway and P/D Router semantics and require separate processes immediately (rejected)
Section titled “Separate Gateway and P/D Router semantics and require separate processes immediately (rejected)”- Pros: Creates the strongest physical isolation
- Cons: Conflicts with the concrete SMG and existing fused-provider workflows, Adds placement and protocol structure before a split workflow exists
- Rejected because: The confirmed first-phase workflow needs semantic separation but permits one fused process.
Separate Gateway, P/D Router, and Engine semantics while permitting a fused Gateway and P/D Router process (accepted)
Section titled “Separate Gateway, P/D Router, and Engine semantics while permitting a fused Gateway and P/D Router process (accepted)”- Pros: Matches the distinct reasons each component changes, Supports current SMG and other fused providers, Allows a later process split without making fusion semantically opaque
- Cons: Requires protocol-v7 migration and compatibility-pair validation
Expose a general component graph for arbitrary fused and split realizations (rejected)
Section titled “Expose a general component graph for arbitrary fused and split realizations (rejected)”- Pros: Could encode future deployment shapes without another schema change
- Cons: Introduces a workflow language with no demonstrated operator need, Weakens canonical topology and ownership invariants
- Rejected because: Current workflows need three closed logical components, not an arbitrary deployment DAG.