Decision record · ADR
ADR-0011: Support both InferLab and SGLang routing backends for SGLang P/D
Status: accepted | Date: 2026-07-11
References: RFC-0003, RFC-0006, RFC-0002
Context
Section titled “Context”SGLang native prefill/decode serving separates model processes from the request router and uses a per-request bootstrap rendezvous for both Mooncake and NIXL. InferLab already owns a generic multi-role lifecycle, but it must decide which routing implementations become product capabilities for SGLang P/D.
Problem Statement
Section titled “Problem Statement”A built-in InferLab proxy would provide a dependency-light experimental baseline, while SGLang Model Gateway provides the upstream routing behavior and evolves with SGLang. Treating either one as the only backend would discard a user-required workflow: the operator has confirmed that both are required.
Constraints
Section titled “Constraints”RFC-0003 keeps routing backend selection independent from framework and transfer mechanism, makes the built-in proxy the default, and assigns lifecycle ownership to InferLab. RFC-0002 makes the committed Pixi serving environment and lock authoritative for workspace-provided runtime packages. RFC-0006 keeps framework-specific command lowering in the Python integration and separately governs InferLab’s workspace-side integration packages. Existing support selects the built-in SGLang proxy; this decision retains it as one capability rather than treating it as an exclusive constraint.
Options Considered
Section titled “Options Considered”The alternatives are a built-in-only product, an upstream-router-only product, or two explicitly selected backends with separate identities and qualification evidence.
Decision
Section titled “Decision”We will expose both builtin and sglang-router as separately selected routing backends for SGLang prefill/decode serving because they satisfy complementary operator workflows rather than interchangeable implementation preferences.
- Dependency-light baseline:
builtinwill be the default experimental route and will not require an upstream router package. - Upstream routing behavior:
sglang-routergives operators SGLang Model Gateway semantics from the workspace’s committed serving environment. - Common lifecycle evidence: both backends use the same resolved SGLang role graph while InferLab retains placement, readiness, process lifecycle, cleanup, and record authority under RFC-0003.
Implementation Notes
Section titled “Implementation Notes”builtin and sglang-router remain distinct recorded identities; neither name aliases the other. The built-in proxy stays limited to the concrete InferLab workload endpoints and controls that are specified and tested, rather than claiming feature parity with SGLang Model Gateway. The SGLang integration owns Model Gateway command lowering under RFC-0006, while the workspace owns the router package through its committed serving environment and lock under RFC-0002. Both backends compose independently with Mooncake and NIXL.
Consequences
Section titled “Consequences”Positive
Section titled “Positive”- Operators can choose a self-contained experimental baseline or the upstream gateway without changing the P/D role and transfer configuration.
- Backend identity, rendered commands, readiness, targets, and cleanup remain explicit in records.
- SGLang’s bootstrap request protocol is the same for Mooncake and NIXL, so routing implementation remains independent from transfer selection.
- SGLang protocol evolution can land first in Model Gateway without requiring built-in feature parity outside InferLab’s bounded public surface.
Negative
Section titled “Negative”- InferLab carries and tests a bounded SGLang request proxy in addition to the integration-managed Model Gateway path. The supported subset still has to follow incompatible changes to SGLang’s bootstrap, request, streaming, and cache-control semantics.
- The Model Gateway path adds an independently versioned workspace dependency whose identity is carried by the committed serving lock.
Neutral
Section titled “Neutral”- Transfer selection changes SGLang server lowering but not the proxy request protocol; backend behavior and transfer mapping are orthogonal verification axes rather than four distinct proxy implementations.
- The existing role, replica, link, endpoint, and allocation shapes represent both backends; exact all-worker Model Gateway readiness remains a separate probe-contract concern.
Alternatives Considered
Section titled “Alternatives Considered”Expose both builtin and sglang-router as separately selected SGLang P/D routing backends (accepted)
Section titled “Expose both builtin and sglang-router as separately selected SGLang P/D routing backends (accepted)”- Pros: Preserves a dependency-light InferLab baseline and upstream SGLang routing in one product contract, Keeps backend identity, commands, lifecycle, and qualification evidence explicit
- Cons: Requires maintaining one bounded built-in proxy plus Model Gateway lowering and lifecycle integration
Support only the InferLab built-in proxy (rejected)
Section titled “Support only the InferLab built-in proxy (rejected)”- Pros: Needs no additional router package in the serving workspace, Keeps the default experimental path under InferLab control
- Cons: Duplicates an evolving SGLang request-routing protocol in Rust, Does not provide the upstream gateway behavior the operator requires
- Rejected because: The operator explicitly requires SGLang Model Gateway in addition to the built-in baseline.
Support only SGLang Model Gateway (rejected)
Section titled “Support only SGLang Model Gateway (rejected)”- Pros: Delegates SGLang request semantics and routing evolution to upstream, Avoids a second SGLang request proxy implementation
- Cons: Removes the built-in dependency-light baseline, Makes every SGLang P/D experiment depend on a separately pinned router package
- Rejected because: The operator explicitly requires the built-in proxy as well as SGLang Model Gateway.