Decision record · ADR
ADR-0012: Support both InferLab and native TensorRT-LLM routing for TensorRT-LLM P/D
Status: accepted | Date: 2026-07-12
References: RFC-0003, RFC-0006, ADR-0013
Context
Section titled “Context”TensorRT-LLM prefill/decode serving separates context and generation workers from a request orchestrator. InferLab must decide whether its product surface exposes only the upstream TensorRT-LLM disaggregated server, only an InferLab-owned proxy, or both as distinct routing backends.
Problem Statement
Section titled “Problem Statement”The native TensorRT-LLM disaggregated server owns the complete evolving upstream orchestration behavior. An InferLab-owned proxy would provide a native-frontend-independent comparison baseline, but unlike the SGLang proxy it must perform a sequential context-only request, preserve the returned opaque transfer metadata, and construct a generation-only request. Treating the native server as builtin would erase implementation identity, while claiming full native parity for an InferLab proxy would create an unbounded compatibility promise.
Constraints
Section titled “Constraints”RFC-0003:C-GATEWAY-PD-ROUTER keeps routing selection independent from framework and transfer mechanism and requires distinct routing evidence. RFC-0006:C-INTEGRATIONS assigns framework-specific native command lowering to the TensorRT-LLM integration while InferLab retains lifecycle and record authority. The inspected TensorRT-LLM serving interface requires a two-stage context-to-generation handoff for context-first disaggregation. The operator has confirmed that both the InferLab-owned baseline and the native TensorRT-LLM path are required.
The built-in implementation can remain bounded to the scalar completion workflow InferLab consumes. Chat, generation-first scheduling, conditional disaggregation, KV-aware routing, conversation routing, dynamic membership, and native metrics aggregation are not required for that baseline.
Options Considered
Section titled “Options Considered”The viable choices are two separately identified backends, native-only routing, built-in-only routing, or retaining the current absence of TensorRT-LLM P/D support.
Decision
Section titled “Decision”We will expose builtin and trtllm-disaggregated as separately selected routing backends for TensorRT-LLM prefill/decode serving because they satisfy complementary experiment workflows without pretending to be interchangeable implementations.
- InferLab-owned baseline:
builtinprovides a static round-robin, context-first path with no native frontend process. - Upstream behavior:
trtllm-disaggregatedruns the native TensorRT-LLM disaggregated server from the workspace serving environment. - Shared topology, distinct evidence: both backends use the same prefill/decode role and NIXL worker configuration while retaining separate commands, implementation identities, readiness, records, and capability claims.
Implementation Notes
Section titled “Implementation Notes”The built-in backend intentionally implements only the bounded handoff contract defined by RFC-0003:C-TENSORRT-LLM-PREFILL-DECODE and transparently carries metadata it does not interpret. Native-only capabilities do not become implied built-in support, and neither backend identity may alias the other.
Consequences
Section titled “Consequences”Positive
Section titled “Positive”- Operators can compare an InferLab-owned routing implementation with native TensorRT-LLM orchestration without changing the worker topology or transfer selection.
- Backend identity, command, targets, readiness, cleanup, and capability evidence remain explicit.
- The built-in proxy’s bounded surface matches the completion workflow InferLab actually consumes.
- Transparent transfer of unknown disaggregated metadata reduces coupling to the current upstream field set.
Negative
Section titled “Negative”- InferLab must track incompatible changes to TensorRT-LLM’s context and generation handoff semantics. The mitigation is a narrow contract and a capability claim that can be downgraded when the protocol changes incompatibly.
- Two routing implementations increase verification and maintenance cost. The mitigation is to share only lifecycle and HTTP mechanics while keeping framework protocol logic in its owning proxy.
- Static round-robin selection and a sequential non-streaming context phase do not promise scheduling behavior or performance equivalence with the native orchestrator.
- Successful worker qualification does not establish either routing implementation; their evidence remains independent.
Neutral
Section titled “Neutral”- The native backend may expose additional upstream endpoints and policies, but those do not expand the built-in capability claim.
- Neither backend advertises prefix-cache reset until the serving interface can apply it across every prefill and decode worker.
Alternatives Considered
Section titled “Alternatives Considered”Expose builtin and trtllm-disaggregated as separately selected TensorRT-LLM P/D routing backends (accepted)
Section titled “Expose builtin and trtllm-disaggregated as separately selected TensorRT-LLM P/D routing backends (accepted)”- Pros: Keeps implementation identity, commands, readiness, and qualification evidence explicit, Preserves an InferLab-owned routing baseline and upstream orchestration in one role graph
- Cons: Requires maintaining a bounded TensorRT-LLM handoff protocol in InferLab alongside native lowering
Support only the native TensorRT-LLM disaggregated server (rejected)
Section titled “Support only the native TensorRT-LLM disaggregated server (rejected)”- Pros: Delegates orchestration semantics and upstream evolution to TensorRT-LLM, Avoids maintaining a second handoff implementation
- Cons: Makes every TensorRT-LLM P/D experiment depend on native frontend availability, Removes the native-frontend-independent InferLab comparison baseline
- Rejected because: The operator explicitly requires the built-in baseline in addition to the native path.
Support only the InferLab built-in TensorRT-LLM proxy (rejected)
Section titled “Support only the InferLab built-in TensorRT-LLM proxy (rejected)”- Pros: Provides a self-contained and tightly bounded experiment baseline
- Cons: Drops the upstream orchestration behavior needed for comparison, Makes InferLab the only authority for an evolving framework handoff protocol
- Rejected because: The operator explicitly requires the native TensorRT-LLM route as a separately observable backend.
Retain the current absence of TensorRT-LLM prefill/decode support (rejected)
Section titled “Retain the current absence of TensorRT-LLM prefill/decode support (rejected)”- Pros: Adds no protocol or maintenance surface
- Cons: Leaves the requested TensorRT-LLM P/D workflow unavailable
- Rejected because: The requested capability cannot be satisfied by the status quo.