Skip to content

Decision record · ADR

ADR-0012: Support both InferLab and native TensorRT-LLM routing for TensorRT-LLM P/D

Status: accepted | Date: 2026-07-12

References: RFC-0003, RFC-0006, ADR-0013

TensorRT-LLM prefill/decode serving separates context and generation workers from a request orchestrator. InferLab must decide whether its product surface exposes only the upstream TensorRT-LLM disaggregated server, only an InferLab-owned proxy, or both as distinct routing backends.

The native TensorRT-LLM disaggregated server owns the complete evolving upstream orchestration behavior. An InferLab-owned proxy would provide a native-frontend-independent comparison baseline, but unlike the SGLang proxy it must perform a sequential context-only request, preserve the returned opaque transfer metadata, and construct a generation-only request. Treating the native server as builtin would erase implementation identity, while claiming full native parity for an InferLab proxy would create an unbounded compatibility promise.

RFC-0003:C-GATEWAY-PD-ROUTER keeps routing selection independent from framework and transfer mechanism and requires distinct routing evidence. RFC-0006:C-INTEGRATIONS assigns framework-specific native command lowering to the TensorRT-LLM integration while InferLab retains lifecycle and record authority. The inspected TensorRT-LLM serving interface requires a two-stage context-to-generation handoff for context-first disaggregation. The operator has confirmed that both the InferLab-owned baseline and the native TensorRT-LLM path are required.

The built-in implementation can remain bounded to the scalar completion workflow InferLab consumes. Chat, generation-first scheduling, conditional disaggregation, KV-aware routing, conversation routing, dynamic membership, and native metrics aggregation are not required for that baseline.

The viable choices are two separately identified backends, native-only routing, built-in-only routing, or retaining the current absence of TensorRT-LLM P/D support.

We will expose builtin and trtllm-disaggregated as separately selected routing backends for TensorRT-LLM prefill/decode serving because they satisfy complementary experiment workflows without pretending to be interchangeable implementations.

  1. InferLab-owned baseline: builtin provides a static round-robin, context-first path with no native frontend process.
  2. Upstream behavior: trtllm-disaggregated runs the native TensorRT-LLM disaggregated server from the workspace serving environment.
  3. Shared topology, distinct evidence: both backends use the same prefill/decode role and NIXL worker configuration while retaining separate commands, implementation identities, readiness, records, and capability claims.

The built-in backend intentionally implements only the bounded handoff contract defined by RFC-0003:C-TENSORRT-LLM-PREFILL-DECODE and transparently carries metadata it does not interpret. Native-only capabilities do not become implied built-in support, and neither backend identity may alias the other.

  • Operators can compare an InferLab-owned routing implementation with native TensorRT-LLM orchestration without changing the worker topology or transfer selection.
  • Backend identity, command, targets, readiness, cleanup, and capability evidence remain explicit.
  • The built-in proxy’s bounded surface matches the completion workflow InferLab actually consumes.
  • Transparent transfer of unknown disaggregated metadata reduces coupling to the current upstream field set.
  • InferLab must track incompatible changes to TensorRT-LLM’s context and generation handoff semantics. The mitigation is a narrow contract and a capability claim that can be downgraded when the protocol changes incompatibly.
  • Two routing implementations increase verification and maintenance cost. The mitigation is to share only lifecycle and HTTP mechanics while keeping framework protocol logic in its owning proxy.
  • Static round-robin selection and a sequential non-streaming context phase do not promise scheduling behavior or performance equivalence with the native orchestrator.
  • Successful worker qualification does not establish either routing implementation; their evidence remains independent.
  • The native backend may expose additional upstream endpoints and policies, but those do not expand the built-in capability claim.
  • Neither backend advertises prefix-cache reset until the serving interface can apply it across every prefill and decode worker.

Expose builtin and trtllm-disaggregated as separately selected TensorRT-LLM P/D routing backends (accepted)

Section titled “Expose builtin and trtllm-disaggregated as separately selected TensorRT-LLM P/D routing backends (accepted)”
  • Pros: Keeps implementation identity, commands, readiness, and qualification evidence explicit, Preserves an InferLab-owned routing baseline and upstream orchestration in one role graph
  • Cons: Requires maintaining a bounded TensorRT-LLM handoff protocol in InferLab alongside native lowering

Support only the native TensorRT-LLM disaggregated server (rejected)

Section titled “Support only the native TensorRT-LLM disaggregated server (rejected)”
  • Pros: Delegates orchestration semantics and upstream evolution to TensorRT-LLM, Avoids maintaining a second handoff implementation
  • Cons: Makes every TensorRT-LLM P/D experiment depend on native frontend availability, Removes the native-frontend-independent InferLab comparison baseline
  • Rejected because: The operator explicitly requires the built-in baseline in addition to the native path.

Support only the InferLab built-in TensorRT-LLM proxy (rejected)

Section titled “Support only the InferLab built-in TensorRT-LLM proxy (rejected)”
  • Pros: Provides a self-contained and tightly bounded experiment baseline
  • Cons: Drops the upstream orchestration behavior needed for comparison, Makes InferLab the only authority for an evolving framework handoff protocol
  • Rejected because: The operator explicitly requires the native TensorRT-LLM route as a separately observable backend.

Retain the current absence of TensorRT-LLM prefill/decode support (rejected)

Section titled “Retain the current absence of TensorRT-LLM prefill/decode support (rejected)”
  • Pros: Adds no protocol or maintenance surface
  • Cons: Leaves the requested TensorRT-LLM P/D workflow unavailable
  • Rejected because: The requested capability cannot be satisfied by the status quo.