Reference
Backend Support Matrix
This public document is the authority for the operator-visible backend capabilities of the InferLab release or development line that ships this file. It describes workflows that InferLab plans, executes, and records; it is not a list of every feature offered by the upstream frameworks.
Status meanings:
- Qualified: implemented and demonstrated by a real downstream execution record for the baseline named below.
- Supported: implemented and covered by deterministic integration tests, but not qualified for every relevant hardware and model shape.
- Limited: implemented only under the conditions stated in the cell.
- Unsupported: a real probe demonstrated a specific conformance failure for the exact integration revision, route, model, and baseline named in the cell.
- Inconclusive: a real probe ran but its transport or HTTP outcome could not establish endpoint capability.
- Unqualified: no complete real public-route record establishes the capability for the exact path in the cell.
- —: rejected by the integration or not exposed by InferLab.
A qualified baseline is evidence for that concrete shape, not blanket certification of every framework version, model, device, or parallel configuration.
The version correspondence for the current product line — inferlab --version
states the adapter protocol version a binary speaks:
| InferLab product | Adapter protocol | Adapter SDK | Framework integrations |
|---|---|---|---|
| 0.13.2–0.13.3 | v9 | 0.9.0 | 0.8.0 (vLLM, SGLang, TensorRT-LLM, TokenSpeed, Specialized Engine) |
| 0.14.0–0.14.2 | v10 | 0.10.0 | 0.9.0 (vLLM, SGLang, TensorRT-LLM, TokenSpeed, Specialized Engine) |
| 0.15.0–0.16.0 | v11 | 0.11.0 | 0.10.0 (vLLM, SGLang, TensorRT-LLM, TokenSpeed); Specialized Engine 0.10.0, 0.10.1 from 0.15.1 |
| 0.17.0–0.17.1 | v11 | 0.11.0 | 0.10.0 (vLLM, SGLang, TensorRT-LLM); TokenSpeed 0.11.0 and Specialized Engine 0.11.0, both on inferlab-gateway-smg 0.1.0 |
| 0.18.0–0.18.1 | v12 | 0.12.0 | 0.11.0 (vLLM, SGLang, TensorRT-LLM), vLLM on inferlab-gateway-dynamo 0.1.0; TokenSpeed 0.12.0 and Specialized Engine 0.12.0, both on inferlab-gateway-smg 0.2.0 |
| 0.18.2 | v12 | 0.12.0 | 0.11.0 (vLLM, TensorRT-LLM), SGLang 0.11.1, vLLM on inferlab-gateway-dynamo 0.1.0; TokenSpeed 0.12.0 and Specialized Engine 0.12.0, both on inferlab-gateway-smg 0.2.0 |
| 0.19.0 | v13 | 0.13.0 | 0.12.0 (vLLM, SGLang, TensorRT-LLM), vLLM on inferlab-gateway-dynamo 0.2.0; TokenSpeed 0.13.0 and Specialized Engine 0.13.0, both on inferlab-gateway-smg 0.3.0 |
| 0.20.0–0.22.1 | v13 | 0.13.0 | 0.12.0 (vLLM, TensorRT-LLM), SGLang 0.12.1, vLLM on inferlab-gateway-dynamo 0.2.0; TokenSpeed 0.13.0 and Specialized Engine 0.13.0, both on inferlab-gateway-smg 0.3.0 |
| 0.23.0–0.23.2 | v14 | 0.14.0 | 0.13.0 (vLLM, SGLang, TensorRT-LLM), vLLM on inferlab-gateway-dynamo 0.3.0; TokenSpeed 0.14.0 and Specialized Engine 0.14.0, both on inferlab-gateway-smg 0.4.0; vLLM, SGLang, and Specialized Engine on inferlab-kvstore 0.1.0 |
| 0.23.3 | v15 | 0.15.0 | 0.14.0 (vLLM, SGLang, TensorRT-LLM), vLLM on inferlab-gateway-dynamo 0.4.0; TokenSpeed 0.15.0 and Specialized Engine 0.15.0, both on inferlab-gateway-smg 0.5.0; vLLM, SGLang, and Specialized Engine on inferlab-kvstore 0.2.0 |
| 0.23.4–0.23.6 | v16 | 0.16.0 | 0.15.0 (vLLM, SGLang, TensorRT-LLM), vLLM on inferlab-gateway-dynamo 0.5.0; TokenSpeed 0.16.0 and Specialized Engine 0.16.0, 0.16.1, 0.17.0, both on inferlab-gateway-smg 0.6.0; vLLM, SGLang, and Specialized Engine on inferlab-kvstore 0.3.0 |
Each framework integration package pins its Adapter SDK exactly, the TokenSpeed and Specialized Engine integrations also pin the shared SMG gateway package, the vLLM integration pins the shared Dynamo gateway package, and the vLLM, SGLang, and Specialized Engine integrations pin the shared KV store package, so pinning the integration versions above is sufficient. A Qualified cell describes the workflow and shape of the execution records it rests on; its records come from the current protocol line unless the cell names an earlier one. Records from earlier protocol lines, including the protocol-v7 candidate records some cells cite, remain historical evidence and have not been re-run on the current packages.
Serving And Control
Section titled “Serving And Control”| Capability | vLLM | SGLang | TensorRT-LLM | TokenSpeed | Specialized Engine |
|---|---|---|---|---|---|
| Integration package | inferlab-integration-vllm |
inferlab-integration-sglang |
inferlab-integration-tensorrt-llm |
inferlab-integration-tokenspeed |
inferlab-integration-specialized-engine |
Single-node single topology |
Qualified for the retained baseline and the protocol-v7 candidate on the Qwen3 MoE SM120 TP1 baseline below | Qualified for the retained baseline and the protocol-v7 candidate on the Qwen3 MoE SM120 TP1 baseline below | Qualified for the retained baseline below; protocol-v7 candidate Unqualified | Qualified for the retained baseline below; protocol-v7 candidate Unqualified | Qualified for one replica and for two replicas on the 8-device and two-replica MoE baselines below on protocol-v14 records, and under the protocol-v15 contract, which renders only allocation facts, for one TP4 replica and two TP4 replicas of the MoE Engine below, with an Eval and serving Benches completing with no failed requests: several replicas route by the typed Gateway policy (least_load by default; cache_aware and power_of_two qualified) and are ready only when SMG’s /workers registry lists every replica healthy with its reported data-parallel rank count; profiling covers a single replica only |
single public component |
Qualified for the retained direct-Engine baseline and the protocol-v7 candidate on Qwen3 MoE SM120 TP1 | Qualified for the retained direct-Engine baseline and the protocol-v7 candidate on Qwen3 MoE SM120 TP1 | Qualified for the retained direct-Engine baseline; protocol-v7 candidate Unqualified | Qualified for the retained direct-Engine baseline; protocol-v7 candidate Unqualified | Supported: the smg Gateway; the protocol-v7 0.3.0 candidate is Unqualified |
Gateway-backed single |
Qualified for dynamo on Qwen3-4B on NVIDIA RTX PRO 6000 Blackwell Server Edition (SM120) TP1 with ai-dynamo 1.5.0 and its vLLM 0.28.0: the Dynamo frontend, one dynamo.vllm worker, and an InferLab-managed etcd discovery process became ready only once the frontend registry listed the worker at its allocated request-plane address; the smoke Eval and serving Benches completed through the Gateway, and cleanup was verified. Workers run one rank per replica |
— | — | — | Supported: smg, from the tokenspeed-smg distribution; the protocol-v7 0.3.0 candidate is Unqualified |
External KV store (mooncake) |
Supported on a direct single server: MooncakeStoreConnector through a rendered MOONCAKE_CONFIG_PATH file, with per-rank kv_store_* Engine settings; its prefix-cache reset adds reset_external=true, which wipes the store, and conditioning stays per rank; P/D and Dynamo reject a store; real-run qualification pending |
Supported on a single server: HiCache with the mooncake storage backend and the master address in its extra config, with per-instance kv_store_* Engine settings; no prefix-cache reset is declared with a store, because /flush_cache leaves the storage backend intact, so reset-based cold and primed starts (replay, dataset, linear-session, and adaptive Benches) are rejected, while generated populations run them namespaced without a reset; P/D rejects a store; real-run qualification pending |
— | — | Qualified on the two-replica MoE baseline below on protocol-v14 records: the store starts before and stops after every Engine, and the Engines share one prefix cache, so a primed start conditions once; from protocol v15 each Engine receives --kv-tier-backend mooncake --kv-tier-store <host:port> and reports its own tier access and publication bound, readiness engaging it only at the store address, pending requalification. An Engine-provided store, named by any other backend and run from the stack’s kv_store_command with --listen and --metrics-port and /health readiness, is Supported: InferLab runs it like the Mooncake master and each Engine receives the declared name and the store’s address; real-run qualification is pending. With the stack setting engine_metrics, each replica is also allocated a recorded metrics port that rank zero receives as --metrics-port, for per-Engine store counters |
External KV store (filesystem) |
— | — | — | — | Supported, real-hardware qualification pending: each machine binds kv_tier_directory in its local machine binding, and every Engine on it receives the directory unchanged as --kv-tier-store with --kv-tier-backend filesystem; InferLab runs no store process and does not create, inspect, or clean the directory; each Engine reports kv_tier_state, kv_tier_store, kv_tier_access, and kv_tier_publication_ms, readiness requires every Engine engaged at its directory with one access per role, and every profiling window reads each Engine’s state |
| Multi-node replica | Supported | Qualified on Qwen3-4B on NVIDIA RTX PRO 6000 Blackwell Server Edition (SM120) with SGLang 0.5.21: one TP2 replica placed as one rank on each of two machines ran one SGLang node per rank initialized at the rank-zero master port, the tensor-parallel group spanned both machines, the smoke Eval and a serving Bench completed through rank zero, and cleanup freed both devices. Multi-node covers serve-role replicas without attention data parallelism; a prefill or decode replica spanning machines, or a multi-node replica with attention data parallelism, is rejected before launch |
— | — | Qualified on protocol-v15 records for one DEP16 replica of the MoE Engine below spanning two 8-device machines: readiness observed a data-parallel rank count of 16, serving Benches at concurrency 16, 64, and 128 completed with no failed requests, and killing one member’s worker failed the running case naming the lost process, skipped the remaining cases, and cleanup was verified on both machines. A serve replica placed as an explicit multi-rank binding runs one Engine per machine; every host receives --nnodes, its own --node-rank, and --dist-init-addr with host 0’s allocated rendezvous address, every host its own machine’s --model, and only host 0 --listen; the qualifying records predate these names, which replaced --host-count, --host-index, --rendezvous-listen, and --rendezvous, and a record under the new names is pending; the Gateway, readiness, and observed facts address host 0 only; unequal hosts, multi-host prefill/decode, and multi-host profiling are rejected before launch |
| Disaggregated prefill/decode | Qualified | Qualified for the pairing-specific baselines below | Qualified: built-in and native trtllm-disaggregated frontend pairs |
Qualified for the maintained 1P1D pairing below | Supported: independent prefill and decode replica counts, each replica one rank process owning its devices; every Engine receives --disaggregation-mode, the declared --disaggregation-transfer-backend, and on prefill an InferLab-allocated --disaggregation-bootstrap-port; a declared KV tier, mooncake or filesystem, reaches both roles, each reporting its own access, and both are probed; profiling is rejected; Qualified for 1P1D on the 4 + 4-device MoE baseline below, with and without a mooncake KV store, on protocol-v14 records; the protocol-v15 contract, which renders only allocation facts, is pending requalification |
| KV-transfer backend | Qualified: Mooncake and NIXL; Mooncake sender workers follow vLLM’s default unless mooncake_num_workers is set; vLLM runs twice that many sender tasks, each held until its request finishes prefill, so a lower value serializes prefill-side transfers |
Qualified: Mooncake and NIXL in the pairing-specific baselines below | Qualified: NIXL with the built-in frontend and native trtllm-disaggregated pair |
Qualified: Mooncake for the maintained 1P1D pairing below | Qualified for mooncake on the 1P1D MoE baseline below on protocol-v14 records: the declared mechanism is passed to the Engine, which rejects one it does not implement before ready, and a smoke Eval and a 256-request Bench completed through the fused SMG Gateway and P/D Router public route, with and without a KV store; the protocol-v15 contract is pending requalification; from protocol v16 the declared mechanism is an open name passed to the Engine unchanged, so an Engine-owned transport runs under its own name; names other than mooncake are Unqualified |
| P/D Gateway backend | Supported: builtin, vllm-router. dynamo over NIXL with one rank per replica is Qualified for 1P1D on Qwen3-4B on NVIDIA RTX PRO 6000 Blackwell Server Edition (SM120) TP1 with ai-dynamo 1.5.0 and its vLLM 0.28.0, on one machine and with prefill and decode on two machines: registry readiness matched both workers at their allocated addresses, decode logged successful NIXL KV transfers, the smoke Eval and serving Benches completed, and cleanup was verified on both machines |
Supported: builtin, sglang-router |
Supported: builtin, trtllm-disaggregated |
Supported: smg |
Qualified: smg, on the 1P1D MoE baseline below |
| P/D Router backend | Supported: builtin, vllm-router, and dynamo (fused with the dynamo Gateway) |
Supported: builtin, sglang-router |
Supported: builtin, trtllm-disaggregated |
Supported: smg, routing by the typed pd_router_settings prefill_policy and decode_policy (round_robin by default) |
Qualified: smg, on the 1P1D MoE baseline below, routing by the typed pd_router_settings prefill_policy and decode_policy (round_robin by default) |
| P/D frontend binding | Supported: one [gateway, pd_router] process |
Supported: one [gateway, pd_router] process |
Supported: one [gateway, pd_router] process |
Supported: one [gateway, pd_router] process |
Qualified: one [gateway, pd_router] process |
| Public workload paths | /v1/completions; /v1/chat/completions |
/v1/completions; /v1/chat/completions |
/v1/completions; /v1/chat/completions |
/v1/completions; /v1/chat/completions |
/v1/completions; /v1/chat/completions |
| AIPerf server-metrics capability | Qualified for direct single on the public endpoint at /metrics: a real protocol-v7 candidate Bench preserved the native export and produced both SPEED acceptance reports; Gateway and P/D public endpoints omit the capability |
Qualified for direct single on the public endpoint at /metrics when effective setting enable_metrics = true; Gateway and P/D public endpoints omit the capability |
— | — | Supported through SMG’s separately allocated prometheus port at /metrics; least_load preserves the single-target routing result while activating canonical Engine load polling; real candidate Bench collection and cleanup are demonstrated from a dirty workspace, so reproducible qualification remains pending |
| Completion request used by InferLab | Qualified for the retained scalar-prompt baseline and the protocol-v7 candidate on Qwen3 MoE SM120 TP1 | Qualified for the retained scalar-prompt baseline and the protocol-v7 candidate on Qwen3 MoE SM120 TP1 | Qualified for the retained scalar-prompt baseline; protocol-v7 candidate Unqualified | Qualified for the retained scalar-prompt baseline; protocol-v7 candidate Unqualified | Supported: deterministic scalar prompt through SMG; the protocol-v7 0.3.0 candidate is Unqualified |
| Exact flat serving Bench | Qualified on Qwen3 MoE SM120 TP1: the AIPerf runner froze scalar prompts for fixed and distributed ISL populations, and every completed request reconciled the observed backend ISL exactly | Qualified on Qwen3 MoE SM120 TP1 for the fixed-ISL full-prefix population; every completed request reconciled the observed backend ISL exactly. Other exact-flat shapes remain Supported | Supported through the declared completions path; candidate real-route qualification is pending | Supported through the declared completions path; candidate real-route qualification is pending | Supported through the declared completions path; candidate real-route qualification is pending |
| Exact locally rendered-chat serving Bench | Qualified on Qwen3 MoE SM120 TP1 for both the tokenizer-default and a definition-supplied template, frozen into exact scalar prompts with backend ISL reconciliation | Supported by the shared runner; candidate real-route qualification is pending | Supported by the shared runner; candidate real-route qualification is pending | Supported by the shared runner; candidate real-route qualification is pending | Supported by the shared runner; candidate real-route qualification is pending |
| Controlled final-prompt prefix geometry | Qualified on Qwen3 MoE SM120 TP1 for zero, partial, and full fixed-ISL sharing plus an inclusive-uniform ISL population; exact flat and locally rendered prompts use one nested canonical-prefix stream, while cache reads remain separate observed evidence | Qualified on Qwen3 MoE SM120 TP1 for full fixed-ISL sharing; other shared-prefix shapes remain Supported | Supported by the shared runner; real cache-read qualification is pending | Supported by the shared runner; real cache-read qualification is pending | Supported by the shared runner; real cache-read qualification is pending |
| Controlled Bench cache start and observed reuse | Qualified for direct single on Qwen3 MoE SM120 TP1 with enable_prompt_tokens_details = true: uncontrolled, cold, and primed full-prefix cases preserved per-request prompt/cache tokens; cold reset before profiling, primed reset and exact conditioning before profiling, and cleanup were verified. Under attention data parallelism a primed start primes each DP rank with one X-Data-Parallel-Rank-pinned conditioning request, requiring a vLLM build that honors that header (recent main; older releases silently ignore it and degrade to single-rank priming). The built-in Mooncake and NIXL P/D pairs fan priming out through the proxy’s POST /prime_prefix_cache, covering every prefill replica × DP rank through the ordinary pairing flow — Qualified on the built-in NIXL pair for DeepSeek-V4-Flash SM120 (prefill TP2, decode TP2, both roles enable_prompt_tokens_details = true): the primed case conditioned the shared prefix through the Gateway and completed with per-request prompt_cache_read_ratio 1.0; the Gateway endpoint declares the cache-read capability only when both roles enable the reporting setting; the vllm-router pair declares no fan-out capability, so a primed start is rejected at planning when more than one prefill-side cache-owning target (replica x attention DP rank) sits behind the Gateway; a single-target shape needs no fan-out and conditions through the ordinary serving flow. Dynamo is Limited for routed single and for 1P1D on one and two machines on Qwen3-4B on NVIDIA RTX PRO 6000 Blackwell Server Edition (SM120) TP1 with ai-dynamo 1.5.0 and its vLLM 0.28.0: cold cases called every worker’s reset per target before profiling and primed cases conditioned the full shared prefix through the Gateway and completed with prompt_cache_read_ratio 0.97, but ai-dynamo 1.5.0 reports reset success even when vLLM declined (see prefix-cache reset below), so either start holds only when no KV blocks were held at reset time; a cold start resets every worker per target, and a primed start conditions through the ordinary serving flow only when one prefill-side cache-owning target exists, since the Dynamo frontend declares no fan-out; the frontend reports cache-read usage itself |
Qualified for direct single on Qwen3 MoE SM120 TP1 with enable_cache_report = true: uncontrolled, cold, and primed full-prefix cases preserved per-request prompt/cache tokens, including SGLang’s declared omitted-zero representation; cold reset, primed conditioning, and cleanup were verified. Under attention data parallelism a primed start primes each DP rank with one X-Data-Parallel-Rank-pinned conditioning request. The built-in P/D pair fans priming out through the proxy’s POST /prime_prefix_cache, covering every prefill replica × DP rank through the ordinary bootstrap pairing flow (real-route qualification pending); the sglang-router pair declares no fan-out capability, so a primed start is rejected at planning when more than one prefill-side cache-owning target (replica x attention DP rank) sits behind the Gateway; a single-target shape needs no fan-out and conditions through the ordinary serving flow |
Unqualified; controlled cold and primed starts are unavailable because the integration exposes no reset capability | Unqualified | Qualified on the two-replica MoE baseline below for cold starts with and without a KV store and for primed starts through a mooncake KV store, on protocol-v14 records; the protocol-v15 contract, which renders only allocation facts, is pending requalification: the SMG Gateway endpoint declares the POST /flush_cache reset and no conditioning fan-out, which SMG Gateways do not serve; one cache-owning target — one replica whose Engine reports one data-parallel rank, or Engines sharing a tier that all report read_write — conditions through the ordinary serving flow, and any other primed start fails after readiness, before any reset, naming the reported counts |
| Chat-completions execution | Qualified for direct single under the protocol-v7 candidate, including an operator-supplied server-side chat template; built-in Mooncake and NIXL P/D preserve the route but remain unqualified |
Supported by deterministic built-in P/D frontend coverage; the integration-rendered pair and every exact public route remain Unqualified | Supported by deterministic built-in P/D frontend coverage for context-first streaming and non-streaming handoff; the integration-rendered pair and every exact public route remain Unqualified | Unqualified: the named path is preserved; the native Gateway/P/D Router pair requires separate route qualification | Unqualified: SMG preserves the path, but no exact public-route record qualifies chat execution |
| Vision smoke Eval | Qualified on Qwen2.5-VL-7B-Instruct SM120 TP1: the vision smoke POSTed one chat-completions request carrying the release-owned fixed test image as an image_url part, the served model’s answer described the image content, and the schema-20 record preserves the vision selection with request and response artifacts |
Supported: same native smoke path; real VLM route qualification is pending | Supported: same native smoke path; real VLM route qualification is pending | Supported: same native smoke path; real VLM route qualification is pending | —: the token-worker contract carries no multimodal input, so an image part cannot reach the Engine (Specialized Engines) |
| Image-decorated random Bench | Qualified on Qwen2.5-VL-7B-Instruct SM120 TP1 for both the synthetic noise supply (two 512x384 images per request) and a digest-bound operator directory (one image per request, shuffle-cycle sampling): every request carried image_url content parts on the chat-completions route, and backend-observed prompt tokens reconciled the 512-token text target plus per-image tokens exactly (1020 and 766) |
Supported: same runner-side decoration; real VLM route qualification is pending | Supported: same runner-side decoration; real VLM route qualification is pending | Supported: same runner-side decoration; real VLM route qualification is pending | —: the token-worker contract carries no multimodal input, so an image part cannot reach the Engine (Specialized Engines) |
| SemiAnalysis AgentX trace replay | Qualified for the 256k corpus on direct single through the release-owned AIPerf integration build, which carries AgentX support, the agentx scenario preset, and a shutdown fix not yet in an upstream AIPerf release: vLLM serving Qwen3.6-35B-A3B FP8 at TP2 on two NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs completed the current profile at root-tree concurrency 8 with the profile’s default 3600-second profiling window and default seed, after the per-lane cache-pressure warmup; the case was scenario-valid with no warmup errors, ordinary failures, or context overflows, preserved diagnostic-level raw records, and cleanup verified. The full-context corpus is Supported: it completes the current profile against a deterministic stand-in server, including requests beyond 256k tokens, but real-hardware qualification is pending |
Supported by the shared runner; real SGLang route qualification is pending | Supported by the shared runner; real TensorRT-LLM route qualification is pending | Supported by the shared runner; real TokenSpeed route qualification is pending | Supported by the shared runner; real Specialized Engine route qualification is pending |
| Synthetic acceptance overlay | Supported: the typed synthetic_acceptance declaration (explicit value or digest-pinned golden acceptance-length curve) is overlaid onto the operator’s --speculative-config as synthetic rejection sampling, resolving the curve-form acceptance length against the draft count parsed from that configuration; real-route qualification is pending |
Supported: the declaration renders the SGLANG_SIMULATE_ACC_* environment over the operator’s --speculative-* flags (requiring a framework build that honors those variables); real-route qualification is pending |
Supported: the declaration renders the forced-acceptance environment with the documented off-by-one adjustment over the operator’s llm-api speculative_config; real-route qualification is pending |
—: the integration rejects a declared synthetic acceptance overlay with a typed planning error | —: the integration rejects a declared synthetic acceptance overlay with a typed planning error |
Auxiliary model weights (draft-model) |
Supported: the server-level auxiliary_models declaration resolves the referenced registry model’s per-machine weight locator onto every model-rank allocation, and the integration splices it as the model key of the operator’s --speculative-config JSON, which must exist and must not already set that key; real-route qualification is pending |
Supported: the declaration renders --speculative-draft-model-path alongside the operator’s speculative flags, requiring --speculative-algorithm as the splice target and rejecting an operator-spelled draft path; real-route qualification is pending |
Supported: the declaration splices the resolved locator into the speculative_model key of the operator’s llm-api speculative_config, which must exist and must not already set that key; real-route qualification is pending |
—: the integration rejects a declared auxiliary model with a typed planning error | —: the integration rejects a declared auxiliary model with a typed planning error |
| Incremental public streaming | Qualified for the built-in NIXL P/D pair on Qwen3 MoE SM120 TP1 over TCP: independent clients observed nonterminal SSE events before [DONE] on both completion routes and an AIPerf 0.12.0 Bench retained request-level timing artifacts; built-in Mooncake is Supported by deterministic public-boundary coverage, while vllm-router remains Unqualified |
Supported for the built-in pair by deterministic public-boundary coverage; sglang-router remains Unqualified |
Supported for the built-in pair by deterministic public-boundary coverage; trtllm-disaggregated remains Unqualified |
Unqualified for the maintained P/D pairing | Unsupported for the dense TP1 baseline’s exact tokenspeed-smg==1.7.0.post20260710 pairing: its worker emits stream chunks only after generation completes; the contract makes no low-latency streaming claim for any Engine |
| Prefix-cache reset between cases | Qualified through POST /reset_prefix_cache for the direct single Qwen3 MoE SM120 TP1 cache-start baseline and for the built-in NIXL P/D pairing on DeepSeek-V4-Flash SM120 (prefill TP2, decode TP2): the cold case’s Gateway reset returned 200 with deterministic fan-out to every prefill and decode engine, and the case completed with per-request cache-read evidence; the built-in Mooncake pairing shares the same fan-out path. vLLM declines a reset with HTTP 200 and body success false while KV blocks are still held, so a reset succeeds only when the engine’s body — or, behind the built-in P/D Router, every engine’s body — reports success true; a declined reset is recorded and retried after a growing delay until it succeeds or the case budget is spent (Supported; no qualified record has yet observed a decline). Releases before vLLM 0.26.0 answer with an empty body that reports no outcome, so their cold and primed starts fail at the reset; vllm-router remains Unqualified; Dynamo is Limited: per-target POST /engine/flush_cache on every worker’s control listener reached every worker and returned 200 with body status ok for routed single and for 1P1D on one and two machines on Qwen3-4B on NVIDIA RTX PRO 6000 Blackwell Server Edition (SM120) TP1 with ai-dynamo 1.5.0 and its vLLM 0.28.0, but ai-dynamo 1.5.0 discards vLLM’s reset result and reports status ok even when vLLM declined, so a recorded success proves the cache was reset only when no KV blocks were held at reset time (reported upstream as ai-dynamo/dynamo#15688) |
Qualified through POST /flush_cache for the direct single Qwen3 MoE SM120 TP1 cache-start baseline and the demonstrated P/D Gateway/P/D Router pairings below |
—; P/D enforces block reuse off at launch | Qualified for single and the maintained P/D pairing below through Gateway POST /flush_cache |
Qualified through Gateway POST /flush_cache on the two-replica MoE baseline below, with SMG reporting every Engine flushed |
| Framework profiling capture | Qualified: managed collection wraps each captured rank process tree with the replica entry as window control; engine trace (local, non-containerized placement only) renders the assigned per-server, per-replica trace directory into --profiler-config and verifies a dedicated-directory storage delta of at least one new trace artifact per device of the replica — Qualified on Qwen3.8-27B-FP8 SM120 TP2: capture start returned 200, both worker ranks flushed their traces, and coverage verified with expected_artifacts = 2. Note vLLM stop_profile blocks until worker traces finish serializing (observed >10 min for a TP2 27B capture); closing an engine-trace window runs inside the finalization deadline, whose engine-trace default allows for this, so check the effective capture_finalization_deadline_seconds in dry-run and raise it for larger captures. Under Dynamo, window control runs on each worker’s control listener through POST /engine/control/start_profile and POST /engine/control/stop_profile, which succeed only when the response body reports status ok; engine trace is Qualified for routed single on Qwen3-4B on NVIDIA RTX PRO 6000 Blackwell Server Edition (SM120) TP1 with ai-dynamo 1.5.0 and its vLLM 0.28.0: both windows returned status ok and the per-rank trace delta verified. Managed collection under Dynamo is Supported; its real-route qualification is pending |
Qualified for single and prefill/decode: every model-serving replica entry controls its captured process tree through POST /start_profile and POST /stop_profile; managed collection declares the CUDA_PROFILER body, while engine trace (local, non-containerized placement only) renders SGLANG_TORCH_PROFILER_DIR and declares the GPU activity — Qualified on Qwen3-30B-A3B-Instruct-2507 SM120 TP1: start and stop returned 200 and the per-rank trace delta verified |
— | — | Supported: captures the Engine process tree while the SMG Gateway controls the window through POST /start_profile and POST /stop_profile; managed capture defaults to CUDA and NVTX tracing, with OS runtime tracing available as an explicit typed override; no concrete Engine route is yet qualified |
For every supported framework profiling path, a positive AIPerf-native Bench warmup drains before InferLab opens the framework capture window; the window still closes at client completion. A failed warmup leaves its planned window unopened and fails the measurement.
Controlled cache-start qualifications rest on records in which cold and primed reset the server. Generated populations in a static matrix now separate reuse by namespace instead of resetting; that path is not yet qualified on real hardware for any backend.
The two named paths are endpoint-contract facts, not route qualification. Chat execution becomes Qualified only after an InferLab workflow produces a real record through the exact integration, route, topology, Gateway backend, P/D Router backend when present, and model being claimed. Optional upstream API extensions such as embeddings and batched prompt arrays remain outside this matrix. A pending upstream pull request or an unreleased dependency does not count as current support.
Except where a cell names its own qualifying records, the P/D Gateway, P/D Router, and fused-binding rows are Supported rather than Qualified because the existing route-level records predate protocol v7’s separate component evidence. The retained topology and pairing qualifications establish those concrete end-to-end workflows, but do not retroactively establish the new component-attribution shape. Requalification requires a protocol-v7 execution record that preserves both backend facts and their one fused process binding.
lm-eval Loglikelihood Routes
Section titled “lm-eval Loglikelihood Routes”Loglikelihood support is an observed property of the complete public serving
route, not a consequence of scalar completion support. The first four columns
below are bounded to the published 0.4.0 integration packages and the named
maintained DeepSeek-V4 SM120 baselines; they do not qualify the protocol-v7
0.6.0 candidates. The Specialized Engine column is bounded to the retained
exact 0.2.1 package. The qualification task was the built-in hellaswag
multiple-choice task with the model-directory Hugging Face tokenizer and text
requests. A worker-only probe cannot qualify a prefill/decode public endpoint.
| Public route or qualification boundary | vLLM (inferlab-integration-vllm==0.4.0) |
SGLang (inferlab-integration-sglang==0.4.0) |
TensorRT-LLM (inferlab-integration-tensorrt-llm==0.4.0) |
TokenSpeed (inferlab-integration-tokenspeed==0.4.0) |
Specialized Engine (inferlab-integration-specialized-engine==0.2.1) |
|---|---|---|---|---|---|
Direct single endpoint |
Qualified: DeepSeek-V4 SM120 TP2/EP2; the prompt-logprob probe passed tokenizer alignment and hellaswag completed with its selected metric gate. |
Unsupported for the probed baseline: DeepSeek-V4 SM120 TP2/EP2 returned a conforming response shape, but the public endpoint exposed 14 prompt positions for the tokenizer’s 13 tokens. | Unsupported for the probed baseline: DeepSeek-V4 SM120 TP2/EP2 returned a conforming response shape, but the public endpoint exposed one prompt position for the tokenizer’s 13 tokens. | Inconclusive for the probed baseline: DeepSeek-V4 SM120 TP2/EP2 returned HTTP 400 to the prompt-logprob probe, so the probe did not establish support or unsupported prompt scoring. | —; the integration requires a Gateway-backed single topology. |
Gateway-backed single endpoint |
— | — | — | — | —: the initial token worker contract rejects log-probability requests. |
| Prefill/decode public endpoint, aggregate | Unqualified: no complete route-level prompt-logprob record; worker behavior is not qualification evidence. | Unqualified: no complete route-level prompt-logprob record; worker behavior is not qualification evidence. | Unqualified: no complete route-level prompt-logprob record; worker behavior is not qualification evidence. | Unqualified: no complete route-level prompt-logprob record; worker behavior is not qualification evidence. | — |
| Built-in Gateway/P/D Router pair | Unqualified: 0.4.0 declares Mooncake and NIXL cases, but no public-endpoint record establishes a concrete placement or per-role shape. |
Unqualified for the maintained single-machine 1P1D TP2/EP2 Mooncake and NIXL pairings. | Unqualified for the maintained 1P1D TP2/EP2 NIXL path. | —; the maintained P/D path uses the integration-rendered TokenSpeed pair. | — |
| Integration-rendered Gateway/P/D Router pair | Unqualified for the maintained vllm-router Mooncake and NIXL paths. |
Unqualified for the maintained sglang-router Mooncake and NIXL pairings. |
Unqualified for the maintained trtllm-disaggregated NIXL path. |
Unqualified for the maintained smg Mooncake path. |
— |
These direct-route failures do not establish the behavior of another model, integration release, built-in frontend, Gateway revision, or P/D Router revision. Requalification must run the tokenizer-alignment probe and representative task through the exact public endpoint being claimed.
Parallelism
Section titled “Parallelism”The rows below describe which user-requested parallel dimensions the integration can lower. “Derived” means the effective kernel dimension is calculated from the declared outer world and the other accepted dimensions rather than configured as an independent public setting.
| Capability | vLLM | SGLang | TensorRT-LLM | TokenSpeed | Specialized Engine |
|---|---|---|---|---|---|
Declared devices |
Supported: fills omitted outer TP as devices / (PP x attention DP x prefill-role CP) and rejects a layout read as a different count |
Supported: fills omitted outer TP as devices / PP and rejects a layout read as a different count |
Supported: fills omitted outer TP as devices / PP and rejects a layout read as a different count |
Supported: fills omitted outer TP as devices and rejects a different outer TP |
Required: every role declares only devices, which places it and becomes its CUDA_VISIBLE_DEVICES; the device count equalled devices on the 8-device and two-replica MoE baselines below on protocol-v14 records |
Device-count reading of parallelism |
outer TP x PP x attention DP, times CP for a prefill_decode prefill role |
outer TP x PP | outer TP x PP | outer TP | components rejected: the Engine owns its layout |
| Outer tensor parallelism | Qualified for the retained baseline; protocol-v7 candidate Unqualified | Qualified for the retained baseline; protocol-v7 candidate Unqualified | Qualified for the retained baseline; protocol-v7 candidate Unqualified | Qualified for the retained baseline; protocol-v7 candidate Unqualified | —: rejected during planning; the layout is an Engine argument in extra_args |
| Outer pipeline parallelism | Supported | Supported | Supported | — | —: rejected during planning |
| Attention data parallelism | Supported | Supported | Limited: 1 or the outer TP size |
Supported | Engine argument in extra_args; readiness reads each Engine’s reported data_parallel_size from SMG’s /workers registry and Bench derives cache-owning targets from it; DP 8 and DP 4 were qualified on the 8-device and two-replica MoE baselines below on protocol-v14 records |
| Attention context parallelism | Supported: single and decode roles lower to decode context parallelism, and prefill_decode prefill roles to device-multiplying prefill context parallelism; model-level applicability (MLA-only prefill CP, GQA head divisibility) remains the framework’s launch-time verdict. Qualified: decode CP on Qwen3.8-27B-FP8 SM120 TP8 DCP2 completed the 8K/1K serving baseline. Engine verdicts observed on DeepSeek family: DCP is rejected at launch for sparse-indexer (DSA) models, and prefill CP on DeepSeek-V4-Flash fails at engine launch in an upstream MoE top-k kernel (is_padding size mismatch under PCP) — a framework defect, not a lowering gap |
Supported: single and prefill roles activate prefill CP (--enable-prefill-cp with a default zigzag strategy, and the whole flag group is verbatim-replaceable through the -- passthrough) and decode roles lower to decode CP, all without changing device counts; lowering verified through launch (engine reports ATTN_CP ranks). Qualified: DeepSeek-V4-Flash SM120 TP2 with declarative CP 2 plus a verbatim --enable-dsa-prefill-context-parallel reaches readiness with the engine deriving attn_cp_size=2 (interleave) and serves the 8K/1K bench. On SM120 the generic zigzag spelling remains blocked upstream (flashinfer ragged prefill crash) and fa4 lacks paged KV on SM120; non-DSA prefill CP qualification waits on the framework |
— | — | Engine argument in extra_args; the Engine rejects it before ready if unsupported |
| MoE expert parallelism | Qualified | Qualified | Qualified | Qualified | Engine argument in extra_args; EP equal to the device count was qualified on the 8-device and two-replica MoE baselines below on protocol-v14 records |
| MoE data parallelism | — | Supported with topology constraints | — | Supported | —: rejected during planning; any Engine-side setting is an Engine argument |
| Independent dense tensor parallelism | — | Supported | — | Qualified | The Engine’s own |
| Effective expert tensor parallelism | Derived | Derived | Derived | Derived; cannot be greater than 1 together with expert parallelism greater than 1 |
The Engine’s own |
Backend-specific constraints remain validated by the integration and are reported during planning. This table records the public capability boundary; it does not duplicate every arithmetic constraint enforced by each adapter.
Maintained Qualification Baselines
Section titled “Maintained Qualification Baselines”| Backend | Real-hardware baseline | Important boundary |
|---|---|---|
| vLLM | Source-built DeepSeek-V4 SM120 TP2/EP2 serving; real two-machine 1P1D vLLM Router serving with Mooncake and NIXL; source-built Qwen3 MoE SM120 TP1 direct serving for protocol-v7 Bench qualification | Multi-node replica lowering is supported but unqualified; the maintained cross-machine baseline is 1P1D. |
| SGLang | Source-built DeepSeek-V4 SM120 TP2/EP2 serving, pairing-specific single-machine 1P1D serving, and source-built Qwen3 MoE SM120 TP1 direct serving for protocol-v7 cache-start qualification | P/D qualification is pairing-specific below; TP4 is outside the maintained baseline. |
| TensorRT-LLM | Source-built DeepSeek-V4 SM120 TP2/EP2 serving and 1P1D NIXL serving with built-in and native routing | SM120 DeepSeek-V4 serving requires the source integration’s FlashInfer sparse-MLA path; the stock NGC image through 1.3.0rc21 is not sufficient. |
| TokenSpeed | Source-built DeepSeek-V4 SM120 TP2/EP2/dense-TP2 serving; single-machine 1P1D serving with TP2/EP2/dense-TP2 per role, native SMG routing, and Mooncake KV transfer | P/D qualification is limited to that concrete routing/transfer pairing; the source-built framework baseline includes its required kernel fixes. |
| Specialized Engine | A dense Qwen3-4B Engine on one NVIDIA RTX PRO 6000 Blackwell Server Edition (SM120), with one TP1 replica behind tokenspeed-smg==1.7.0.post20260710; an 8-device MoE Engine with speculative decoding on eight NVIDIA RTX PRO 6000 Blackwell Server Edition (SM120) GPUs, as one attention DP 8 × EP 8 replica, as two DP 4 × EP 4 replicas with GPU-local prefix caches and with a shared Mooncake KV store, and as a 1P1D prefill/decode pair of DP 4 × EP 4 per role with and without the KV store; and, under the protocol-v15 contract, an MoE Engine as one TP4 replica and as two TP4 replicas on one 8-device machine, and as one DEP16 replica spanning two 8-device machines |
Each baseline qualifies the contract for its Engine and shape, not other Engine implementations. Other layouts are Engine arguments that only the Engine validates, and no other layout is qualified; P/D beyond 1P1D is Supported but Unqualified; chat and log probabilities remain unqualified or unsupported as stated above. |
SGLang P/D Pairings
Section titled “SGLang P/D Pairings”The qualified entries use source-built DeepSeek-V4 on SM120 in a single-machine 1P1D topology with TP2/EP2 per role. Qualification is per pairing; Supported cells are implemented but have not been separately qualified on real hardware.
| Gateway/P/D Router backend pair | Mooncake | NIXL |
|---|---|---|
| Built-in Gateway/P/D Router pair | Qualified | Supported |
| SGLang Router | Supported | Qualified |
Maintenance Rules
Section titled “Maintenance Rules”Update this document in the same change as an integration when the change affects any matrix row or qualification statement. In particular:
- use Supported for deterministic implementation coverage and Qualified only after a real record proves the exact workflow and shape;
- name material limitations, including dependencies on downstream framework patches, instead of presenting them as general support;
- remove or downgrade capabilities when the integration stops exposing them;
- retain the underlying execution evidence internally, but cite it here only when a public qualification artifact exists;
- never expose unpublished internal identifiers, machine-local record paths, or private downstream revisions; and
- do not list pending upstream pull requests or future releases as current support.