Decision record · ADR
ADR-0060: Integrate Dynamo as a fused Gateway and P/D Router backend with InferLab-managed discovery
Status: accepted | Date: 2026-10-04
References: RFC-0003, RFC-0006, RFC-0004, ADR-0058
Context
Section titled “Context”Operators need three Dynamo workflows: comparing a Dynamo-served framework with the same framework served natively, running Dynamo disaggregated prefill/decode, and spreading either across machines. They did not ask for KV-aware routing across multiple aggregated replicas, so no new topology is in scope. Dynamo is not a frontend in front of unchanged Engines, as SMG is in ADR-0058: dynamo.frontend owns the OpenAI HTTP surface, tokenization, routing, and in prefill/decode mode an in-process prefill router, while every model worker is replaced by a framework wrapper (python -m dynamo.vllm and peers) that registers into Dynamo discovery and serves Dynamo’s request plane instead of HTTP. Selecting Dynamo therefore changes the Engine launch command as well as the frontend, which the TokenSpeed integration already does when SMG prefill/decode replaces its Engine with the SMG gRPC servicer. The ai-dynamo distribution pins exact framework versions only in its optional extras, and inspection of its 1.5.0 wheel found no framework version check at import, so a workspace can install it beside its own framework build and establish compatibility by running it.
Dynamo 1.5.0 was exercised with vLLM 0.28.0, aggregated and 1P1D over NIXL on one machine and 1P1D across two machines. Discovery defaults to etcd; its file backend watches a directory with inotify, which does not observe writes another host makes over NFS, so only etcd serves several machines, and conda-forge etcd 3.4.16 works. NATS is unnecessary because the request plane defaults to TCP and the event plane to ZMQ. The frontend’s /health returns 200 with no workers. A worker’s system-port /health reports ready only when endpoint health tracking is enabled, and then once its model has loaded. The frontend’s /health lists every registered instance with its role and its request-plane address, which is the address InferLab allocates. The served model can appear in /v1/models while a prefill worker is still unregistered, so neither the model list nor a worker’s own health proves the prefill/decode topology complete. Without a worker system port the frontend downloads tokenizer metadata from Hugging Face by served name into the user’s home directory, and it caches model metadata there by default; with a system port the worker serves that metadata, so the frontend machine needs no model files. The frontend has no route that clears worker caches; each vLLM worker’s system port serves a cache flush route, which emptied the prefix cache, and profile start and stop routes, which drove both a managed Nsight Systems capture and an engine torch trace. These routes answer HTTP 200 with an error status in the body on failure. Only Dynamo’s own reasoning parser splits reasoning content; vLLM’s reasoning-parser option is accepted by the worker and has no effect because the frontend post-processes. The worker parses its system port as a signed 16-bit integer, so ports above 32767 abort it, and leaving the frontend’s response-stream host unset selected a reachable interface across machines. Of 4636 downstream Bench records, 1643 (35%) use a cold or primed cache start, 53% of those that record a cache start.
Decision
Section titled “Decision”Dynamo is a fused Gateway and P/D Router backend named dynamo. A routed single server selects gateway_backend = "dynamo", and a prefill_decode server selects gateway_backend = pd_router_backend = "dynamo" over NIXL; dynamo.frontend realizes the Gateway, and in prefill/decode also the P/D Router, in the existing [gateway] and [gateway, pd_router] bindings. Mixed pairs, a direct single server with Dynamo workers, and Dynamo’s standalone router processes are rejected. The frontend’s routing mode is a typed, recorded setting.
The framework integration renders Dynamo workers, starting with vLLM, reusing its own engine-argument lowering; a shared package, inferlab-gateway-dynamo, owns Dynamo-specific lowering — the frontend command, discovery and transport configuration, worker wrapper options, and the readiness and action declarations — as inferlab-gateway-smg does for SMG. The Rust control plane learns nothing about Dynamo. InferLab pins neither ai-dynamo nor the framework; the workspace’s stack environment supplies both and etcd, and the installed versions are recorded.
Each Dynamo server receives one InferLab-managed etcd process as a new derived, zero-device discovery process with the closed binding [discovery], placed on the Gateway’s machine. It must report healthy before any model rank spawns and is cleaned up after every other process. The integration renders its command; the control plane allocates its client endpoint and an empty per-server data directory under the discovery machine’s runtime cache, since that machine may be remote. Workers and frontend use etcd discovery under a per-server namespace, the TCP request plane, and the ZMQ event plane. Every listener another process must reach is allocated on its machine’s network address; the frontend’s response-stream listener receives only an allocated port. The lowering rejects a system port above 32767 with an explanation while that Dynamo limit stands.
Readiness has three layers: discovery health; each rank-zero worker’s system-port health with endpoint health tracking enabled; and Gateway readiness, which requires every expected worker to appear in the frontend registry with its role and its allocated request-plane address and the served model to appear in the model list. The adapter protocol gains a framework-neutral registry-membership readiness probe for the Gateway condition. Prefix-cache reset becomes a per-target capability: each rank-zero worker declares its reset action, and the control plane invokes it on every model-serving replica and succeeds only when all succeed. Capture-window control uses each worker’s engine profile routes through the existing replica-entry binding. Because these routes report failure inside a 200 response, HTTP actions gain an optional JSON success predicate. The existing reasoning_parser setting renders as Dynamo’s reasoning parser, and vLLM’s own --reasoning-parser is rejected in the escape hatch under Dynamo. Stopping uses the existing cleanup unchanged.
Consequences
Section titled “Consequences”Operators can compare Dynamo-served and native vLLM under the same Bench definitions, including cold and primed cache starts, and run Dynamo prefill/decode on one or several machines with the usual records, cleanup, and profiling. Every Dynamo fact stays in Python packages that can follow Dynamo releases without a product change, and the discovery service’s lifecycle, command, logs, and cleanup are recorded like any other process. Binding requirements belong to amendments of RFC-0003:C-SERVE-TOPOLOGY (a second derived process role, discovery, which is not a frontend), RFC-0003:C-RUNTIME-WORKFLOWS (one process ready before model ranks spawn), a new Dynamo clause in RFC-0003, RFC-0006:C-INTEGRATIONS (the discovery binding, the registry-membership probe, the success predicate, and per-target reset, which advance the adapter protocol by one version that every integration package adopts together), and RFC-0004:C-BENCH-CACHE-STATE (reset coverage over every replica). Workspaces add ai-dynamo without extras and etcd to their stack environment and own the framework/Dynamo compatibility their records demonstrate. Dynamo’s metadata and download caches are redirected into each process’s per-run cache directory on its machine, so a run writes nothing under the user’s home directory. A Dynamo server needs more ports from the machine pool — request, system, and side-channel ports per worker, two for the frontend, and one for etcd — and its system ports must stay at or below 32767 until Dynamo fixes its parser. Each worker still opens ephemeral ports, listening on all interfaces, for its event publishers and engine internals that InferLab does not allocate. A primed start remains unavailable when more than one prefill replica holds cache, because one priming request warms only one prefill replica and no conditioning fan-out exists for Dynamo. Graceful deregistration is not awaited: cleanup may escalate to SIGKILL as it already does for native frameworks, which is safe because the per-server etcd is removed last, and because cleanup is judged by process-group absence, records may show a worker exit code of 0 or 1.
Alternatives Considered
Section titled “Alternatives Considered”Realize Gateway and P/D Router with dynamo.frontend, render workers in the framework integration through a shared inferlab-gateway-dynamo package, and run an InferLab-managed etcd discovery process (accepted)
Section titled “Realize Gateway and P/D Router with dynamo.frontend, render workers in the framework integration through a shared inferlab-gateway-dynamo package, and run an InferLab-managed etcd discovery process (accepted)”- Pros: Reuses the existing routed-single and fused prefill/decode bindings and the SMG precedent for a frontend choice that changes the Engine command, Framework engine-argument lowering stays in one place per framework while Dynamo lowering stays in one shared package, Discovery lifecycle, logs, and cleanup become recorded evidence, and one discovery mechanism serves one and several machines
- Cons: The topology gains a discovery process that must be ready before model ranks spawn, The adapter protocol advances for the readiness probe, success predicate, per-target reset, and discovery binding
Publish a standalone inferlab-integration-dynamo package that selects the framework internally (rejected)
Section titled “Publish a standalone inferlab-integration-dynamo package that selects the framework internally (rejected)”- Pros: One package owns everything Dynamo
- Cons: Framework engine-argument lowering must be copied or imported across integration packages, A framework’s native and Dynamo-served settings would be validated by different packages
Embed Dynamo lowering in each framework integration without a shared package (rejected)
Section titled “Embed Dynamo lowering in each framework integration without a shared package (rejected)”- Pros: No new package
- Cons: The frontend command, discovery, and transport lowering would be repeated per framework and change in several places per Dynamo release
Use Dynamo file discovery in a run-owned directory (rejected)
Section titled “Use Dynamo file discovery in a run-owned directory (rejected)”- Pros: No discovery process to launch, ready, or clean up
- Cons: The file backend watches with inotify, which misses registrations written by another host over NFS, so it cannot serve multiple machines, Supporting multiple machines would still need etcd, leaving two discovery mechanisms
Require an operator-provided etcd endpoint in local bindings (rejected)
Section titled “Require an operator-provided etcd endpoint in local bindings (rejected)”- Pros: No topology change
- Cons: The discovery service’s state and lifecycle fall outside InferLab’s evidence, and operators must run a standing service, Runs sharing one etcd depend on namespace hygiene that InferLab cannot verify
Treat the frontend /health response, the model list, or a per-role instance count as Gateway readiness (rejected)
Section titled “Treat the frontend /health response, the model list, or a per-role instance count as Gateway readiness (rejected)”- Pros: No new readiness probe in the protocol
- Cons: The frontend /health answers 200 before any worker registers, A model list or a count cannot name the missing replica or detect a worker from another server
Leave prefix-cache reset unsupported for Dynamo (rejected)
Section titled “Leave prefix-cache reset unsupported for Dynamo (rejected)”- Pros: No per-target reset capability or success predicate in the protocol
- Cons: Cold and primed Bench cases, about 35% of downstream Bench records, could not run on Dynamo, Dynamo-versus-native comparisons under a controlled cache start would be impossible
Add a separate dynamo_reasoning_parser setting (rejected)
Section titled “Add a separate dynamo_reasoning_parser setting (rejected)”- Pros: The setting name states which parser vocabulary applies
- Cons: Two settings would express one operator intent, and the inert vLLM parser option would remain selectable
Place an SMG or built-in InferLab Gateway in front of dynamo.frontend (rejected)
Section titled “Place an SMG or built-in InferLab Gateway in front of dynamo.frontend (rejected)”- Pros: Keeps one Gateway implementation across native and Dynamo-served comparisons
- Cons: Two routers would sit in series, and the Dynamo frontend’s routing policy and prefill router would be hidden behind another Gateway’s recorded behavior, Dynamo workers serve only Dynamo’s request plane, so the outer Gateway could still reach them only through dynamo.frontend