Skip to content

Decision record · ADR

ADR-0073: Extend the Specialized Engine contract to replicas that span machines

Status: accepted | Date: 2026-11-06

References: ADR-0071, RFC-0002, RFC-0003

A Specialized Engine serves expert-parallel layouts whose expert span crosses machines: dep16 over two 8-device machines now, dp2tp4ep8 on two machines, and larger layouts on a rack-scale fabric later. Its developers’ rule is that every serving performance number comes from an InferLab Bench record, so these layouts need InferLab to manage one replica across several machines (downstream request, 2026-11-06).

One deployment of that Engine over H machines is H processes of the same build, one per machine, started together. Host 0 supervises, admits members on a rendezvous address, and is the only endpoint; members connect to that address with their host index and serve no endpoint. Every process takes --host-count; each uses its machine’s devices through device visibility and reads its machine’s local checkpoint copy; host 0 refuses a member whose options, build, or checkpoint identity differ. A member that has not joined within the Engine’s bound fails startup, and a failure of any process fails every host.

ADR-0071 limits the Specialized Engine contract to allocation facts and requires each replica to be one rank process that owns its devices. InferLab already manages replicas spanning machines for other integrations: a local binding may place one replica as an explicit list of rank bindings, each with its machine and devices, and RFC-0002:C-LOCAL-PLACEMENT forbids machine-pool allocation from splitting a replica silently. Each rank is launched on its machine, over SSH when remote, from that machine’s workspace and locked environment; readiness distinguishes rank zero from the other ranks, each process is recorded with its command, logs, and exit, and cleanup reaches every machine, including survivors of an interrupted run. A port the integration names as a primary port is allocated only to rank zero of a multi-rank replica, and the render input carries every rank’s allocation; that is how SGLang’s other ranks learn rank zero’s master port.

A Specialized Engine serve replica may span machines. A replica placed through an explicit multi-rank binding runs one rank process per rank binding, in rank order; rank zero is host 0, and the host index is the rank index. The binding decides the host count and each host’s devices; machine-pool allocation still never splits a replica. Rank bindings normally name distinct machines, but two ranks on one machine with disjoint devices are not rejected. A replica placed on one machine is unchanged.

Contract arguments. The host count, host index, and rendezvous address are allocation facts, so they become contract arguments under ADR-0071. The rendezvous port is a primary port, allocated only to rank zero from its machine’s port pool.

  • Rank zero receives every contract argument of a one-machine Engine, plus --host-count with the replica’s rank count and --rendezvous-listen with its rendezvous address.
  • Every other rank receives --model with its own machine’s model locator, --served-model-name, --host-count, --host-index with its rank index, --rendezvous with rank zero’s rendezvous address, and, with a KV tier, --kv-tier-backend and --kv-tier-store with its own machine’s tier location. It receives no --listen; the endpoint the placement allocates to it is recorded but not rendered.
  • Every rank receives its own devices through device visibility and the role’s Engine arguments, which are therefore identical on every host. No layout option is rendered.

Engine obligations. Each host’s Engine checks that its layout’s world equals its visible device count times the host count and exits before SERVING otherwise. Host 0 answers for the whole replica: it reports SERVING only after every member has joined and can admit requests; its data_parallel_size is the replica’s count across all hosts; its FlushCache clears every host’s prefix-cache and tier entries; and with a tier, its reported tier state, access, and publication bound cover every host.

Control plane. Readiness, routing, and observation address rank zero only: the Gateway targets rank zero’s listen address, the registry readiness and the observed data-parallel size are rank zero’s, and the serve-state probe reads rank zero. The other ranks are ready while their processes run. Launch, records, and cleanup treat every rank as the process it is, under the existing multi-process rules, including reverse-order cleanup.

Rejected before launch. A replica whose ranks hold different device counts, naming each rank’s machine and device count, when its allocation is rendered; a prefill or decode replica placed across machines; and a server that requests profiling with a replica placed across machines, since the Gateway’s profiling window reaches only host 0 and managed collection would wrap hosts whose Engine is never told to profile.

Device leases and any other host-local launch policy stay in the workspace: an engine_command wrapper on the stack environment’s PATH applies them on every host.

  • that Engine’s cross-machine layouts can produce Bench records through the managed path; the Bench definition and its metrics are unchanged, since traffic reaches rank zero exactly as for a one-machine replica.
  • A multi-host server requires an explicit multi-rank binding in the local bindings; it cannot come from a machine pool. Rank zero’s declared host and port pool must be reachable from every member host, as for SGLang’s multi-node replicas.
  • RFC-0003:C-SPECIALIZED-ENGINE is amended: one process per rank instead of per replica; the contract arguments --host-count, --host-index, --rendezvous-listen, and --rendezvous and the member argument set; the Engine obligations to validate its world against device visibility times host count and to answer for the whole replica from host 0; and the before-launch rejections of unequal ranks, multi-host prefill/decode replicas, and multi-host profiling.
  • The adapter protocol needs no new member: the rendezvous port is a primary port allocated only to rank zero, as SGLang’s master port already is, and the render input carries every rank’s allocation. The implementation confirms this against the real render before claiming it.
  • A member’s allocated endpoint port is reserved and recorded but never bound.
  • Cleanup follows the existing reverse spawn order, so members are terminated before host 0, each with the termination grace before SIGKILL. An Engine whose members do not exit cleanly on termination still has its processes killed and verified gone.
  • After the server is ready, InferLab does not watch for a process exit; a member’s exit fails the deployment through the Engine’s own fail-all, and InferLab cleans every host at stop. Watching ready servers is a gap for every multi-process server and is tracked separately.
  • Multi-host prefill/decode and multi-host profiling stay unsupported until a workflow needs them.
  • Qualification needs two exclusive machines and a real multi-host Bench record before the backend support matrix may claim the capability.

Run each replica as one rank process per machine of an explicit multi-rank binding, and render the host count, host index, and rendezvous address as contract arguments (accepted)

Section titled “Run each replica as one rank process per machine of an explicit multi-rank binding, and render the host count, host index, and rendezvous address as contract arguments (accepted)”
  • Pros: Reuses the placement, launch, cleanup, readiness, and evidence mechanisms multi-machine replicas already have, Every new argument is an allocation fact, as ADR-0071 requires
  • Cons: Every multi-host server needs an explicit local binding per host

Declare the host count on the serve role or case (rejected)

Section titled “Declare the host count on the serve role or case (rejected)”
  • Pros: Machine-pool placement could stay automatic
  • Cons: The host count is a fact of where the replica runs, not of the portable intent: the same 16-device layout runs on one 16-device machine or two 8-device ones
  • Rejected because: It moves a placement fact into portable intent

Let machine-pool allocation split a replica across machines (rejected)

Section titled “Let machine-pool allocation split a replica across machines (rejected)”
  • Pros: No explicit rank bindings would be needed
  • Cons: RFC-0002:C-LOCAL-PLACEMENT forbids silently splitting a replica, because the split decides the Engine’s world
  • Rejected because: The split is an operator decision the Engine’s layout depends on; the existing explicit-binding rule already expresses it

Render each host’s device count as –tensor-parallel-size (rejected)

Section titled “Render each host’s device count as –tensor-parallel-size (rejected)”
  • Pros: Matches the request’s current command line
  • Cons: Re-introduces a layout option ADR-0071 removed; the device visibility already carries the count
  • Rejected because: ADR-0071 renders no layout option; each host’s device count is its device visibility

Add a per-machine launch prefix or pre-launch and post-exit hooks for device leases (rejected)

Section titled “Add a per-machine launch prefix or pre-launch and post-exit hooks for device leases (rejected)”
  • Pros: Every device process could run under a site lease tool without Engine-side wrapping
  • Cons: Puts one workspace’s device policy into the control plane and adds a general command-wrapping surface
  • Rejected because: Lease tooling is workspace policy; an engine_command wrapper on the stack environment’s PATH applies it on every host without an InferLab surface

Let the Engine’s own multi-host launcher start every host from host 0 (rejected)

Section titled “Let the Engine’s own multi-host launcher start every host from host 0 (rejected)”
  • Pros: No InferLab change; the Engine’s launcher already starts hosts together
  • Cons: InferLab would place, observe, record, and clean up only host 0; other hosts’ processes, logs, and devices would be invisible to records and cleanup
  • Rejected because: Lifecycle, placement, and cleanup authority belong to the control plane, and records must cover every process