Skip to content

Decision record · ADR

ADR-0068: Declare serve devices as the placement authority

Status: accepted | Date: 2026-10-23

References: RFC-0003, RFC-0006, RFC-0005

RFC-0003:C-SERVE-PARALLELISM gives servers nine framework-neutral parallelism components and leaves their algebra to each integration, which derives the whole-replica device count that placement consumes. Integrations read the same declaration differently. SGLang, TensorRT-LLM, and TokenSpeed treat outer.tensor_parallel_size as the replica world that attention data parallelism divides, so outer.tensor_parallel_size = 8 with attention.data_parallel_size = 8 places 8 devices. vLLM treats it as per-data-parallel-rank tensor parallelism that data parallelism multiplies, so the same declaration places 64. Because the clause leaves the algebra to integrations, each reading is that integration’s own choice; no neutral rule picks one. RFC-0003:C-SERVE-CONTEXT-PARALLELISM adds a role-dependent exception that lets a prefill role’s context parallelism multiply devices.

Framework layouts keep getting finer and more per-component: prefill and decode context parallelism are different mechanisms, attention and experts take different data-parallel and expert-parallel splits, and engines such as a data-parallel × expert-parallel Specialized Engine combine them in one process. A neutral vocabulary would have to track all of that to stay meaningful.

The control plane itself uses the shape for two facts only: the whole-replica device count for placement, and the attention data-parallel rank count that priming fan-out and the built-in P/D proxies’ routing need. Splitting a replica across machines already comes from explicit replica placement.

Add an optional per-role devices declaration. When it is declared, it is the sole authority for the role’s whole-replica device count: integrations fill omitted components consistently with it, reject a declared shape that implies a different count under their own reading, and the control plane rejects a plan whose returned count differs. When it is absent, today’s derivation is unchanged.

The existing parallelism components stay, but no longer as a framework-neutral vocabulary: they become a mapping onto framework options whose reading each integration owns and documents, and are deprecated in favor of devices plus framework-owned layout settings. Removing them is deferred to a later decision, after typed integration settings cover the layouts operators actually use and the data-parallel rank count has another source (the cache-owning target set). Until then attention.data_parallel_size remains the control plane’s only source for the data-parallel rank count, so attention data parallelism stays declared through it rather than through framework settings or arguments.

This adds a protocol member and a record field and changes no existing declaration’s meaning.

Positive: an operator can state the device count once and get the same placement under every integration; a layout that disagrees with it fails before launch instead of silently placing a different count. No existing workspace changes meaning, and vLLM keeps its native reading. Framework-specific layouts no longer need a neutral equivalent before they can be used together with devices.

Negative: until removal, two ways to express device count coexist, and declarations without devices keep the cross-integration ambiguity. Consistency checks cover only layout settings an integration renders; layout passed through framework arguments is judged by the framework at launch. Attention data parallelism expressed only through framework arguments would leave priming fan-out and built-in P/D proxy routing at one rank without failing; operator guidance keeps it in the vocabulary until the cache-owning target set provides the count. The adapter protocol version and the server record schema change.

Canonicalize the vocabulary so outer is the replica world for every integration (rejected)

Section titled “Canonicalize the vocabulary so outer is the replica world for every integration (rejected)”
  • Pros: One declaration means one device count and decomposition everywhere; records compare across frameworks
  • Cons: Breaks vLLM’s reading of existing declarations; keeps InferLab owning a vocabulary that must track ever finer per-component layouts, and context-parallel mechanisms already resist one rank identity

Remove the parallelism vocabulary now; operators declare devices and layouts are framework settings (rejected)

Section titled “Remove the parallelism vocabulary now; operators declare devices and layouts are framework settings (rejected)”
  • Pros: One placement authority with no neutral algebra left to maintain
  • Cons: Breaks every workspace, the adapter protocol response, and records at once, before typed integration settings and the cache-owning target set exist to replace what the vocabulary still carries

Derive the device count only in each integration from framework-native layout settings (rejected)

Section titled “Derive the device count only in each integration from framework-native layout settings (rejected)”
  • Pros: No second declaration that can disagree with the layout
  • Cons: Every integration keeps world algebra, and layouts passed through framework arguments or defined by a Specialized Engine cannot be derived at all

Add an operator-declared devices count that wins when present and keep the vocabulary as deprecated option mapping (accepted)

Section titled “Add an operator-declared devices count that wins when present and keep the vocabulary as deprecated option mapping (accepted)”
  • Pros: Placement becomes framework-independent without breaking any declaration; leaves time to design the vocabulary’s removal
  • Cons: Two ways to express device count coexist until removal; consistency is checked only for layout settings an integration renders