Decision record · ADR
ADR-0014: Use stacks, servers, and local rank placement as the execution API
Status: accepted | Date: 2026-07-13
References: RFC-0002, RFC-0003, RFC-0005, RFC-0006, RFC-0007, ADR-0005
Context
Section titled “Context”InferLab’s first public workspace API composes recipes from model, serve profile, source set, serving environment, workload suite, and ordered recipe cases. That shape proved the initial closed-loop workflows, but it makes an independently managed server selectable only through a recipe, gives framework stack identity several owners, and leaves local placement too coarse for heterogeneous multi-node prefill/decode deployments.
Problem Statement
Section titled “Problem Statement”The four framework integrations now expose the same operator needs: launch a named server without inventing a measurement recipe, reuse that server across recipes and manual Bench runs, select a stable server case, qualify an image against the same server intent, and bind nP/mD replicas whose prefill and decode roles use different parallelism, device counts, rank decompositions, and machines. The existing environment, source-set, serve-profile, recipe-case, and global-placement surfaces cannot express those workflows without duplicated authority or implicit defaults.
Constraints
Section titled “Constraints”- RFC-0002 keeps shareable source and dependency facts in the committed workspace and machine-private facts in local bindings.
- RFC-0003 keeps lifecycle, placement, readiness, cleanup, and public endpoint authority in the framework-neutral control plane.
- RFC-0006 assigns framework-specific lowering to independently released integration packages.
- RFC-0005 requires exact effective values and standalone durable evidence.
- ADR-0005 keeps runtime image production a named closed-loop workflow with recipe-referenced validation.
- The design must support canonical serve, prefill, decode, and zero-device router roles without introducing a universal deployment DAG or copying integration source into downstream workspaces.
Representative operator workflow
Section titled “Representative operator workflow”A committed workspace names a vllm stack, a model, a model-pd server with two prefill and three decode replicas, one tuned server case, and a qualification recipe that combines that server with one workload suite. The shared files contain no host names, device numbers, weight paths, or credentials.
On one cluster, .inferlab/local.toml names the machines and an explicit rank placement. One prefill replica spans two machines, decode uses a different rank and card shape, the router has one device-less rank, and the same model resolves to different weight paths on the participating machines. The operator runs inferlab serve start model-pd --case tuned --placement cluster or runs the named recipe with the same selectors.
InferLab plans logical per-role demand in the stack Pixi environment, validates the placement, resolves every rank-local locator, port, cache, and endpoint, renders all integration-owned processes, freezes one resolved object, and creates the durable record before checks and launch. Model ranks start in canonical order and the router starts last. A partial failure rolls back in reverse order; a successful direct start remains owned by its record until stop, while a recipe executes the suite and cleans up. The record preserves role, replica, rank, machine, locator source, command, routing, and cleanup evidence without consulting the configuration again.
Options Considered
Section titled “Options Considered”We considered retaining the recipe-first configuration and adding placement flags, making recipes own complete deployment cases, introducing a general deployment graph, delegating placement and lifecycle to integrations, and establishing stacks and independently launchable servers with machine-local rank placement.
Decision
Section titled “Decision”InferLab will use stacks, independently launchable servers, server-owned cases, and machine-local placement as its public execution model.
A stack is the single shareable authority for one framework integration, its Pixi environment, its source paths, its checks, and its image post-processing behavior. A server is the single shareable authority for one model-serving intent: stack, model, topology, canonical roles, common inherited settings, and named typed cases. A recipe composes one server with one workload suite; it does not redefine or filter server cases.
Machine-private placement remains outside the shared server. It binds each canonical role, replica, and rank to a machine, devices, ports, launch context, and the model locator available on that machine. The model-serving integration boundary is split into logical planning and concrete rendering: the integration plans role-specific behavior and whole-replica device count, Rust chooses rank placement and resolves local values, and the integration renders exactly one process for each supplied allocation. Routing is represented by closed ownership choices—direct, InferLab built-in, or integration native—instead of an arbitrary public endpoint requirement.
One immutable resolved execution object is frozen after configuration, adapter planning, placement, locator, port, cache, and network resolution. Dry-run output, execution, cleanup, and durable records use that same object. Records preserve the role → replica → rank hierarchy and keep runtime evidence keyed separately by process identity.
This choice is based on four demonstrated needs: direct server lifecycle independent of a recipe, reuse of the same serving intent by recipes and manual measurements, heterogeneous multi-node nP/mD placement, and exact evidence of effective per-rank execution. It deliberately does not introduce a universal deployment or workflow graph.
Consequences
Section titled “Consequences”Positive
Section titled “Positive”- Operators can start, inspect, stop, and reuse a named server without creating a measurement recipe.
- Direct serving, recipe runs, manual Bench runs, and image validation resolve the same server intent and case semantics.
- Per-role planning plus explicit rank placement can represent nP/mD deployments whose prefill and decode replicas use different parallelism, device counts, machines, and model-weight locations.
- Framework integrations retain framework-specific lowering while Rust retains one lifecycle, placement, cleanup, and evidence authority.
- Dry-run output and records describe the exact execution that was launched instead of reconstructing effective values from configuration.
Negative
Section titled “Negative”- The workspace schema, CLI, adapter protocol, resolved execution model, and record format change together and require a coordinated migration of every integration package and downstream workspace.
- Complex multi-node deployments require explicit rank bindings when an entire replica cannot fit on one machine; InferLab will reject ambiguity instead of inventing a split.
- Per-role settings and routing ownership require stricter semantic validation than the previous server-wide parallelism and endpoint model.
Neutral and follow-up effects
Section titled “Neutral and follow-up effects”- Integration packages remain independently released and downstream workspaces continue to own concrete framework closures and private local bindings.
- Shared server and image definitions remain portable because placement, credentials, machine names, model-weight locators, and runtime cache paths stay local.
- Image validation remains a closed-loop workflow, but it references a server and optional server case rather than a recipe-owned deployment case.
- Schema 1 and protocol 4 inputs will be rejected rather than partially interpreted; migration guidance and regenerated schema, models, fixtures, and records are part of the same governed change.
Alternatives Considered
Section titled “Alternatives Considered”Use stacks, independently launchable servers, server-owned cases, recipes that compose one server with one workload suite, and machine-local rank placement behind a two-stage integration protocol (accepted)
Section titled “Use stacks, independently launchable servers, server-owned cases, recipes that compose one server with one workload suite, and machine-local rank placement behind a two-stage integration protocol (accepted)”- Pros: One authority for framework stack, server behavior, and local allocation facts, Direct serving, recipes, images, and manual measurements reuse the same server resolution, Role-specific planning and explicit ranks express heterogeneous multi-node nP/mD deployments
- Cons: Requires one breaking workspace, protocol, record, CLI, and downstream-workspace migration
Retain environment, source-set, serve-profile, and recipe-case definitions and add placement flags to existing commands (rejected)
Section titled “Retain environment, source-set, serve-profile, and recipe-case definitions and add placement flags to existing commands (rejected)”- Pros: Smallest implementation delta
- Cons: An independently managed server remains hidden behind a recipe, Stack identity and case behavior keep multiple authorities
- Rejected because: It patches the missing placement selector without correcting the ownership problem demonstrated by direct serving, image validation, and backend reuse.
Make each recipe own the complete deployment and measurement matrix (rejected)
Section titled “Make each recipe own the complete deployment and measurement matrix (rejected)”- Pros: One named object starts every closed-loop experiment
- Cons: Servers cannot be shared independently across recipes or manual Bench runs, Image validation must duplicate or reach into recipe deployment state
- Rejected because: Recipe and server lifecycles have different reuse and ownership boundaries; combining them recreates the duplication the redesign removes.
Introduce a general deployment and workflow DAG for stacks, servers, images, and measurements (rejected)
Section titled “Introduce a general deployment and workflow DAG for stacks, servers, images, and measurements (rejected)”- Pros: Can express arbitrary future orchestration
- Cons: Adds planning concepts and public nouns without a demonstrated operator workflow
- Rejected because: The confirmed serve, recipe, Bench, and image workflows are fully expressible without a universal graph.
Let each framework integration choose machines, ranks, process lifecycle, and record structure (rejected)
Section titled “Let each framework integration choose machines, ranks, process lifecycle, and record structure (rejected)”- Pros: Framework code can make backend-specific allocation choices directly
- Cons: Placement, cleanup, and evidence semantics diverge across integrations, Machine-private facts cross the adapter boundary before a framework-neutral plan exists
- Rejected because: It violates the established control-plane boundary and prevents one reproducible lifecycle and evidence model across backends.