Decision record · ADR
ADR-0039: Move engine-trace window close into the global finalization budget
Status: accepted | Date: 2026-08-21
References: RFC-0004, RFC-0003
Context
Section titled “Context”Real-hardware qualification of engine-trace capture on 2026-08-21 showed that vLLM’s window-closing /stop_profile latency carries no completion information and defeats the calibrated budgets in both directions. On a DeepSeek-V4-Flash TP2 server the stop response returned in about a second while the two worker traces (about 190/130 MB) appeared 43 minutes later; on a Qwen3.8-27B-FP8 TP2 server the stop call blocked past a 600-second control deadline and about 800 MB of per-rank traces landed roughly 11 minutes after dispatch. SGLang on a small TP1 shape answered stop in well under a second.
RFC-0004:C-WORKLOAD-PROFILING states that a successful window-closing control response is treated as artifact flush completion; the DeepSeek observation falsifies that rule. The clause’s defaults (60-second control actions, 300-second finalization, per RFC-0003:C-RESOLUTION) were calibrated for managed Nsight Systems collection, where stopping is a local process signal. Under engine trace, every real capture records a spurious window-closing control failure, and large-model captures fail the 300-second finalization budget although the evidence lands intact minutes later. The same clause already adjudicates window-closing budget expiry by coverage verification, and its rationale already states that a late stop response must not fail a capture whose evidence exists — the semantics had converged on coverage without the budget topology or the falsified sentence following.
Decision
Section titled “Decision”We will redefine engine-trace window closing as a dispatch into the single global finalization budget, with coverage verification as the sole completion verdict. Window opening keeps its per-action control budget and fail-fast semantics, because a lost start silently shifts every later range index off its planned window identity. For engine-trace targets, the close request is dispatched when the measured phase ends; a delivery failure (connection refused, engine process gone, or a prompt error status) fails the capture, while a slow or absent response is recorded as neutral flush-pending evidence. Stop-response consumption, flush waiting, and coverage verification consume the one capture finalization budget without restarting it, and its default becomes mechanism-aware: 300 seconds for managed collection and 3600 seconds for engine trace — the observed slowest flush ran 2580 seconds (43 minutes), so the default buys about 40 percent headroom over that observation while still bounding a wedged engine. The falsified rule equating a window-closing response with flush completion is removed, and window-closing failure adjudication narrows to delivery failure.
The obligations land in RFC-0004:C-WORKLOAD-PROFILING (window-close dispatch semantics, adjudication, and the removed flush-completion sentence) and RFC-0003:C-RESOLUTION (the mechanism-aware finalization default); this ADR records why dispatch-and-adjudicate prevailed over retuning the synchronous budgets.
Consequences
Section titled “Consequences”Records no longer carry spurious window-closing control failures for healthy engine-trace captures, and large-model captures succeed whenever the per-device artifacts actually land. The budget topology becomes one owner per phase — arm governs preparation, control governs window identity at open, finalization governs evidence landing — and engine-trace closing joins the existing globally computed arm/finalization accounting instead of drawing a fresh per-action control budget per target while engines flush concurrently. Managed collection is untouched: Nsight stop is a fast local signal and keeps its 60-second control action and 300-second finalization defaults.
The cost is worst-case latency when an engine accepts the stop but never flushes: the measurement hangs until finalization expiry rather than failing at the 60-second mark, which the 3600-second default, the operator override, and the preserved dispatch evidence bound. Operators also lose nothing they could act on: the stop response latency never indicated progress. The mechanism-aware default in RFC-0003:C-RESOLUTION adds one conditional to an otherwise flat default table, which is recorded as preferable to a single default that is wrong for one of the two mechanisms by two orders of magnitude.
Alternatives Considered
Section titled “Alternatives Considered”Dispatch engine-trace window close into the global finalization budget; coverage verification is the sole completion verdict; mechanism-aware finalization default of 3600 s for engine trace (accepted)
Section titled “Dispatch engine-trace window close into the global finalization budget; coverage verification is the sole completion verdict; mechanism-aware finalization default of 3600 s for engine trace (accepted)”- Pros: Matches the observed information content of the stop response: delivery is verifiable, response latency is not progress, Removes the spurious window-closing control failure from every real engine-trace record, Joins the existing globally computed arm/finalization accounting; one budget owner per phase, Managed collection keeps its calibrated fast-path semantics unchanged
- Cons: An engine that accepts the stop but never flushes costs the full finalization budget before the capture fails, The capture control deadline no longer bounds any engine-trace action after the window opens
Keep synchronous window closing under the per-action control deadline and only raise the engine-trace default budgets (rejected)
Section titled “Keep synchronous window closing under the per-action control deadline and only raise the engine-trace default budgets (rejected)”- Pros: Smallest semantic delta; no change to the window-control classification rules
- Cons: Any finite control default remains arbitrary against model-size-dependent serialization that already ranged from seconds to tens of minutes, Records keep a control-failure entry for what is normal engine behavior, teaching operators to ignore failure evidence, Per-action control budgets hand each target a fresh budget while engines flush concurrently, serializing the wait without gaining a verdict
- Rejected because: Retuning cannot fix a semantic mismatch: the stop response latency carries no completion information on the engines that motivated the mechanism, so bounding it precisely buys nothing.
Keep current semantics and defaults; document that operators must raise the deadlines for engine-trace runs (rejected)
Section titled “Keep current semantics and defaults; document that operators must raise the deadlines for engine-trace runs (rejected)”- Pros: No clause or code changes
- Cons: The shipped defaults are objectively unusable for engine trace on real models, so every operator rediscovers the same two failures, The falsified flush-completion rule stays in the normative text
- Rejected because: A default that fails on every qualified real-hardware run of the mechanism is a defect, not a documentation gap.
Make the engine-trace finalization wait unbounded, like capture-armed readiness (rejected)
Section titled “Make the engine-trace finalization wait unbounded, like capture-armed readiness (rejected)”- Pros: No default to calibrate; arbitrarily slow flushes always complete
- Cons: A wedged engine that never flushes hangs the measurement indefinitely; readiness can terminate on process death or interruption, while finalization’s only terminal signal is the artifacts that may never appear
- Rejected because: Capture-armed readiness tolerates an unbounded wait because the loop still dies with the process group; finalization has no equivalent external terminator, so a generous bounded default with an operator override keeps failure deterministic.