Skip to content

Decision record · ADR

ADR-0039: Move engine-trace window close into the global finalization budget

Status: accepted | Date: 2026-08-21

References: RFC-0004, RFC-0003

Real-hardware qualification of engine-trace capture on 2026-08-21 showed that vLLM’s window-closing /stop_profile latency carries no completion information and defeats the calibrated budgets in both directions. On a DeepSeek-V4-Flash TP2 server the stop response returned in about a second while the two worker traces (about 190/130 MB) appeared 43 minutes later; on a Qwen3.8-27B-FP8 TP2 server the stop call blocked past a 600-second control deadline and about 800 MB of per-rank traces landed roughly 11 minutes after dispatch. SGLang on a small TP1 shape answered stop in well under a second.

RFC-0004:C-WORKLOAD-PROFILING states that a successful window-closing control response is treated as artifact flush completion; the DeepSeek observation falsifies that rule. The clause’s defaults (60-second control actions, 300-second finalization, per RFC-0003:C-RESOLUTION) were calibrated for managed Nsight Systems collection, where stopping is a local process signal. Under engine trace, every real capture records a spurious window-closing control failure, and large-model captures fail the 300-second finalization budget although the evidence lands intact minutes later. The same clause already adjudicates window-closing budget expiry by coverage verification, and its rationale already states that a late stop response must not fail a capture whose evidence exists — the semantics had converged on coverage without the budget topology or the falsified sentence following.

We will redefine engine-trace window closing as a dispatch into the single global finalization budget, with coverage verification as the sole completion verdict. Window opening keeps its per-action control budget and fail-fast semantics, because a lost start silently shifts every later range index off its planned window identity. For engine-trace targets, the close request is dispatched when the measured phase ends; a delivery failure (connection refused, engine process gone, or a prompt error status) fails the capture, while a slow or absent response is recorded as neutral flush-pending evidence. Stop-response consumption, flush waiting, and coverage verification consume the one capture finalization budget without restarting it, and its default becomes mechanism-aware: 300 seconds for managed collection and 3600 seconds for engine trace — the observed slowest flush ran 2580 seconds (43 minutes), so the default buys about 40 percent headroom over that observation while still bounding a wedged engine. The falsified rule equating a window-closing response with flush completion is removed, and window-closing failure adjudication narrows to delivery failure.

The obligations land in RFC-0004:C-WORKLOAD-PROFILING (window-close dispatch semantics, adjudication, and the removed flush-completion sentence) and RFC-0003:C-RESOLUTION (the mechanism-aware finalization default); this ADR records why dispatch-and-adjudicate prevailed over retuning the synchronous budgets.

Records no longer carry spurious window-closing control failures for healthy engine-trace captures, and large-model captures succeed whenever the per-device artifacts actually land. The budget topology becomes one owner per phase — arm governs preparation, control governs window identity at open, finalization governs evidence landing — and engine-trace closing joins the existing globally computed arm/finalization accounting instead of drawing a fresh per-action control budget per target while engines flush concurrently. Managed collection is untouched: Nsight stop is a fast local signal and keeps its 60-second control action and 300-second finalization defaults.

The cost is worst-case latency when an engine accepts the stop but never flushes: the measurement hangs until finalization expiry rather than failing at the 60-second mark, which the 3600-second default, the operator override, and the preserved dispatch evidence bound. Operators also lose nothing they could act on: the stop response latency never indicated progress. The mechanism-aware default in RFC-0003:C-RESOLUTION adds one conditional to an otherwise flat default table, which is recorded as preferable to a single default that is wrong for one of the two mechanisms by two orders of magnitude.

Dispatch engine-trace window close into the global finalization budget; coverage verification is the sole completion verdict; mechanism-aware finalization default of 3600 s for engine trace (accepted)

Section titled “Dispatch engine-trace window close into the global finalization budget; coverage verification is the sole completion verdict; mechanism-aware finalization default of 3600 s for engine trace (accepted)”
  • Pros: Matches the observed information content of the stop response: delivery is verifiable, response latency is not progress, Removes the spurious window-closing control failure from every real engine-trace record, Joins the existing globally computed arm/finalization accounting; one budget owner per phase, Managed collection keeps its calibrated fast-path semantics unchanged
  • Cons: An engine that accepts the stop but never flushes costs the full finalization budget before the capture fails, The capture control deadline no longer bounds any engine-trace action after the window opens

Keep synchronous window closing under the per-action control deadline and only raise the engine-trace default budgets (rejected)

Section titled “Keep synchronous window closing under the per-action control deadline and only raise the engine-trace default budgets (rejected)”
  • Pros: Smallest semantic delta; no change to the window-control classification rules
  • Cons: Any finite control default remains arbitrary against model-size-dependent serialization that already ranged from seconds to tens of minutes, Records keep a control-failure entry for what is normal engine behavior, teaching operators to ignore failure evidence, Per-action control budgets hand each target a fresh budget while engines flush concurrently, serializing the wait without gaining a verdict
  • Rejected because: Retuning cannot fix a semantic mismatch: the stop response latency carries no completion information on the engines that motivated the mechanism, so bounding it precisely buys nothing.

Keep current semantics and defaults; document that operators must raise the deadlines for engine-trace runs (rejected)

Section titled “Keep current semantics and defaults; document that operators must raise the deadlines for engine-trace runs (rejected)”
  • Pros: No clause or code changes
  • Cons: The shipped defaults are objectively unusable for engine trace on real models, so every operator rediscovers the same two failures, The falsified flush-completion rule stays in the normative text
  • Rejected because: A default that fails on every qualified real-hardware run of the mechanism is a defect, not a documentation gap.

Make the engine-trace finalization wait unbounded, like capture-armed readiness (rejected)

Section titled “Make the engine-trace finalization wait unbounded, like capture-armed readiness (rejected)”
  • Pros: No default to calibrate; arbitrarily slow flushes always complete
  • Cons: A wedged engine that never flushes hangs the measurement indefinitely; readiness can terminate on process death or interruption, while finalization’s only terminal signal is the artifacts that may never appear
  • Rejected because: Capture-armed readiness tolerates an unbounded wait because the loop still dies with the process group; finalization has no equivalent external terminator, so a generous bounded default with an operator override keeps failure deterministic.