Skip to content

Decision record · ADR

ADR-0015: Use operation-owned deadlines with separate cleanup grace

Status: accepted | Date: 2026-07-15

References: RFC-0009, RFC-0003, RFC-0004, RFC-0005, RFC-0006, ADR-0003, ADR-0004

InferLab has accumulated several correct local timeout mechanisms, but their composition has no common semantic model. Server readiness owns an elapsed-time deadline while individual probes use fixed I/O caps; container-removal polling can restart a full duration inside an outer deadline; profiler control can add connection time to its response deadline; and measurement definitions currently pass one timeout_seconds value to the Rust supervisor, Python runner, native engine, and individual endpoint requests. The attempted lm-eval outer-timeout padding exposed that local arithmetic cannot settle which layer owns the terminal outcome.

InferLab needs one way to reason about finite and deliberately unbounded runtime operations without turning every duration into a global setting or every workflow into a generic operation graph. The design must make an operator-visible budget compositional across nested attempts, keep Rust authoritative for lifecycle and measurement outcomes, and keep cleanup bounded without hiding cleanup time inside the business operation.

  • RFC-0003:C-RESOLUTION keeps one frozen resolved execution object as the authority for execution and records.
  • RFC-0003:C-RUNTIME-WORKFLOWS and RFC-0006:C-INTEGRATIONS keep timeout, interruption, process-group lifecycle, and cleanup authority in the Rust control plane.
  • RFC-0004:C-MEASUREMENTS exposes one timeout in each effective Eval or Bench definition and requires manual and closed-loop parity.
  • RFC-0004:C-WORKLOAD-PROFILING already proves that some operations must be explicitly unbounded while retaining non-time termination conditions.
  • RFC-0005:C-EVIDENCE requires truthful process and cleanup outcomes.
  • RFC-0009 defines the resulting observable time-bound semantics.
  • The repository design principles reject a universal workflow language and abstractions justified only by similar implementation structure.
  • Regression prevention must validate observable semantics. A source-pattern or constant blacklist would encode implementation spelling rather than the ownership invariant and is therefore not an acceptable substitute for a sound API and boundary tests.

We considered continuing local duration arithmetic, applying one global timeout value or policy object, introducing a generic timed-operation scheduler, enforcing allowed timeout spelling with a mechanical verification guard, and using one owner per concrete operation with a small private deadline capability shared only where nested budget consumption is required.

We will give each concrete runtime operation one time-budget owner and let nested work consume that owner’s remaining budget through a small private deadline capability because:

  1. One authority per fact: The frozen duration becomes a deadline once, at the operation’s declared start boundary, instead of being copied into nested supervisors.
  2. Composition without a workflow layer: Connections, probes, external-tool guards, and waits can cap themselves to remaining time while server, measurement, adapter, profiler, and cleanup domains retain their own lifecycle behavior.
  3. Truthful terminal semantics: The owner accepts success or failure before expiry or classifies timeout at expiry; downstream guards cannot extend or preempt that outcome.
  4. Separate cleanup accounting: TERM, KILL, container removal, profiler finalization, reaping, and verification use explicitly named cleanup grace after the business outcome and never become hidden additions to the operator budget.
  5. Semantic regression resistance: APIs should require an existing deadline where work is nested, and tests should observe budget start, expiry, outcome, and cleanup separation. We will not add a source-pattern, direct-Instant, duration-arithmetic, or constant-name verification guard for this decision.

The shared mechanism remains private and minimal: finite remaining-time queries plus an explicit unbounded form where the governing workflow permits it. It does not own retries, cancellation graphs, process lifecycle, or public configuration.

Operation owners keep their domain-specific attempt caps and cleanup policies, but nested calls receive the existing operation or cleanup deadline rather than the original duration. Polling cadence, progress timing, and SSH keepalive retain distinct names and types.

For external Eval and Bench clients, Rust remains the terminal authority. A runner may receive remaining-budget information to cap tokenizer, endpoint, and native operations and publish specific evidence, but it does not gain an independent whole-case budget. The exact start synchronization mechanism is an implementation detail to validate before migration and must not become an arbitrary task protocol.

  • A configured operation duration has one observable meaning regardless of nesting depth.
  • Sequential probes and native-engine calls cannot each restart the whole measurement budget.
  • Cleanup remains bounded without delaying or rewriting the business terminal outcome.
  • Explicit unbounded workflows no longer need fake maximum integers or arbitrary wall-clock limits at the InferLab authority boundary.
  • API shape and behavior tests target the ownership invariant instead of policing source spelling.
  • Existing helpers that accept bare durations cannot all migrate atomically. The mitigation is a staged migration by owning workflow while preserving the single semantic contract in RFC-0009.
  • External tools expose inconsistent timeout controls: some are per request, some cover process startup, and some cannot be disabled. The mitigation is to qualify each real native control as an attempt cap or subordinate guard and never treat it as the InferLab terminal authority.
  • A hard control-plane deadline can terminate a runner before it publishes its most specific inner failure. The mitigation is durable pre-effect and pre-native evidence plus an authoritative supervisor outcome, not extra business-budget padding.
  • Semantic boundary tests require controllable clocks or bounded fixture processes and are more deliberate than a source-pattern guard. That cost is accepted because those tests verify behavior rather than implementation syntax.
  • Operator-facing timeout field names and values do not need to multiply merely to implement this model.
  • Domain-specific cleanup grace values may remain different when their lifecycle evidence justifies different policies.
  • This decision does not add arbitrary timeouts to installation, builds, capture-armed readiness, or other deliberately unbounded workflows.

Use one owner per concrete operation and pass a small private deadline capability to nested work (accepted)

Section titled “Use one owner per concrete operation and pass a small private deadline capability to nested work (accepted)”
  • Pros: Makes one operation budget compositional without exposing a new public workflow concept, Preserves domain ownership for server, measurement, adapter, profiler, and cleanup behavior, Allows finite and deliberately unbounded operations to share precise remaining-budget semantics
  • Cons: Requires staged migration of existing duration-taking helpers and evidence

Continue fixing each timeout conflict with local offsets or grace constants (rejected)

Section titled “Continue fixing each timeout conflict with local offsets or grace constants (rejected)”
  • Pros: Small immediate patches
  • Cons: Copies the same duration into multiple authorities, Makes elapsed behavior depend on nesting and finalization timing
  • Rejected because: The lm-eval conflict demonstrates that additive padding hides rather than resolves terminal ownership.

Use one global timeout value or universal timeout-policy object for all runtime work (rejected)

Section titled “Use one global timeout value or universal timeout-policy object for all runtime work (rejected)”
  • Pros: Centralizes configuration and default values
  • Cons: Conflates operation budgets, attempt caps, cleanup grace, cadence, and transport liveness, Cannot represent the already-confirmed unbounded capture-readiness workflow truthfully
  • Rejected because: The workflows need common semantics, not one duration or one global policy owner.

Represent runtime execution as a generic timed operation graph or scheduler (rejected)

Section titled “Represent runtime execution as a generic timed operation graph or scheduler (rejected)”
  • Pros: Could centralize deadline propagation and cancellation
  • Cons: Introduces a workflow abstraction before concrete product workflows require one, Moves lifecycle ownership away from existing server, measurement, adapter, and profiler boundaries
  • Rejected because: A small remaining-budget capability solves the demonstrated composition problem without a planning or scheduling layer.

Prevent regressions with a source-pattern or timeout-constant verification guard (rejected)

Section titled “Prevent regressions with a source-pattern or timeout-constant verification guard (rejected)”
  • Pros: Can cheaply reject selected spellings in continuous verification
  • Cons: Cannot prove that a duration is not restarted through an allowed spelling, Couples governance to private implementation syntax and encourages compliance-oriented code
  • Rejected because: The ownership invariant must be enforced by API shape and observable boundary tests, not a mechanical blacklist.