Skip to content

Decision record · ADR

ADR-0067: Judge environment completion waits by observed progress, not predicted deadlines

Status: accepted | Date: 2026-10-17

References: RFC-0009, RFC-0003

InferLab cleanup waits for the environment to finish actions it has already requested: a process group or a marked process exiting after SIGKILL, a container disappearing after docker rm -f, device memory freeing after the processes holding it are gone. Each wait was bounded by a fixed total deadline — 10 seconds after SIGKILL, 30 seconds for container removal, 30 seconds for the remote cleanup script that performs the same waits over SSH, 30 seconds for device memory to settle — and expiry recorded unverified cleanup.

A downstream image build’s DeepSeek-V4.1 validation recorded unverified cleanup on every run while its server held about 190 GB of pinned host memory. On one reproduction pinning 192 GiB (Docker 27.0.3, NVIDIA driver 595.58.03): after SIGKILL the process leader turns zombie within 0.3 seconds while one thread stays runnable and consumes CPU for the whole teardown; memory charged to the process stays almost flat for about 10 seconds, then falls linearly, and the process or container is gone after 20 to 22 seconds, with or without Docker. Docker 27.0.3’s Kill gives up after 12 seconds (a 10-second exit wait, a direct SIGKILL, then 2 seconds) with “did not receive an exit event”, although the container still disappears on its own later. A genuinely stuck exit — a thread in uninterruptible sleep on an unanswered FUSE request — consumed no CPU and released no memory for as long as it was observed.

That single footprint took about 0.1 seconds per pinned GiB. Teardown duration is expected to grow with footprint, and the flat first phase may grow with it; only one footprint was measured. A fixed deadline is therefore a prediction that some larger footprint will defeat.

Adopt the completion-wait model of RFC-0009:C-COMPLETION-WAITS for every cleanup wait on the environment: process-group and marked-process exit after SIGKILL (RFC-0003:C-RUNTIME-WORKFLOWS), container disappearance, and device-memory release. Remote cleanup is not a separate kind of wait: its SSH transport is a bounded attempt, and the waits it performs on the launch machine are completion waits observed there.

The stall window measures the absence of progress, not the duration of the work, so it does not scale with model size and needs no operator setting. The attention ceiling answers a different question — how long a workflow may withhold control before recording a cleanup as unverified — and its expiry leaves the cleanup retryable through the stop command, so ending too early costs a retry rather than evidence. The ceiling is what keeps an active but non-converging teardown from waiting forever; the stall window alone cannot, because activity is not convergence.

Positive: a slow but progressing teardown verifies whenever it finishes within the attention ceiling, which sits far above the measured 20 to 22 seconds; only footprints beyond the ceiling need a later stop. Downstream workflows no longer need to accept cleanup failures, and no size-dependent timeout is exposed to operators. A stuck teardown is reported after one stall window rather than after a guessed total, with evidence of which signals were flat. Docker’s and other tools’ internal limits stop deciding InferLab outcomes.

Negative: cleanup verification must read per-target signals — task CPU time, cgroup memory and CPU for containers, device memory attributable to the targets — locally and over SSH, which is more machinery than a timer. A teardown that consumes CPU without converging keeps the workflow waiting until the attention ceiling. Cleanup records change shape as RFC-0009:C-CLEANUP-GRACE now requires, which affects consumers of the old deadline fields.

Keep fixed total deadlines per cleanup stage (rejected)

Section titled “Keep fixed total deadlines per cleanup stage (rejected)”
  • Pros: Simplest; one timer per stage
  • Cons: Every value predicts teardown duration, which grows with server footprint; a 190 GB server already exceeds the 10-second post-kill grace

Expose the cleanup deadline as a typed per-server setting (rejected)

Section titled “Expose the cleanup deadline as a typed per-server setting (rejected)”
  • Pros: Operators of large servers can raise it once; follows the typed-setting rule of RFC-0009:C-TIME-CONTROL-OWNERSHIP for operator-dependent latency
  • Cons: Moves the duration prediction to the operator, who cannot make it either; still fails the first run that exceeds it

Judge completion waits by attributable progress with a fixed stall window and a fixed attention ceiling (accepted)

Section titled “Judge completion waits by attributable progress with a fixed stall window and a fixed attention ceiling (accepted)”
  • Pros: No size-dependent value; separates slow from stuck as measured; ending early costs a retry, not evidence
  • Cons: Needs per-target progress observation locally and over SSH; a non-converging but active teardown waits until the ceiling

Count only resource release as progress, without a ceiling (rejected)

Section titled “Count only resource release as progress, without a ceiling (rejected)”
  • Pros: Every observed progress moves the exit toward completion, so no livelock ceiling is needed
  • Cons: The measured teardown releases almost nothing for its first 10 seconds, and that phase likely grows with footprint, so the stall window would again scale with size, Differs from the accepted option by both its progress definition and the missing ceiling; the ceiling is what bounds a non-converging teardown

Count any activity as progress, without a ceiling (rejected)

Section titled “Count any activity as progress, without a ceiling (rejected)”
  • Pros: Never ends a legitimate teardown early
  • Cons: A task that consumes CPU without finishing keeps an unattended workflow waiting forever, losing cleanup terminability, Differs from the accepted option only by the missing ceiling

Derive the deadline from the observed footprint, such as pinned or charged memory times a measured slope (rejected)

Section titled “Derive the deadline from the observed footprint, such as pinned or charged memory times a measured slope (rejected)”
  • Pros: Scales with the server automatically without an operator setting
  • Cons: Rests on one measured slope on one host; driver, kernel, and memory kind change it, and the flat first phase is not modeled, so the derived deadline is still a prediction