Decision record · ADR
ADR-0064: Own server processes by an inherited environment marker
Status: accepted | Date: 2026-10-14
References: RFC-0003, RFC-0005
Context
Section titled “Context”Server cleanup owned one process group per launched process: it signals the group, verifies the group is empty, and then probes assigned devices, counting only memory of processes that no longer exist as residual. A managed Nsight Systems collection launches the serving command through nsys launch, whose launcher starts the target in a new session. When an interrupted startup outlived the termination grace, the group kill reached only the nsys processes; the vLLM tree, re-parented to init, kept four GPUs while the record claimed verified cleanup and four freed devices. Reproduced with Nsight Systems 2026.6: after SIGTERM, the grace, and SIGKILL to the launch group, a target that does not exit within the grace survives with its whole tree. The same exposure applies to any launch tool or framework that opens its own session or process group.
Decision
Section titled “Decision”Ownership of a server process is an inherited environment marker that holds a record identity — the server record and the process’s logical identifier in the resolved topology — rather than a process group or PID. Every descendant that keeps its environment inherits it through any session or group change, and the record identity cannot match a recycled process. Cleanup sweeps the processes carrying a process’s marker after its process-group termination, on every cleanup path including one whose group has already exited, because that is exactly when an escaped tree is otherwise unreachable. The device residual probe keeps its meaning: the sweep now proves the server’s own processes gone, so a live process is still not residual. RFC-0003:C-RUNTIME-WORKFLOWS owns the marker spelling, signal sequence, and verification; RFC-0005:C-EVIDENCE owns the evidence.
Consequences
Section titled “Consequences”Interrupted or stopped servers whose tree left the launch group are terminated, and a record no longer claims verified cleanup while a marked process lives. Each cleanup adds one scan of the launch machine’s process environments within the existing cleanup graces and deadline. A process that clears its environment or runs as another user stays outside the sweep and outside what the record can claim; containerized processes remain covered by container removal. The Nsight Systems session agent inherits the marker and is terminated by the sweep after profiler finalization has been attempted, so the sweep evidence, not the agent cleanup, records its termination. Record evidence changes under RFC-0005:C-EVIDENCE, and handles from earlier records keep group-only cleanup.
Alternatives Considered
Section titled “Alternatives Considered”Mark every launched server process with an inherited environment marker and sweep marked processes after the group (accepted)
Section titled “Mark every launched server process with an inherited environment marker and sweep marked processes after the group (accepted)”- Pros: Reaches the serving tree wherever a launch tool moves it, including after the launch group has already exited, Uses a record identity, so it cannot signal an unrelated process that recycled a PID
- Cons: Adds one process-environment scan to every cleanup, Cannot reach a process that clears its environment or runs as another user
Walk the process tree at cleanup time from the launch leader to the launcher’s child and signal its session as a second group (rejected)
Section titled “Walk the process tree at cleanup time from the launch leader to the launcher’s child and signal its session as a second group (rejected)”- Pros: Small change specific to the Nsight Systems launcher
- Cons: Fails when the leader has already exited, because the escaped tree is re-parented to init and the link is gone, Needs a tool-specific rule for every launcher or framework that opens its own session
Run every server process in a dedicated cgroup or systemd scope and kill the scope (rejected)
Section titled “Run every server process in a dedicated cgroup or systemd scope and kill the scope (rejected)”- Pros: Kernel-enforced membership that no process can leave
- Cons: Requires delegated cgroup or user systemd on every launch host, which shared SSH and Slurm hosts do not guarantee
Lengthen the termination grace so the launch tool can forward termination before SIGKILL (rejected)
Section titled “Lengthen the termination grace so the launch tool can forward termination before SIGKILL (rejected)”- Pros: No new mechanism
- Cons: A target that ignores or stalls on SIGTERM still escapes, and the record still claims verified cleanup
Make the launching InferLab a child subreaper so an escaped tree re-parents to it instead of init and can be walked (rejected)
Section titled “Make the launching InferLab a child subreaper so an escaped tree re-parents to it instead of init and can be walked (rejected)”- Pros: Kernel-maintained ancestry without a new identity
- Cons: A server outlives the command that started it: stop and recovery run later from the record, often from another invocation, and an SSH launch has no long-lived InferLab parent on the launch machine