Decision record · ADR
ADR-0050: Pin the Bench toolchain to an AIPerf integration build until upstream releases AgentX parity
Status: superseded | Date: 2026-09-26 Superseded by: ADR-0061
References: RFC-0004, ADR-0049, ADR-0044, ADR-0035, ADR-0031, ADR-0003
Context
Section titled “Context”ADR-0049 kept the Bench toolchain on official AIPerf releases. It restored an AgentX profile the official releases accept, at the cost of parity with InferenceX, whose scripts run the SemiAnalysisAI/aiperf fork with a per-lane cache-warmup budget and a per-trace idle-gap cap. Qualifying that decision exposed facts that change its cost balance.
AIPerf 0.13.0 regressed teardown. Its new heartbeat watchdog keeps judging services that the controller stops before exporting results, so any export longer than about 20 seconds makes a complete run exit 1 with RequiredServiceReaped. AgentX raw exports routinely take longer than that. The regression was reported upstream as ai-dynamo/aiperf#1467, with a standalone reproduction, and fixed in ai-dynamo/aiperf#1468. The same run fails on unpatched main and passes with the fix. Neither change is in a release, and 0.12.0 predates the watchdog.
ai-dynamo/aiperf#1413 ports the fork’s AgentX harness, including the per-lane budget and a runtime per-tree idle cap that the inferencex-agentx-mvp scenario admits. Its own changes apply to v0.13.0 with a single conflict, in a generated schema file, and the upstream test suite passes on the result. Current InferenceX main still pins a 10-request per-lane budget and a 300-second per-trace cap, so the ADR-0044 profile values describe live InferenceX behavior. The revision the catalog named is no longer reachable upstream. The operator chose to carry these upstream changes on a personal fork rather than wait for an AIPerf release.
Decision
Section titled “Decision”The Bench toolchain pins an AIPerf integration build: upstream v0.13.0 plus ai-dynamo/aiperf#1468 (the heartbeat-watchdog teardown fix) and #1413 (the AgentX harness port), carried on the operator’s public fork as a commit-pinned revision with its own version label. The AgentX release profile returns to the ADR-0044 values, the 10-request per-lane cache warmup and the 300-second per-trace idle-gap cap, and the catalog binds the reachable InferenceX revision whose runtime settings declare them. The embedded toolchain reads the expected AIPerf version of this git-sourced package from its lock, so the runner handshake, catalog evidence, and installed runtime agree on one label. The build is retired, and the toolchain returns to an official package-index release, once an upstream release carries both pull requests. This decision supersedes ADR-0049.
Consequences
Section titled “Consequences”Positive: AgentX cases run on the shipped toolchain with parameters identical to current InferenceX submissions. Long exports no longer turn complete runs into failures. The integration build is a public, commit-pinned revision whose delta from v0.13.0 is two upstream pull requests and a version label, so the change is auditable and easy to retire.
Negative: InferLab releases now depend on a personal GitHub fork. Installing the toolchain needs network access to that repository, and its availability and history are outside the upstream project’s control. #1413 is unmerged and may change in review, so parity follows the port rather than an upstream release, and a changed port needs another qualification. Rebuilding on each newer upstream release is manual work.
Neutral: RFC-0004 obligations are unchanged, because the toolchain clause requires a release-pinned engine, not a particular package index. Records identify the integration build through its version label and the toolchain lock digest.
Alternatives Considered
Section titled “Alternatives Considered”Pin an integration build of v0.13.0 plus ai-dynamo/aiperf#1468 and #1413 from the operator’s fork, restore the ADR-0044 profile, and retire the build once an official release carries both (accepted)
Section titled “Pin an integration build of v0.13.0 plus ai-dynamo/aiperf#1468 and #1413 from the operator’s fork, restore the ADR-0044 profile, and retire the build once an official release carries both (accepted)”- Pros: Restores InferenceX parity now and removes the teardown failure, The delta from an official release is two upstream pull requests, which apply with one generated-file conflict
- Cons: Release artifacts depend on a personal fork and an unmerged upstream port
Stay on official AIPerf 0.12.0 with the duration-mode AgentX profile until an upstream release carries the fixes (rejected)
Section titled “Stay on official AIPerf 0.12.0 with the duration-mode AgentX profile until an upstream release carries the fixes (rejected)”- Pros: Official distribution with no new teardown watchdog, and a previously real-qualified shape
- Cons: AgentX stays not parameter-identical to InferenceX until an upstream release, with no scheduled date
Stay on official AIPerf 0.13.0 and widen the heartbeat window through AIPerf environment settings (rejected)
Section titled “Stay on official AIPerf 0.13.0 and widen the heartbeat window through AIPerf environment settings (rejected)”- Pros: Keeps an official distribution
- Cons: Only widens the window, so a longer export still fails, and it delays detection of real service deaths; parity is still missing
Pin the SemiAnalysisAI/aiperf fork revision that InferenceX checks out (rejected)
Section titled “Pin the SemiAnalysisAI/aiperf fork revision that InferenceX checks out (rejected)”- Pros: Byte-identical to what InferenceX runs
- Cons: The fork branches from pre-0.13 upstream and self-reports 0.12.0, which collides with the official release; it also lacks the teardown fix