Decision record · ADR
ADR-0061: Rebuild the AIPerf integration build on upstream main and replay AgentX under the agentx preset
Status: accepted | Date: 2026-10-06
References: RFC-0004, ADR-0050, ADR-0044, ADR-0035
Context
Section titled “Context”ADR-0050 pinned an AIPerf integration build of v0.13.0 plus ai-dynamo/aiperf#1468, the heartbeat-watchdog teardown fix, and the then-unmerged AgentX harness port ai-dynamo/aiperf#1413. Two facts have since changed. #1413 merged into upstream main with review changes the build lacks, among them a fix for a barrier-retained subagent join that was reported as deferred and so pinned its parent’s lane during phase handoff. #1468 is still open, and its own regression test fails on upstream main: services stopped before the export are still reaped, so stock main still turns a long AgentX export into exit 1. No official release carries either change.
InferenceX switched its AgentX replay to the SemiAnalysisAI/aiperf agentx scenario preset. The preset copies every invariant of the inferencex-agentx-mvp scenario, so AIPerf’s native scenario-validity rules are the same for both; it differs in its name and in the defaults it supplies: a 3600-second default duration, seed 42, 1-second metric slices, and several transport and reporting settings, split between command-line defaults and environment defaults. Upstream AIPerf carries neither the preset nor the scenario-defaults mechanism it depends on. The command-line defaults are applied only when AIPerf converts command-line flags; InferLab drives AIPerf through a configuration file, where only the environment defaults and the scenario’s own duration and trajectory defaults apply. Selecting agentx therefore changes the recorded scenario identity and nothing a profile that declares every value does not already fix; that identity matters because operators compare InferLab AgentX exports with InferenceX submissions, which now record agentx. The InferenceX leaderboard matrix also defaults AgentX profiling to 3600 seconds, while InferLab’s profile defaulted to the 1800-second MVP scenario value, a default ADR-0044 deliberately kept and left open to revisit.
Decision
Section titled “Decision”The Bench toolchain pins a rebuilt AIPerf integration build: upstream main with the merged AgentX port, plus ai-dynamo/aiperf#1468, the SemiAnalysisAI scenario-defaults and agentx preset commits, and the security fixes the upstream 0.13.0 release branch carries but main lacks, adapted as needed for upstream’s test suite and published as a tagged commit with its own version label on the operator’s fork. The toolchain continues to read the build’s expected AIPerf version from its lock, as ADR-0050 decided. The AgentX release profile runs AIPerf’s agentx scenario and is the sole authority for every replay value: it declares each value the preset applies and passes it explicitly, whether or not the preset would also supply it on the configuration-file path, so the native run equals the preset value for value, the record carries each effective value, and where the profile and preset overlap the profile value is what runs. The preset contributes only the scenario identity. The values this adds or changes are the ones an operator observes: the default profiling duration becomes 3600 seconds, a Bench that omits seed resolves to the profile’s default seed of 42 rather than the ordinary Bench default, and metric slices apply to every case rather than only server-metric cases; the remaining preset values become profile fields too. This reverses the profiling-duration part of the ADR-0044 alternative that kept upstream-only parameters out of the profile; the tokenizer trust flag and client-side context-length filter stay out as ADR-0044 decided, since they are InferenceX script inputs rather than preset values, and the server-metrics default is unaffected because metric slices are not server metrics. AIPerf keeps owning AgentX replay semantics as ADR-0035 decided; the profile only fixes the values it runs with. As provenance for the preset values, the catalog binds the InferenceX revision that runs the preset and the SemiAnalysisAI/aiperf revision it checks out. The build is retired once an official AIPerf release carries #1468 and an agentx preset with these values. This decision supersedes ADR-0050.
Consequences
Section titled “Consequences”Positive: a default InferLab AgentX run carries the agentx scenario identity and matches the preset value for value, including the one-hour default duration and seed 42, so it is parameter-identical to a current InferenceX submission. It is not implementation-identical: InferLab runs the reviewed upstream port, whose subagent-join fix and other review changes are absent from the SemiAnalysisAI revision InferenceX checks out, so lane occupancy around phase handoff is a known residual source of divergence. The build keeps the teardown fix that upstream main still lacks.
Negative: release artifacts still depend on the operator’s personal GitHub fork, so installing the toolchain needs network access to it and its availability is outside the upstream project’s control, as under ADR-0050. The build now bases on unreleased upstream main rather than a tagged release, so its delta from any official release is identified by commit, and it carries SemiAnalysisAI changes upstream has not reviewed, including an exemption of the preset’s environment-defaults mapping from upstream’s numeric-bounds property test. Retirement now also depends on upstream adopting the scenario-defaults mechanism and preset; if it never does, InferLab stays on a fork or falls back to the upstream MVP scenario with the profile’s explicit values. Adding the preset values to the effective replay policy changes the measurement wire contract, so the control plane and the measurement integration must release together. Records produced before this change ran inferencex-agentx-mvp with an 1800-second default, seed 0 unless declared, and no metric slices without server metrics; the profile name is unchanged, so cross-release comparisons must use the recorded effective policy, not the profile name. A one-hour default doubles default case length and GPU occupancy, and a Bench that omits duration but declares a timeout sized for the old default now fails at its deadline; dry-run shows the resolved duration and timeout together, and operators must raise such timeouts.
Neutral: RFC-0004:C-BENCH-AGENTIC-TRACE-REPLAY already requires a closed release-qualified profile without fixing its values; it needs a one-sentence amendment so that omission of seed resolves to the selected profile’s default seed, parallel to its duration rule, and that amendment is a precondition for implementing this decision.
Alternatives Considered
Section titled “Alternatives Considered”Rebuild the integration build as upstream main 80662af plus #1468 and the SemiAnalysisAI scenario-defaults and agentx preset commits, run the agentx scenario, and declare every preset value in the profile (accepted)
Section titled “Rebuild the integration build as upstream main 80662af plus #1468 and the SemiAnalysisAI scenario-defaults and agentx preset commits, run the agentx scenario, and declare every preset value in the profile (accepted)”- Pros: Takes the reviewed AgentX port with its subagent join fix and keeps the teardown fix upstream main lacks, Runs carry the agentx label and every preset value, so default runs compare directly with current InferenceX submissions
- Cons: The base is unreleased upstream main and the build carries SemiAnalysisAI changes upstream has not reviewed
Rebuild on upstream main plus #1468 only and keep the inferencex-agentx-mvp scenario with the profile’s explicit values (rejected)
Section titled “Rebuild on upstream main plus #1468 only and keep the inferencex-agentx-mvp scenario with the profile’s explicit values (rejected)”- Pros: Carries no SemiAnalysisAI-only code
- Cons: Runs keep a scenario label that current InferenceX submissions no longer use, although the replay values could match
Pin the SemiAnalysisAI/aiperf revision that InferenceX checks out (rejected)
Section titled “Pin the SemiAnalysisAI/aiperf revision that InferenceX checks out (rejected)”- Pros: Byte-identical to what InferenceX runs
- Cons: It branches from pre-0.13 upstream and self-reports 0.12.0, lacks the teardown fix, and lacks the review changes of the merged port
Select the agentx scenario but leave its values to the preset (rejected)
Section titled “Select the agentx scenario but leave its values to the preset (rejected)”- Pros: The profile would not restate values the preset already owns
- Cons: The preset’s command-line defaults never reach the configuration-file path, so runs would silently use seed 0 and no metric slices while carrying the agentx label