Skip to content

Decision record · ADR

ADR-0044: Align AgentX profile defaults with InferenceX upstream semantics

Status: accepted | Date: 2026-08-30

References: RFC-0004, ADR-0031, ADR-0035

InferLab’s release-qualified AgentX profile is meant to reproduce the SemiAnalysis InferenceX replay so that InferLab runs are comparable with InferenceX submissions. A parameter-level audit against the pinned upstream (benchmark_lib.sh at the cataloged revision, plus a live 256k qualification run in a downstream workspace) found three drifts. The profile leaves the per-trace idle-gap cap unset while upstream pins 300 seconds, so InferLab replays a sparser timeline than every comparable submission. The profile declares cache-pressure warmup in AIPerf’s duration mode (600 seconds) while upstream uses the mutually exclusive count mode (10 requests per lane). And InferLab layers an extra acceptance gate on top of the native outcome: any warmup error record fails the case, although upstream leaves warmup aborts to AIPerf’s native outcome and otherwise lets the native failure threshold and submission validity decide.

The extra gate is not a theoretical divergence. A faithful replay of a recorded one-token turn on a reasoning model legitimately returns empty visible content; upstream tolerates this within its ten-percent failure threshold, while InferLab failed an otherwise valid qualification case with zero profiling errors. RFC-0004:C-BENCH-AGENTIC-TRACE-REPLAY already forbids recomputing a second acceptance threshold, so the gate is in tension with the governing contract it implements. The phase-barrier decision in ADR-0031 fails closed on a natively invalid warmup outcome to protect capture authority; it does not justify a stricter post-hoc acceptance rule for ordinary runs.

The decision drivers are comparability and restraint: default runs must be parameter-identical to upstream submissions, deviations must be deliberate and recorded, and no new operator-facing knob is justified without a demonstrated workflow that needs it.

Correction (RFC-0004 v0.41.0): this record originally stated that upstream aborts only on snapshot-warmup failure. The release-pinned AIPerf, like the revision InferenceX runs, aborts before profiling on any failed or cancelled root-stream warmup request, whether a snapshot or a cache-pressure request, and a descendant warmup failure does not abort by itself; a real AgentX run aborted on one reset cache-pressure request. Removing InferLab’s own warmup gate, the decision recorded here, is unaffected.

Make the release-qualified AgentX profile parameter-identical to the pinned InferenceX upstream in the three places where it had drifted, and change nothing else. The profile declares the upstream per-trace idle-gap cap of 300 seconds; it is orthogonal to the existing global idle-gap cap of 10 seconds, which upstream shares and which stays. The profile declares cache-pressure warmup in AIPerf’s count mode with the upstream budget of 10 requests per lane, replacing the duration-mode declaration; the two modes are mutually exclusive in AIPerf, and the count mode is what every comparable submission uses. The zero-warmup-error acceptance gate is removed: AIPerf’s native submission validity, failure threshold, and context-overflow decisions are once again the sole acceptance authority, exactly as RFC-0004:C-BENCH-AGENTIC-TRACE-REPLAY states. Removing the gate also extends ADR-0035’s delegation of AgentX semantics to AIPerf onto the acceptance axis: InferLab records native outcomes, it does not re-adjudicate them. Native warmup aborts still apply: AIPerf aborts the case before profiling on a failed or cancelled root-stream warmup request, behavior shared with upstream. Warmup error outcomes remain recorded as warmup evidence, separate from profiling metrics.

No new operator-facing configuration is introduced. The existing controls — concurrency, duration, seed, timeout, server-metric selection, and the synthetic-acceptance declaration — remain the complete manual surface, and the replay policy stays closed inside the release-qualified profile. Parameters that were considered and deliberately left alone (client-side context-length filter, tokenizer trust flag, server-metrics default, profiling-duration default) can be revisited when a concrete workflow demonstrates the need.

Positive: default AgentX runs are parameter-identical to InferenceX submissions, so InferLab numbers can be compared with published results without a deviation appendix. Faithful replays of recorded one-token turns stop false-failing on reasoning models. The acceptance authority returns to the single native threshold the RFC already mandates, and the profile change is three small edits with no new operator surface to document, validate, or support.

Negative: the existing qualified AgentX baseline was produced under duration-mode warmup and the strict gate, so it is not parameter-identical to future runs; because records carry effective replay policy values, the difference is visible but must be called out when comparing across the change. InferLab no longer fails a case on warmup errors by itself — a run whose warmup errors are all descendant failures can pass if the native threshold and submission validity pass, while a failed root-stream warmup request still aborts natively before profiling; warmup error counts remain in the record as first-class evidence.

Neutral: RFC-0004:C-BENCH-AGENTIC-TRACE-REPLAY needs a small amendment so its text admits a profile-declared per-trace idle-gap cap and the count-mode cache-pressure warmup form; that amendment is a precondition for implementing this decision. The AIPerf pin, the phase barrier, and all adapter boundaries are untouched.

Align the three profile-level drifts to upstream values and remove the extra warmup acceptance gate, adding no operator-facing knobs (accepted)

Section titled “Align the three profile-level drifts to upstream values and remove the extra warmup acceptance gate, adding no operator-facing knobs (accepted)”
  • Pros: Default runs become parameter-identical with every InferenceX submission: same per-trace idle-gap cap, same count-mode cache-pressure warmup, same acceptance authority, Removing the gate restores the RFC rule that AIPerf’s failure-rate and submission-validity decisions are the only acceptance authority, eliminating false failures on faithful one-token-turn replays, Zero new configuration surface: the existing operator controls (concurrency, duration, seed, timeout, server metrics, synthetic acceptance) remain the complete manual surface
  • Cons: The previously qualified AgentX baseline ran under duration-mode warmup and the strict gate, so historical records differ in parameters from future ones; records carry effective values, so the difference stays auditable, Descendant warmup failures no longer fail a case by themselves; mitigation is that warmup error counts remain recorded evidence and a failed root-stream warmup request still aborts natively

Keep the strict zero-warmup-error gate as the default with an operator opt-out (rejected)

Section titled “Keep the strict zero-warmup-error gate as the default with an operator opt-out (rejected)”
  • Pros: Preserves the strongest guarantee that profiling begins from an undegraded warmup state
  • Cons: Constitutes a second acceptance threshold, which the governing clause already forbids, False-fails faithful replays of recorded one-token turns on reasoning models, a population upstream explicitly tolerates, Adds a knob whose only demonstrated effect so far is one spurious qualification failure

Open the replay policy to workspace-declared overrides (idle-gap cap, warmup mode and value, warmup handoff policy) (rejected)

Section titled “Open the replay policy to workspace-declared overrides (idle-gap cap, warmup mode and value, warmup handoff policy) (rejected)”
  • Pros: Lets a workspace reproduce a hypothetical future upstream variant without waiting for a release
  • Cons: Breaks the closed-profile property that makes two InferLab runs mutually comparable; each workspace could silently drift from the qualified policy, No demonstrated workflow needs per-run variation of these parameters; the knobs would be speculative generality

Also align the remaining upstream-only parameters (client-side context-length filter, tokenizer trust flag, server metrics always on, 3600-second duration default) (rejected)

Section titled “Also align the remaining upstream-only parameters (client-side context-length filter, tokenizer trust flag, server metrics always on, 3600-second duration default) (rejected)”
  • Pros: Maximizes surface-level parameter parity with the upstream command line
  • Cons: None of these has a failing workflow: the 256k corpus already drops oversized requests for the qualified context window, the pinned tokenizer works without the flag, server metrics are already an operator switch, and upstream treats duration as a per-run workflow input rather than a load property