Skip to content

Decision record · ADR

ADR-0004: Run built-in OpenAI smoke in the InferLab process

Status: accepted | Date: 2026-07-01

References: RFC-0004, ADR-0003, RFC-0004:C-MEASUREMENTS, RFC-0004:C-INFERLAB-TOOLCHAIN, RFC-0005:C-EVIDENCE, RFC-0004:C-WORKLOAD-PROFILING

During a design discussion of RFC-0004, the project found that ADR-0003 had been applied too broadly. That earlier decision correctly moved lm-eval and AIPerf into release-owned installed environments, but the implementation also routed the built-in OpenAI completions smoke through the Eval Python runner even though that probe uses no lm-eval capability. This ADR records the design rationale that informed the amended contract.

InferLab needs a minimal correctness and connectivity gate that ships with the core CLI while retaining lm-eval for dataset tasks and AIPerf for serving Bench. Applying the external-runner boundary to every Eval forces smoke-only users to install both measurement environments and obscures the distinction between a built-in protocol probe and an external evaluation engine.

ADR-0003 continues to govern the lm-eval and AIPerf environment topology, installation identity, and external runner boundaries. Its statements about the Eval runner boundary apply to lm-eval-backed definitions rather than the built-in smoke.

The discussion considered keeping all Eval measurements in the installed Python runtime, moving only the built-in smoke into the InferLab process, introducing a standalone Rust runner, and building a generic native evaluator framework.

We will execute only the built-in OpenAI completions smoke directly in the invoking InferLab process while retaining lm-eval and AIPerf as installed external engines because:

  1. Remove accidental installation cost: The smoke is one standard HTTP probe and gains no capability from the lm-eval environment.
  2. Reuse existing orchestration: Endpoint binding, profiling windows, gate classification, timeout, artifacts, and records already belong to the common Eval path.
  3. Avoid another distribution unit: A standalone Rust runner would preserve a subprocess shape at the cost of another binary and lifecycle for no present isolation benefit.
  4. Defer abstraction until evidence exists: One native probe does not justify task, dataset, scorer, registry, or plugin abstractions.

The Eval definition kind remains the authority for selecting the boundary specified by RFC-0004:C-MEASUREMENTS. OpenAI smoke uses the built-in path; lm-eval definitions retain the release-owned external runner.

Use one typed execution distinction between built-in smoke and external lm-eval. Keep common Eval orchestration shared across both paths. A future second native task should provide the evidence for any further extraction rather than inherit a predesigned evaluator framework.

  • Smoke-only recipes no longer require materializing the lm-eval and AIPerf environments.
  • The smallest correctness gate ships and versions with the core CLI.
  • Dataset evaluation and serving Bench retain their already qualified external engines.
  • The design adds one concrete execution distinction without committing to a general evaluator framework.
  • Eval orchestration handles both a built-in operation and an external client process. The mitigation is a closed typed execution distinction selected during planning rather than duplicated orchestration paths.
  • Built-in smoke has no client process or process logs. The mitigation is explicit not-applicable record evidence instead of fabricated empty process artifacts.
  • HTTP client behavior becomes part of the core binary dependency closure. The mitigation is to use a maintained dependency already present in the release workspace and qualify timeout, interruption, response parsing, and artifact behavior at the measurement boundary.
  • inferlab toolchain install continues to install both external runtime closures when explicitly invoked.
  • lm-eval definitions, AIPerf Bench definitions, workspace syntax, and recipe selection remain unchanged.
  • A future second native evaluation task may justify extracting shared evaluator components, but this decision does not pre-commit to their shape.

Execute only the built-in OpenAI smoke in the InferLab process (accepted)

Section titled “Execute only the built-in OpenAI smoke in the InferLab process (accepted)”
  • Pros: Removes installation cost from the smallest correctness gate, Reuses existing Eval planning, profiling, gate, and record orchestration, Keeps lm-eval and AIPerf behind their qualified external boundaries
  • Cons: Eval has both in-process and external execution paths

Keep all Eval measurements in the installed Python runtime (rejected)

Section titled “Keep all Eval measurements in the installed Python runtime (rejected)”
  • Pros: Retains one Eval subprocess path
  • Cons: Requires a large toolchain installation for a single standard HTTP request, Conflates the built-in probe with the lm-eval engine
  • Rejected because: The uniform subprocess shape does not justify an external dependency that contributes no smoke capability.

Ship a standalone Rust smoke runner (rejected)

Section titled “Ship a standalone Rust smoke runner (rejected)”
  • Pros: Preserves the typed request and result subprocess boundary
  • Cons: Adds a second binary, subprocess lifecycle, and packaging surface for a small HTTP probe
  • Rejected because: Process isolation and a separate executable provide no current user-visible value for this built-in operation.

Build a generic native evaluator framework (rejected)

Section titled “Build a generic native evaluator framework (rejected)”
  • Pros: Could host future datasets and scorers without Python
  • Cons: Introduces task, dataset, scorer, and extension abstractions before a second native task exists, Duplicates part of the lm-eval ecosystem
  • Rejected because: The current requirement is only a built-in smoke gate; broader abstractions must wait for concrete workloads.