Decision record · ADR
ADR-0004: Run built-in OpenAI smoke in the InferLab process
Status: accepted | Date: 2026-07-01
References: RFC-0004, ADR-0003, RFC-0004:C-MEASUREMENTS, RFC-0004:C-INFERLAB-TOOLCHAIN, RFC-0005:C-EVIDENCE, RFC-0004:C-WORKLOAD-PROFILING
Context
Section titled “Context”During a design discussion of RFC-0004, the project found that ADR-0003 had been applied too broadly. That earlier decision correctly moved lm-eval and AIPerf into release-owned installed environments, but the implementation also routed the built-in OpenAI completions smoke through the Eval Python runner even though that probe uses no lm-eval capability. This ADR records the design rationale that informed the amended contract.
Problem Statement
Section titled “Problem Statement”InferLab needs a minimal correctness and connectivity gate that ships with the core CLI while retaining lm-eval for dataset tasks and AIPerf for serving Bench. Applying the external-runner boundary to every Eval forces smoke-only users to install both measurement environments and obscures the distinction between a built-in protocol probe and an external evaluation engine.
Constraints
Section titled “Constraints”- RFC-0004:C-MEASUREMENTS keeps Eval planning, gates, profiling windows, lifecycle, and records in InferLab.
- RFC-0004:C-INFERLAB-TOOLCHAIN retains release-pinned lm-eval and AIPerf environments for measurements that use those engines.
- RFC-0005:C-EVIDENCE requires truthful execution and result evidence.
- RFC-0004:C-WORKLOAD-PROFILING applies the same capture-window semantics across measurement execution boundaries.
- InferLab remains an experiment harness rather than a general evaluation ecosystem.
ADR-0003 continues to govern the lm-eval and AIPerf environment topology, installation identity, and external runner boundaries. Its statements about the Eval runner boundary apply to lm-eval-backed definitions rather than the built-in smoke.
Options Considered
Section titled “Options Considered”The discussion considered keeping all Eval measurements in the installed Python runtime, moving only the built-in smoke into the InferLab process, introducing a standalone Rust runner, and building a generic native evaluator framework.
Decision
Section titled “Decision”We will execute only the built-in OpenAI completions smoke directly in the invoking InferLab process while retaining lm-eval and AIPerf as installed external engines because:
- Remove accidental installation cost: The smoke is one standard HTTP probe and gains no capability from the lm-eval environment.
- Reuse existing orchestration: Endpoint binding, profiling windows, gate classification, timeout, artifacts, and records already belong to the common Eval path.
- Avoid another distribution unit: A standalone Rust runner would preserve a subprocess shape at the cost of another binary and lifecycle for no present isolation benefit.
- Defer abstraction until evidence exists: One native probe does not justify task, dataset, scorer, registry, or plugin abstractions.
The Eval definition kind remains the authority for selecting the boundary specified by RFC-0004:C-MEASUREMENTS. OpenAI smoke uses the built-in path; lm-eval definitions retain the release-owned external runner.
Implementation Notes
Section titled “Implementation Notes”Use one typed execution distinction between built-in smoke and external lm-eval. Keep common Eval orchestration shared across both paths. A future second native task should provide the evidence for any further extraction rather than inherit a predesigned evaluator framework.
Consequences
Section titled “Consequences”Positive
Section titled “Positive”- Smoke-only recipes no longer require materializing the lm-eval and AIPerf environments.
- The smallest correctness gate ships and versions with the core CLI.
- Dataset evaluation and serving Bench retain their already qualified external engines.
- The design adds one concrete execution distinction without committing to a general evaluator framework.
Negative
Section titled “Negative”- Eval orchestration handles both a built-in operation and an external client process. The mitigation is a closed typed execution distinction selected during planning rather than duplicated orchestration paths.
- Built-in smoke has no client process or process logs. The mitigation is explicit not-applicable record evidence instead of fabricated empty process artifacts.
- HTTP client behavior becomes part of the core binary dependency closure. The mitigation is to use a maintained dependency already present in the release workspace and qualify timeout, interruption, response parsing, and artifact behavior at the measurement boundary.
Neutral
Section titled “Neutral”inferlab toolchain installcontinues to install both external runtime closures when explicitly invoked.- lm-eval definitions, AIPerf Bench definitions, workspace syntax, and recipe selection remain unchanged.
- A future second native evaluation task may justify extracting shared evaluator components, but this decision does not pre-commit to their shape.
Alternatives Considered
Section titled “Alternatives Considered”Execute only the built-in OpenAI smoke in the InferLab process (accepted)
Section titled “Execute only the built-in OpenAI smoke in the InferLab process (accepted)”- Pros: Removes installation cost from the smallest correctness gate, Reuses existing Eval planning, profiling, gate, and record orchestration, Keeps lm-eval and AIPerf behind their qualified external boundaries
- Cons: Eval has both in-process and external execution paths
Keep all Eval measurements in the installed Python runtime (rejected)
Section titled “Keep all Eval measurements in the installed Python runtime (rejected)”- Pros: Retains one Eval subprocess path
- Cons: Requires a large toolchain installation for a single standard HTTP request, Conflates the built-in probe with the lm-eval engine
- Rejected because: The uniform subprocess shape does not justify an external dependency that contributes no smoke capability.
Ship a standalone Rust smoke runner (rejected)
Section titled “Ship a standalone Rust smoke runner (rejected)”- Pros: Preserves the typed request and result subprocess boundary
- Cons: Adds a second binary, subprocess lifecycle, and packaging surface for a small HTTP probe
- Rejected because: Process isolation and a separate executable provide no current user-visible value for this built-in operation.
Build a generic native evaluator framework (rejected)
Section titled “Build a generic native evaluator framework (rejected)”- Pros: Could host future datasets and scorers without Python
- Cons: Introduces task, dataset, scorer, and extension abstractions before a second native task exists, Duplicates part of the lm-eval ecosystem
- Rejected because: The current requirement is only a built-in smoke gate; broader abstractions must wait for concrete workloads.