Decision record · ADR
ADR-0047: Serve and measure vision-language models through a built-in smoke image and governed image request sources
Status: accepted | Date: 2026-09-07
References: RFC-0003, RFC-0004, RFC-0005, RFC-0006, ADR-0017, ADR-0036, ADR-0042, ADR-0046
Context
Section titled “Context”Vision-language models — LLMs with an embedded vision tower — are a required serving and measurement target. Serving them needs no control-plane change: a model is an opaque locator, and a framework serve command consumes VLM weights exactly as text-model weights. The gaps are all in the verification and measurement loop: the built-in OpenAI smoke sends a plain-text completions request, so InferLab cannot demonstrate that a VLM’s vision path is alive; and every serving-Bench request source is text-only, so image-dominated prefill behavior cannot be characterized at all.
The pinned measurement tools already own the multimodal semantics: AIPerf 0.12.0 generates synthetic noise images of governed dimensions, indexes an operator-supplied image directory, and attaches image_url content parts on its chat endpoint; lm-eval 0.4.12’s chat-completions model classes accept image inputs for multimodal tasks. Gateway routing is already transparent to request members InferLab does not own (RFC-0003:C-OPENAI-WORKLOAD-ROUTES), so image content parts need no contract change there.
Constraints shaping the choice: InferLab must not open the AIPerf configuration language (ADR-0017) or become a second dataset authority for multimodal semantics (ADR-0036); the existing random-source selectors reject means, standard deviations, and unbounded ranges in favor of fixed values and inclusive-uniform bounds; protocol version 10 has shipped with product 0.14.0, so the extension rule of ADR-0046 forces a hard cut rather than in-place growth; and per ADR-0042, framework-specific spelling stays with the integrations. Multimodal Eval tasks, audio and video loads, and separate-process vision encoders are explicitly out of scope for this step.
Decision
Section titled “Decision”The built-in OpenAI smoke definition gains an optional vision declaration (RFC-0004:C-MEASUREMENTS). A vision smoke routes to the chat-completions path with one user message whose content carries the effective prompt text part plus one image_url part whose data URI embeds a fixed test image whose bytes the invoking InferLab release owns. The smoke remains an endpoint liveness check: success is the mirrored transport and response-shape rule over message.content, and image understanding quality is never asserted. The definition stays one kind — the vision form is a declared mode whose effective shape is preserved in evidence.
A random serving-Bench request source gains an optional images table (RFC-0004:C-BENCH-REQUEST-SOURCES): a positive per-request image count defaulting to one, and required fixed positive width and height pixel values. No distribution language is offered, mirroring the selector governance already applied to token lengths. The table MAY declare one source: omission resolves to AIPerf’s synthetic noise supply, while a declared source names a workspace-relative directory of operator-supplied images, a closed three-value sampling vocabulary defaulting to shuffle-cycle, and an optional expected_sha256 binding the directory content through a release-owned deterministic enumeration digest. A source declaring images MUST route to chat completions; image carriage on random_mixture, dataset, or replay sources and on audio or video inputs is rejected until a demonstrated workflow requires it. Declared input_tokens keep their text-token meaning; images are a per-request decoration attached by the measurement runtime at request time and are never frozen into the population.
The adapter protocol hard-cuts to version 11 to carry the vision smoke declaration and the effective image source configuration (RFC-0006:C-INTEGRATIONS). Workload record schema version 20 preserves the effective image configuration in request-source evidence (RFC-0005:C-BENCH-REQUEST-SOURCE-EVIDENCE): shape, count, source form, declared and observed directory digest, and sampling policy, without claiming a frozen image sequence.
Consequences
Section titled “Consequences”Positive: a VLM launch becomes self-verifying with zero additional operator configuration, and image-decorated load is expressible with the same fixed-shape governance as text token lengths; an operator image directory becomes a closed, digest-bound source without opening AIPerf’s dataset surface; the routing and serving contracts remain untouched.
Negative: the adapter protocol hard-cuts again — version mismatches fail closed, so the vision smoke and image decoration are entirely unavailable until the adapter SDK, framework integrations, and workspace pins move in lockstep and older integrations reject the new control plane. InferLab now owns one directory-content enumeration digest as a new invariant, and the fixed smoke image is a release-carried byte artifact that must stay stable within a release. Because images attach at request time, per-request image identity is not measurement evidence; a run that needs that audit level must wait for a demonstrated workflow. Noise images are incompressible, so their transport bytes systematically exceed natural images — operators who care about transfer fidelity must declare a real image source.
Neutral: framework VLM serve parameters remain escape-hatch territory; no first-class serving setting is added because InferLab owns no behavior that consumes them. The vision smoke image’s content carries no scoring semantics, and its bytes are only stable within one release — vision smoke evidence is not byte-comparable across releases. Multimodal Eval through lm-eval’s chat model classes, audio and video loads, and operator image-pool statistics remain available to later efforts without changing this decision.
Alternatives Considered
Section titled “Alternatives Considered”Built-in fixed smoke image and a vision mode on the existing OpenAI smoke kind, plus a governed images table on the random request source (accepted)
Section titled “Built-in fixed smoke image and a vision mode on the existing OpenAI smoke kind, plus a governed images table on the random request source (accepted)”- Pros: Zero-configuration VLM liveness: the smoke’s job is minimal setup, so the release owns the image bytes, One smoke kind with a declared mode keeps evidence on one shape; the mode is recorded, not a second definition vocabulary, Images as a
random-source decoration reuse the seeded population, cache-state, warmup, and evidence machinery unchanged, Fixed width/height and the closed sampling vocabulary extend the standing rejection of distribution languages to the image dimension - Cons: The release carries a binary image artifact whose bytes become part of smoke evidence semantics, Directory enumeration digest is a new owned invariant
Operator-declared smoke image path (rejected)
Section titled “Operator-declared smoke image path (rejected)”- Pros: No release-carried artifact; operators choose a meaningful image
- Cons: Every VLM workspace must declare and bind an image before the smallest liveness check runs, defeating the smoke’s minimal-configuration purpose, Image identity becomes machine-local state that evidence must resolve per machine
- Rejected because: The built-in smoke is the zero-configuration endpoint check; an operator-curated image belongs to deliberate measurements, which the Bench image source already covers.
A separate vision smoke definition kind (rejected)
Section titled “A separate vision smoke definition kind (rejected)”- Pros: Each definition form has exactly one route and one response shape
- Cons: Two kinds duplicate defaults, timeout governance, evidence identity, and synthetic-acceptance interplay for one conceptual check, The route difference is a consequence of the declared mode, not a separate measurement concept
- Rejected because: The smoke asserts endpoint liveness; the vision mode changes request shape, not the assertion, so one kind with recorded mode keeps the evidence authoritative.
A new multimodal request-source kind (rejected)
Section titled “A new multimodal request-source kind (rejected)”- Pros: Multimodal load gets an explicit top-level identity
- Cons: A second source kind would duplicate random population generation, selector governance, prefix geometry, and corpus composition under a new name, Images do not replace the request origin; they decorate each request, which is exactly the axis a source kind cannot express
- Rejected because: The request source answers where request content comes from; images answer what else rides on each request, so they attach to the existing random source.
Expose AIPerf’s full --image-* option surface, or open audio and video at the same time (rejected)
Section titled “Expose AIPerf’s full --image-* option surface, or open audio and video at the same time (rejected)”- Pros: One pass covers every multimodal knob AIPerf offers
- Cons: Distribution means and standard deviations contradict the standing selector governance that rejects them for token lengths, Audio and video have no demonstrated InferLab workflow, and their options (codecs, FFmpeg coupling, embedded audio tracks) drag in framework semantics InferLab does not own
- Rejected because: ADR-0017 keeps the AIPerf configuration language closed; each knob enters only with a demonstrated workflow, and only fixed shapes match the existing governance.
Freeze the per-request image sequence into the population (rejected)
Section titled “Freeze the per-request image sequence into the population (rejected)”- Pros: Per-request image identity would become auditable and replayable
- Cons: The noise supply is an unbounded runtime pool with no sequence to freeze, so two source forms would diverge in evidence semantics, Freezing image bytes or indices into the population mixes a runtime decoration into the text-population authority and re-opens dataset-materialization ownership that ADR-0036 delegates to the measurement tool
- Rejected because: Population freezing governs text content; the image decoration stays a recorded runtime policy, identical in status to the noise supply.