Decision record · ADR
ADR-0037: Let generative Eval declare prompt rendering authority
Status: accepted | Date: 2026-08-07
References: RFC-0004, RFC-0005, ADR-0016, ADR-0028, ADR-0029
Context
Section titled “Context”ADR-0016 derived every lm-eval request form from the resolved task output type, so a generate_until task always used the chat-completions path with lm-eval structured chat construction. Operators evaluating gsm8k and comparable few-shot generative tasks observe large exact-match losses on that route, because the task scoring filters expect the demonstration continuation format that chat rendering replaces. A base model without a usable chat template cannot run a generative task at all. That regression is already visible in this repository: the release-qualification gate threshold in the committed workspace fixture was calibrated while lm-eval traffic still used the completions route, and it became unmeetable once ADR-0016 moved generative tasks to chat rendering. It was never recalibrated, so the route change persisted as a standing unmet gate rather than as a reported measurement failure. ADR-0028 and ADR-0029 meanwhile gave serving Bench an explicit prompt-authority table whose omission resolves to flat, so the product already treats prompt rendering authority as a declared workload property on the Bench side of the measurement surface while the Eval side keeps a derived route. The release-pinned lm-eval 0.4.12 bounds the available combinations: LocalChatCompletion.loglikelihood raises NotImplementedError, and client-side template rendering is reachable only with tokenized_requests=true, which changes transport to token identifiers and activates the client default 2048-token context truncation that InferLab currently leaves inactive.
Decision
Section titled “Decision”We will let an lm-eval definition whose resolved task output type is generate_until declare prompt rendering authority as flat or server_chat, reusing the serving-Bench prompt vocabulary. flat uses the declared completions path with the release-pinned local-completions client and no chat construction. server_chat keeps the current chat-completions path, local-chat-completions client, and structured chat construction. An omitted table will resolve to flat, matching the serving-Bench default and running the resolved task in the form its own demonstrations and scoring filters were authored for, so server_chat becomes the declared deviation rather than the inherited baseline. Because the authority still moves a metric that a definition compares against its threshold, records will separate a declared authority from a defaulted one so a reported metric always carries the rendering that produced it. Every other resolved output type keeps its derived completions route and declares nothing, because the pinned client offers no alternative there. InferLab will not own a per-task default table and will make no claim about which rendering authority a task should use. RFC-0004 owns the declaration domain and validation contract, and RFC-0005 owns the evidence.
Consequences
Section titled “Consequences”Existing generative Eval definitions keep resolving without edits, but their reported metric changes at the release boundary because the resolved authority moves from server chat to flat. That shift is a correction toward the form each task was authored for rather than a regression, so it belongs in release notes and in the declared-versus-defaulted record field rather than in a required definition field; a threshold calibrated before the chat route becomes meetable again with no edit, while a threshold calibrated against the chat-rendered numbers must be recalibrated. gsm8k and comparable few-shot generative tasks can reproduce their established harness numbers, and a model without a usable server-side chat template becomes evaluable on generative tasks. Two runs of the same task under different rendering authorities are not comparable, so records must carry the resolved authority and its declared-versus-effective provenance rather than presenting the metric alone. Server-side template controls such as chat_template and chat_template_kwargs remain legal only under server_chat and are rejected under flat, matching the existing serving-Bench rule. Because the route stays derived from the declared authority, no independent endpoint selector appears in the definition surface. Client-rendered generative Eval prompts remain unavailable, so an operator who wants a locally applied chat template on the completions route has no path; reconsidering it requires deciding token-identifier transport and an explicit context-length bound first.
Alternatives Considered
Section titled “Alternatives Considered”Declare flat or server_chat on generative Eval and resolve an omitted table to flat (accepted)
Section titled “Declare flat or server_chat on generative Eval and resolve an omitted table to flat (accepted)”- Pros: Reproduces established harness numbers without removing the chat route, Runs a resolved task in the form its own filters were authored for unless an operator asks otherwise, Reuses the serving-Bench prompt vocabulary and its omission default
- Cons: Existing generative Eval definitions change their reported metric once at the release boundary
Keep the route derived from output type (rejected)
Section titled “Keep the route derived from output type (rejected)”- Pros: One authority for request form and no new definition surface
- Cons: Generative tasks cannot reproduce their established scoring format, A model without a usable chat template cannot run a generative task
- Rejected because: The derived route silently changes a gated metric and excludes a supported model class.
Require the declaration on every generative Eval instead of defaulting it (rejected)
Section titled “Require the declaration on every generative Eval instead of defaulting it (rejected)”- Pros: No existing definition changes its reported metric without an edit, The rendering authority is always visible in the recipe
- Cons: Treats
flatandserver_chatas equal choices whenflatis the form the task already assumes, Adds a line to every generative definition to guard against one release-boundary migration - Rejected because: The baseline does not need to be restated; only the deviation from it does, and the migration belongs in release notes.
Default the omitted authority from an InferLab-owned per-task table (rejected)
Section titled “Default the omitted authority from an InferLab-owned per-task table (rejected)”- Pros: Operators do not restate a well-known task convention in every recipe
- Cons: Makes InferLab a second authority on task semantics that lm-eval owns, Encodes only the task half of a task-and-model choice, so it is wrong for reasoning models that need server-side template controls, A versioned table makes the default a moving reproducibility input with no recipe diff
- Rejected because: The correct authority depends on the model as well as the task, and a shipped table would decide a gated metric outside the recipe.
Also offer the serving-Bench rendered_chat authority (rejected)
Section titled “Also offer the serving-Bench rendered_chat authority (rejected)”- Pros: Would complete the vocabulary shared with serving Bench
- Cons: The pinned lm-eval client reaches local rendering only with token-identifier transport, That transport activates a default 2048-token context truncation that silently shortens few-shot prompts
- Rejected because: Local rendering is not separable from two unrelated transport changes in the pinned client, so it needs its own evidence and decision.