Skip to content

Decision record · ADR

ADR-0059: Key Eval pass thresholds by workspace model on the Eval definition

Status: accepted | Date: 2026-10-03

References: RFC-0004, RFC-0005, RFC-0003

An lm-eval definition declares one pass threshold, and a recipe selects only a server and a workload suite (RFC-0003:C-RECIPE-COMPOSITION). A workspace that gates several models on the same task needs a different threshold per model, because the expected score is a property of the task and the model together. Today it can only duplicate the Eval and its suite for every model, about ten repeated lines per model kept consistent by comments, or override the threshold on each command line with --set, which leaves the catalog without the value. A downstream workspace asked for a recipe-level gate threshold override instead.

An lm-eval definition may declare model_thresholds, a table from workspace model identifiers to pass thresholds; threshold remains required and is the default. When an Eval is planned against a server, the threshold keyed by that server’s model is the effective threshold, otherwise the default. Every key must name a declared model, and every value follows the threshold rules of the definition. Invocation overrides apply to the declared fields before the selection, so overriding threshold changes the default and overriding model_thresholds.<model> changes that model’s entry. The effective definition carries only the selected threshold, and the plan and record name the model whose entry selected it. A recipe still owns no measurement behavior.

A workspace states each model’s expected score once, beside the Eval it qualifies, and every recipe and image validation that runs the Eval against that model uses it. Records show the effective threshold and whether a model entry selected it. A misspelled model key fails workspace loading instead of silently falling back to the default. Thresholds that depend on more than the model, such as a server case, remain a separate future decision.

Key thresholds by workspace model on the Eval definition: model_thresholds maps model identifiers to thresholds, and the declared threshold remains the default (accepted)

Section titled “Key thresholds by workspace model on the Eval definition: model_thresholds maps model identifiers to thresholds, and the declared threshold remains the default (accepted)”
  • Pros: The threshold stays with the Eval that owns its meaning, keyed by the fact it depends on, Applies wherever the Eval runs against that model, including image validations, without repeating it per recipe
  • Cons: A threshold that also depends on the server case, such as quantization, is not expressible yet

Let a recipe override its suite gate’s threshold with a recipe field (rejected)

Section titled “Let a recipe override its suite gate’s threshold with a recipe field (rejected)”
  • Pros: Small change local to the recipe
  • Cons: Gives a recipe a second authority over measurement behavior, against the recipe composition contract, One scalar covers only the gate; the other Evals of the suite stay unkeyed

Duplicate the Eval and its suite for every model (rejected)

Section titled “Duplicate the Eval and its suite for every model (rejected)”
  • Pros: No InferLab change
  • Cons: Repetition grows with models times Evals and drifts without anything checking it