Evaluating a language model requires a decision about what to sample. Researchers may have hundreds of tasks, with a model that can pass one attempt and fail another. Each output consumes time or money. The same limited budget can cover more task types or measure variability by repeating a smaller set. A point estimate alone hides the uncertainty created by that choice. A paper revised on November 18 proposes a way to use repeat outcomes to tighten confidence intervals while preserving a fixed statistical guarantee. Its result is an evaluation-design method with a specific evidence base, not a new leaderboard of model quality. Cost-efficient LLM evaluation paper

The proposal starts from a practical situation. Suppose an organisation has a coding test with 880 tasks and plans five attempts at each. That creates 4,400 outcomes. It cannot afford every run, so it must decide which 1,100 to observe. A simple strategy samples 1,100 task-run pairs uniformly. The framework discussed in the paper first covers every task once, then spends the remaining 220 outputs repeating a random subset. If repeated outcomes agree within a task, those audits provide evidence about unobserved runs and the full planned score.

Abstract grid of repeated task runs with a highlighted limited sample

A model score sits beside a question of precision#

Evaluation estimates a quantity over a particular set of tasks, runs and settings. The authors call that pre-specified collection a fixed cohort. It might include a specific model version, prompt, decoding parameters, set of benchmark questions and number of attempts per question. The target in their coding example is the average pass or fail result over all 4,400 planned task-run pairs. Naming that population matters because a confidence interval only has meaning once the quantity being estimated is fixed.

Researchers often focus on the estimated average. If a model passes 72 percent of sampled tasks, the number seems easy to compare with another version. Yet a difference of one or two percentage points can matter when a product decision depends on a threshold. A team may have to decide whether a model clears a release criterion, whether another testing phase is justified or whether an apparent improvement warrants cost. An interval expresses the uncertainty around the estimate and can show whether plausible values fall on both sides of that decision line.

A narrow interval can help resolve a comparison at the same confidence level. That phrase contains two constraints. The interval should keep its stated coverage guarantee, and the evaluation should spend the same amount. Reducing interval width by relaxing the confidence level would alter what the interval promises. Narrowing it by running more tests would add cost. The paper asks how sampling can use the available outputs more efficiently while preserving both conditions.

This question is especially relevant for stochastic models and agents. A task can succeed on one run and fail on another because of sampling, tool calls, code execution or a long reasoning path. Evaluation therefore has two axes. It can sample across tasks to understand breadth, and it can repeat tasks to see variation. A budget that goes entirely to one axis leaves uncertainty on the other. The amount of each kind of uncertainty depends on the task set and the repetition pattern.

The target also needs a declared scoring rule. The paper studies a binary setting where each run eventually records a decisive success or failure. In an agent benchmark, a run may include a sequence of model responses or tool actions. The cost unit in the mathematical guarantee is an output or episode, and the selected run must be budgeted up to its maximum length. That allows the method to state a hard cap. It does not automatically measure API tokens, elapsed time or dollar cost, which vary with response length and infrastructure.

The fixed cohort gives a comparison a stable frame. It does not mean every possible task or future sample is represented. A score can describe the planned tasks, model snapshot and generation settings that define that cohort. If a product team changes the prompts halfway through, it has changed the measurement conditions. A confidence interval computed for the original grid will not silently account for that change. The evaluation can be redesigned around the new settings and a fresh cohort declared. This discipline makes it easier to tell a model improvement from an effect created by a revised task list, a different number of attempts or a different version of the run harness. The interval is useful because the question under measurement has been made explicit.

Repetition reveals variation inside a task#

One observation per task can estimate an average when tasks receive equal weight. It provides little direct information about how much other runs on those same tasks might vary. If the observed outcome for a question is a pass, the evaluator cannot infer from that one result whether the next four planned runs would also pass. A confidence interval must account for possible unseen outcomes, including cohorts where some tasks have high within-task variation.

That issue remains even if the sampled mean happens to equal the mean across every planned outcome. Estimation and precision are separate. A sample can land exactly on the target by luck while uncertainty about the unsampled outcomes stays large. The interval has to work across every fixed cohort covered by the guarantee, not only the cohort whose repeated outcomes all happen to agree.

Repeated runs act as audits. The framework samples two distinct outcomes from selected tasks and counts how often they disagree. A low disagreement count suggests outcomes within tasks may be stable. That information can tighten a bound on the unobserved variation and, in turn, reduce interval width. A high count signals that repetitions matter, which supports keeping the uncertainty larger. The method turns observed agreement or disagreement into an input to the interval calculation.

The framework does not assume a probability model for how a particular task generates its outcomes. Instead, it randomises which planned runs it observes from a fixed cohort and constructs an interval that has its stated coverage for every possible set of outcomes. This design is attractive when the evaluator cannot justify strong assumptions about the model’s run-to-run behaviour. It also means the guarantee is tied to the sampling mechanism and the declared finite population.

An interval that uses observed disagreement can behave intuitively on stable tasks. When sampled runs for a task agree, the evidence is compatible with a low within-task variance. A pair that disagrees shows the evaluator that another attempt can change the task outcome. The calculation turns that observation into a bound while preserving cases that were not sampled. A single successful run provides no such comparison by itself. It may be typical, or it may be an unusually favourable draw. Repetition gives the evaluator information about outcomes across the planned attempts, even when the full set remains unobserved.

For every-task coverage, the proposed allocation draws one run from each task and a second run from a random subset. The choice of repeated tasks happens before observing outcomes. That detail matters because a sampling rule that selects repeats after looking at success or failure could bias the estimate. The proposed design uses random selection to make the inclusion process auditable. More adaptive policies are considered in the theory, and the paper establishes limits that apply even when later choices depend on earlier results.

Confidence coverage has a precise meaning#

A 95 percent confidence interval is a procedure that covers the target at least 95 percent of the time over repeated random selections from the same fixed set of planned outcomes. It does not say there is a 95 percent probability that a particular fixed score lies in the interval after the data are observed. The probability statement concerns the sampling process. A confidence interval that adapts its width to observed results still has to preserve its promised coverage. Full paper and proofs

The paper uses the term “honest” for intervals that satisfy this coverage condition for every fixed cohort. That is a demanding guarantee. It includes cases where outcomes vary across runs, where task outcomes differ and where observations may be dependent. The framework therefore cannot simply infer low uncertainty from a lucky sample. It needs to bound what the unobserved data could still contain.

When a run can stop early at its first decisive outcome, early stopping may save evaluation cost. A strict budget must also handle a difficult cohort where events occur at the end of each run. In that worst case, partial paths reveal no terminal result, and the evaluator can afford only a limited number of completed paths. The paper proves a lower bound for the mean squared estimation error in this setting. This helps distinguish a practically convenient policy from a guarantee that holds for every possible fixed cohort.

The authors prove sharp rates for interval width when the planned runs within each task agree. With full task coverage, the width decreases at a rate tied to both task count and the number of additional repeated runs. If the evaluator may skip some tasks, those saved first observations can fund additional repeats, but then the interval must also account for uncertainty on skipped tasks. The best rate shifts because sampling has to cover two sources of uncertainty at once.

These rate statements describe a worst-case theoretical regime. They do not guarantee a certain percentage reduction on any new model evaluation. If task outcomes disagree frequently, replication provides a warning that within-task variability is substantial. If results agree, the method can obtain a narrower bound under its assumptions and sampling plan. The realized benefit depends on the number of tasks, planned attempts, budget, observed disagreements and chosen coverage design.

Full coverage and repeated audits make a trade-off#

With full coverage, an evaluator obtains at least one result for every task. This is useful when every benchmark item matters to a release decision or when a team wants a complete task profile. Repeats then consume outputs that could have been used to cover further attempts on other tasks. The paper’s task-covering design selects one run uniformly per task and a second run for a random subset. It averages results within each task and then gives tasks equal weight.

If complete task coverage is optional, the evaluator can leave out a random subset and use the saved outputs for more repeated runs elsewhere. The interval then needs a term for tasks never sampled, in addition to uncertainty about unobserved runs on tasks that were sampled. Skipping questions is not a free way to increase precision. The measurement target still averages over the original set, so unobserved tasks remain part of the uncertainty calculation.

The paper derives lower bounds for both designs. The authors show how hard-budget guarantees constrain every evaluator, including policies that use earlier results to decide what to run next. A pathological cohort with late decisive outcomes offers little information from partial runs. Hidden disagreements can leave two plausible fixed populations consistent with the sampled observations. An honest interval has to cover both possible scores. This limits how much any sampling strategy can promise from a small budget.

The practical interval, called Joint in the experiments, uses the number of disagreements among repeated task pairs to bound score error and combines that bound with the observed estimate. It also intersects the result with logically possible scores based on labels already seen. That intersection costs no extra outputs. The paper compares Joint with a fixed uniform baseline that samples task-run pairs from the full grid and uses the exact count distribution.

The task-coverage decision is a design choice that depends on the evaluation’s purpose. A broad capability report may require a result for every category. A local coding benchmark may have enough tasks to sample a representative subset and devote more of its cost to repetition. Teams should declare which population their score describes, whether each task receives equal weight and whether omissions are permitted. Without that definition, a narrower interval could answer a different question from the one an organisation intended to ask.

The LiveCodeBench result uses a fixed budget#

The empirical comparison replays existing outcomes from LiveCodeBench. For each of 16 model configurations, the authors use the first five recorded pass-or-fail results for each of 880 coding tasks. That forms a fixed cohort of 4,400 planned outputs per configuration. The paper compares intervals at 1,100 outputs and a 95 percent confidence level. The full-coverage design takes one random output per task and repeats 220 randomly selected tasks.

At the same output count, Joint produced a narrower average interval than uniform random sampling in 15 of the 16 configurations. The median width reduction was 30.6 percent. This is a result about the replayed configurations and recorded outcomes under the defined sampling protocol. It shows the value of repetition information in that setting. It does not mean every benchmark, model family or evaluation plan should expect the same reduction. LiveCodeBench replay details

The experiment’s budget unit is outputs. Coding tasks in LiveCodeBench have finite pass-or-fail outcomes. Real model runs can have variable token use, tool costs or execution time. A provider that counts every attempt equally will see a closer cost match than a team with long and short tasks billed differently. To use the framework for a token budget, the evaluator would need a policy that maps variable paths to a hard cost bound or a separate analysis that accounts for variable run costs.

The use of recorded results is practical and transparent, but it has a consequence. The 1,100 outputs are sampled from an existing grid of five observed runs per task. The replay can compare sampling strategies against that fixed cohort. It does not create new draws from future model behaviour. A new evaluation with different prompts, model snapshots, tasks or run settings can have a different disagreement pattern and a different interval width.

The paper also studies controlled departures from within-task agreement and longer paths. These experiments help explain when repetition is informative and how output budgets interact with execution length. The empirical findings remain anchored in the defined tasks, recorded outputs and model configurations. A reader can use the theoretical guarantee to understand the design, then treat each application as its own measurement problem.

Hard-budget control is distinct from managing expected spend. A workflow that stops when it sees a passing result can often complete more easy tasks with the same average usage. A strict limit must still hold in the most expensive set of outcomes. The paper reserves enough budget for every sampled run to reach its maximum path length. Its guarantees therefore concern a ceiling on outputs or episodes, even when some paths stop sooner. Teams operating an API account may want a separate cost plan for variable response sizes, retries and tool calls. Those quantities can be layered onto the sampling design once their relationship to the run count is known.

The framework changes the meaning of an evaluation plan#

For a benchmark with repeated stochastic attempts, the evaluation plan should state the full cohort before sampling. It should include model version, task list, prompts, settings, run count and success rule. The cohort definition fixes the quantity to be estimated. A random sampling plan can then select observations from it. The resulting interval describes uncertainty about that cohort’s average score, not every possible future prompt or workload.

This is particularly useful for decisions near a threshold. Suppose a model moves from 79 to 81 percent in a release test whose threshold is 80. The mean alone leaves a question about how much of that change may come from which tasks and runs happened to be sampled. A narrow interval could support a decision if it stays on one side of the threshold. A wide interval may justify more evaluation. The framework offers a way to allocate a fixed number of outputs so observed agreement can inform the size of that interval.

An evaluation team can also use replication to diagnose a suite. Tasks with frequent disagreement may be more stochastic, harder to prompt consistently or sensitive to execution conditions. Repeated outcomes make that variation visible. A task with repeated agreement contributes different evidence. The framework does not classify the cause of disagreement by itself. Logs, prompt inspection and failure analysis remain necessary to understand whether variation comes from model reasoning, a flaky tool, nondeterministic data or an unstable grader.

Task skipping carries an additional product decision. A broad public report may need every category represented. An internal deployment decision might prioritise a smaller set of tasks that reflect customer work, while still estimating a declared weighted score. If weights differ from equal task weighting, the sampling design must reflect them. The paper’s proofs define the target clearly, so teams can see when their own weighting scheme needs a modified interval.

The confidence guarantee is also specific to the selected sampling rule. Audit, Joint, fixed uniform sampling and adaptive policies have different properties. A team should not take the empirical 30.6 percent figure and apply it directly to its test suite. It can instead implement a sampling design, define its target and confidence level, and verify the resulting interval calculation. The paper supplies theory for the methods it studies and reports the conditions behind its replay experiment.

What the result establishes#

The paper contributes a sampling framework for repeated binary outcomes under a hard output budget. It uses within-task disagreement to construct an honest confidence interval, proves rate limits for full coverage and optional task coverage, and tests the approach on a retrospective LiveCodeBench grid. The same nominal budget yielded narrower intervals for most of the tested configurations, with a 30.6 percent median reduction.

The result speaks to evaluation precision. A model’s mean score and the uncertainty around that mean answer related but distinct questions. Repeated runs can reveal variation that a single pass cannot expose. The framework formalises how some of those observations may improve an interval while retaining a coverage guarantee over a fixed population of planned outcomes.

The main boundary is cost interpretation. An output-count cap is easy to reproduce in a binary coding test. It is not automatically equivalent to dollars, tokens or wall-clock time across diverse workloads. Another boundary is external validity. The 16 configurations and historical five-run cohort are evidence for this replay, while future model snapshots and benchmarks may behave differently. A team can take the design principles into its evaluation process, with its own cohort, cost unit and task requirements made explicit.

A reliable model comparison depends on both the estimate and the sampling plan that supports it. This paper makes that plan a first-class part of evaluation. Its central practical point is that repeating selected tasks can tell an evaluator how much unobserved run variation remains. The LiveCodeBench result provides an example of the precision that can recover at a fixed output count. Further applications will show how that benefit changes with other tasks, outcomes and operating costs.