Researchers from Faros AI published a paper on November 11 proposing a way to evaluate coding-agent patches without executing them. An agent first examines the repository state before a task and writes weighted, binary criteria grounded in that codebase. A separate model then reads the issue, the patch and those criteria, and scores how many weighted requirements it appears to satisfy. The paper tests the result on 7,291 labeled patches from fifteen systems across SWE-bench Verified. Its main finding is a strong ranking association within the tasks where submissions include both resolved and unresolved patches. The authors frame the score as a supplementary measure for settings where suitable tests or a working execution environment are unavailable. Repository-grounded rubric paper
The proposal targets a familiar expense. Running a code patch may require restoring a particular repository version, installing dependencies, configuring services and waiting for a test suite. Some benchmark tasks have incomplete tests. Some repositories have a costly or fragile runtime. A score based on tests can still be the most direct evidence that code behaves correctly on tested inputs, but obtaining that evidence is not free and test coverage is never identical to the full set of task requirements. A static rubric offers another lens. It examines whether a diff seems to implement requested behavior, preserve relevant code and avoid collateral changes.
That is a different question from whether the program works. The distinction is the centre of the paper. A patch may satisfy visible requirements and still fail on execution. A test may fail a patch that implements the stated issue through an acceptable alternative. A rubric can offer graded, decomposable judgments without claiming to replace a test suite. The key question is how much a score from that instrument tells us, and what evidence remains outside it.
How repository-grounded criteria are made#
For each task, the first agent sees the problem statement and the repository at the commit before the change. It explores the codebase and writes a rubric consisting of natural-language criteria with binary verdicts and importance weights. The criteria are arranged along four axes: file changes, alignment with the task specification, integrity of surrounding code and runtime behavior that can be inferred from the diff. File-change criteria describe scope and locality. Specification criteria describe what the request requires. Integrity covers collateral changes such as weakened tests or unnecessary dependencies.
The agent generates criteria before seeing either the candidate patch or the reference solution content. It sees only the reference patch’s changed-line count, which the paper uses as a rough scope signal to size a rubric. The authors set a soft target using a capped linear rule. Across 500 SWE-bench Verified tasks, rubrics contained a median of ten criteria, with a range from eight to 24. A larger reference change generally led to more criteria. Rubric generation and validation method
After the criteria are written, the judge receives the issue description, a candidate diff and the criteria text. It has no repository access or tools. For every criterion it marks whether the patch satisfies that property. The final score is the importance-weighted fraction marked satisfied. An example rubric might require a changed path to handle a particular input condition, update dependent call sites or avoid weakening the tests. Each item is intended to describe one property, not one specific implementation choice.
The use of repository context distinguishes this instrument from a judge that sees only the issue and a code diff. Repository exploration can reveal a local contract, relevant types, existing call sites and how adjacent behavior is implemented. The second-stage judge cannot inspect those facts directly, so the generated criteria act as a compressed description of task context. The authors intend the criteria to be implementation-independent: two patches with little code in common should be able to satisfy the same requirements if both solve the task.
The instrument has two stochastic stages. The repository-reading agent chooses which requirements become criteria. The judge decides whether the diff satisfies each criterion. That creates at least two kinds of error. The rubric may omit a real requirement or include one that is not actually required. The judge may then misread an otherwise sound criterion or diff. A stable judge cannot fix a missing criterion, and a comprehensive rubric cannot guarantee that every verdict is correct. The score combines both stages.
A score is not a percent complete#
The authors define the intended target as the weighted share of all requirements implied by a task and repository that can be decided from the problem statement and diff. That full requirement set is never enumerated by a human in this study. The generated rubric samples from it. A score of 0.8 therefore means the instrument estimates that a patch satisfies 80 percent of the weighted criteria it sampled for that task. It is not an 80 percent probability of passing tests and not a claim that 80 percent of the work is complete.
Raw rubric scores are comparable among patches scored against the same rubric. Separately generated rubrics may sample different criteria, even for tasks of similar size. The paper consequently uses within-task ranking to compare resolved and unresolved submissions, rather than assuming that a score of 0.8 on one issue has the same meaning as 0.8 on another. This matters if an organization wants a cross-task dashboard. The score may support comparisons across models under a shared task rubric, but its absolute magnitude is not calibrated as a universal unit of patch quality.
The distinction from test results runs in both directions. A patch can pass designated tests while missing a requirement the tests do not cover. A patch can satisfy many stated requirements yet fail the tests because the tests enforce an implementation detail absent from the issue. The authors cite cases where public test suites demand an unstated return value or reject a valid alternative path. A pass/fail label measures behavior on particular designated inputs. A rubric estimates sampled requirement satisfaction visible in task text and diff. The measures may agree in direction when they cover the same requirements, but they do not describe identical evidence.
Reference-patch similarity has another limitation. It can be computed without running code, but it rewards resemblance to one implementation. A valid patch may use another file or algorithm. The rubric aims to score requirements independent of the reference’s exact route. In practice, the paper finds that reference-like patches do score higher on average, especially on the file-change axis. The design reduces but does not eliminate over-specification. Score interpretation and reference-similarity analysis
The instrument cannot assess behavior that requires execution unless that behavior is inferable from text and diff. A patch that introduces an import error, deadlock, performance regression or incompatibility with an unshown caller can receive a high static score. Conversely, a rubric judge can miss a plainly visible defect. The score’s definition is intentionally limited to what can be judged without tests, repository execution or runtime traces. The reliability of the rubric judgment should not be confused with executable validation.
Validation on SWE-bench Verified#
The authors evaluate the rubric on SWE-bench Verified, a subset of 500 Python tasks across twelve repositories. They use public patches and published test verdicts from fifteen systems. The data contain 7,299 labeled patches after excluding empty cells, with 3,688 patches marked as resolving the task. For a within-task comparison, a task needs at least one resolved and one unresolved attempt. Four hundred tasks meet that condition. Tasks where every submission succeeds or every submission fails cannot tell us whether a score orders the two classes.
The average per-task AUROC is 0.826 across the 400 contested tasks. AUROC here means the probability that a randomly chosen resolved patch receives a higher score than a randomly chosen unresolved patch from the same task, with ties counting halfway. The 95 percent bootstrap interval over tasks is 0.805 to 0.847. On the 415 tasks the prompt optimizer did not use, average per-task AUROC is 0.838. The statistic says the instrument usually orders the test-resolved patch higher within a task. It does not validate score distances or tasks with only one outcome class. SWE-bench Verified evaluation results
The paper also asks whether the rubric score adds information when a system fails its tests. Pass/fail resolution gives every failure the same label, whether a patch barely missed one test or broke many requirements. When systems are ranked by their mean rubric score on failing patches, their ordering closely matches rankings by resolve rate. The Spearman rank correlation is 0.968 across the fifteen systems, with a reported interval from 0.928 to 0.971. Comparisons restricted to failing patches within the same task retain a 0.950 correlation.
That pattern suggests rubric scores can distinguish partial progress hidden by a binary failure label. Among unresolved patches, those that pass some target tests tend to score more highly than patches that pass none. Patches that pass all target tests but introduce a regression also occupy a different group. The rubric can expose those gradations through criterion-level findings. It can help teams study where agents fall short without waiting for a separate execution report for every candidate.
There is a limit to the ranking result. A high correlation among systems on this pool does not prove the correct ordering of every pair of failing patches. It also does not show that the score matches human judgments of which patch is closer to a finished solution. The authors frame the evidence as agreement with test outcomes on a single benchmark population. Independent human rankings and newly collected tasks would provide another validation axis.
Repeatability depends on the rubric#
The study measures judgment repeatability by scoring selected patches five times against one fixed rubric. Across 400 patches from 100 tasks, run-to-run score correlation averages 0.985, and average within-patch standard deviation is 0.020. The mean Kendall ranking agreement within a task is 0.960. These numbers show that a fixed rubric and judge configuration usually produce similar scores when the judgment is repeated.
Generating the rubric again introduces more variation. For 1,451 labeled submissions across the same 100 tasks, scores assigned to a patch differ by about 0.11 on average across rubric generations. The task-level score standard deviation between rubrics is roughly four times the within-rubric judgment standard deviation. Aggregate discrimination varies less, but individual patch values can move as the criteria sample changes. Comparing patches scored by different rubric versions therefore mixes patch quality with the sampling of requirements.
The judge configuration also changes results. With medium reasoning effort, repeatability is lower than with the production setting reported in the paper. Within-patch variation rises, and 107 of the 400 sampled patches differ by more than 0.10 in their five-run mean scores across effort settings. This finding supports treating the judge and effort setting as part of the measurement instrument. It does not identify which setting is more accurate. A reproducible score requires versioning the rubric generator, strategy, judge model, prompt and settings, not simply recording the final number.
The generated criteria are not an exhaustive list of everything a maintainer could require. They are a finite sample from a much larger, undefined set of requirements. The generator can omit a relevant condition or include one that is too narrow. Repeating the same rubric and getting consistent scores addresses rating repeatability, but it does not reveal whether the list itself was complete. Generating another rubric can expose sensitivity to criteria selection. It still cannot tell an evaluator which omitted requirement should have been added without an independent source of task requirements.
Correct reference patches do not invariably receive a score of 1.0. Their mean score is 0.814 and the median is 0.880. Fifty-nine of 500 satisfy all generated criteria, while 47 score below 0.5. A reference patch is one valid solution, not necessarily a perfect implementation of every inferred criterion. Lower scores can reflect an overly narrow rubric, a shortcoming of the reference, or a judge error. The distribution alone cannot decide which explanation applies.
Disagreement can expose failures on either side#
The paper analyses why test outcomes and rubric scores can disagree. Some tests encode an unstated implementation choice or additional behavior beyond the issue description. A candidate can satisfy the reported request and fail because a test expects a particular helper function or exact return value. Such a result may show a test mismatch, not an invalid patch. An OpenAI audit cited by the paper found issues in the specifications or tests of 59.4 percent of a selected group of difficult SWE-bench Verified tasks. That number is specific to the selected hard subset, not the full benchmark. OpenAI audit of SWE-bench Verified
One cited example makes the issue concrete. The task describes a crash on an out-of-range year. A patch catches the error in an existing handler and uses the fallback already used for other malformed input. The reference patch returns a specific string, and the designated tests require that literal even though the issue did not state it. Multiple submissions make the same broader change and receive a perfect rubric score. In this case, the test result and rubric answer slightly different questions. The example does not prove that the rubric is always right. It shows why a disagreement needs to be read against the issue and the actual behavior being tested.
Execution can also catch defects that a diff-only judge misses. One case study describes a high-scoring patch that could not import due to a missing symbol. Another involves a failure that requires an external runtime condition. Textual inspection cannot establish that a patch builds, starts services or preserves behavior across the full repository. The authors explicitly mark build and runtime correctness as unassessed by their validation.
Rubric generation and judging introduce their own error cases. A criterion may accidentally require the reference solution’s file or method even though another implementation would work. The judge can misread a patch, confuse a similar symbol or fail to notice how a change interacts with unchanged code. In case studies, the authors inspect both test mismatch and rubric mistakes. These examples show possible mechanisms. They do not estimate how frequently each one occurs across all evaluated patches.
Contamination is another concern. The generator and judge could have seen public SWE-bench issues or reference patches during training. The paper compares patches written differently from the reference and still finds discrimination, though scores favour reference-like patches. Those checks reduce one explanation but cannot prove that model memory played no role. Evaluation on new or private tasks would offer stronger evidence about transfer beyond the public benchmark.
Where the approach may be useful#
Repository-grounded rubrics could support teams that want to compare agents on their own codebases but cannot build a stable execution environment for every task. A team can select historical issues, restore the repository state before the fix, generate a rubric and score candidate diffs. The criteria can also show whether misses cluster around specification alignment, runtime implications, file scope or code integrity. This is a graded diagnostic signal, especially when an evaluation needs partial credit.
The paper reports a median cost of about 25 cents to generate a task rubric and about half a cent to judge one candidate, using September 2026 list prices and its selected providers. Once a rubric exists for a task set, another model or harness can be scored at the lower per-candidate judging cost. These estimates are tied to the authors’ prompts, models, caching and patch lengths. A different model, longer diff or repository can change the cost. The initial repository exploration also remains a per-task setup step.
The validation pool also shapes the cost and interpretation. It contains fifteen public systems across 500 Python issues in twelve repositories, from Django and SymPy to smaller groups in Flask and Seaborn. A task with only resolved patches or only unresolved patches cannot contribute to the within-task ordering statistic. The result is therefore based on 400 contested tasks, not all 500. Smaller repositories also provide fewer contested examples, so repository-level estimates there are descriptive. This is a useful sample for comparing multiple attempts to the same issue. It remains narrower than a fresh private evaluation across languages and work types.
The same rubric generation method could produce acceptance criteria before code is written. In that setting, the system would explore the repository and outline weighted requirements as a planning aid. The paper notes that its current rubric sizing formula depends on the reference patch’s line count. A prospective planning rubric would need a size rule that does not depend on a solution that does not exist yet. The article describes this as a possible application, not a result validated in the study.
Criterion-level scores may also help select models for task categories. If a lower-cost model matches a stronger model on small fixes in a repository module, it may be a candidate for that subset. This use depends on comparable rubrics, stable scoring and representative tasks. Scores from independently generated rubrics should not be treated as directly comparable magnitudes. Ranking patches within the same task is the tested use.
What the evidence establishes#
The paper introduces an execution-free instrument that estimates weighted, repository-specific requirement satisfaction from an issue, prior repository state and candidate diff. On SWE-bench Verified, its score usually ranked resolved patches above unresolved ones within contested tasks. Scores on failing patches also produced a system ranking that closely tracked resolve rates. Criterion-level outputs provide more information than a single binary pass/fail result.
The validation does not establish absolute score calibration or independent requirement completeness. It does not show that a high-scoring patch builds, runs or behaves correctly under conditions absent from the diff. The study covers Python repositories, one benchmark, one rubric generator configuration and one judge configuration. Rubric regeneration itself creates more variability than repeated judgments against a fixed rubric. These boundaries are central to interpreting the score.
The practical opportunity is a useful middle layer between binary tests and a holistic model judge. Repository exploration gives criteria local context. Static patch reading makes it cheap to score many candidates. Weighted items show which sampled requirements appear satisfied. The result is most defensible when kept alongside test outcomes and used to examine partial progress. Where execution remains possible, tests continue to provide direct behavioral evidence on their inputs. Where it is not, a rubric can offer a structured estimate, with its uncertainty and scope plainly stated.