Researchers introduced STEMBench in an arXiv paper posted on November 18. It covers twelve problems in high-energy physics, astrophysics and neuroscience. Each task uses a metric from its field, such as ranking signal candidates, estimating uncertainty, recovering a physical parameter or generalising across medical-imaging sites. The benchmark also provides two executable reference points for every task. One is a deliberately generic submission with no domain-specific insight. The other reproduces a published method. Eight frontier models were tested with one shared agent framework. The paper finds that task wins and domain leadership do not line up neatly. STEMBench paper

That design takes aim at a difficult question. A science problem often lacks a single answer key. Several methods can be reasonable, and the important question may be which predicts better on data that neither method has seen. A benchmark must choose what “better” means before it runs a model. STEMBench adopts the score used by the relevant research community for each task. That makes its central idea more concrete than a model answering a difficult scientific question in prose. It asks whether a submitted, runnable method can improve a measured research result.

A score needs a scientific question#

The benchmark is motivated by a gap between scientific reasoning and scientific work. A multiple-choice test can show whether a model recognises a concept. A unit test can show whether its code passes a set of expected checks. A paper-reproduction benchmark can ask whether it recovers a published result. Each captures something useful. A researcher also wants to know whether the model can produce a method that performs well under the field’s own evaluation rule. That rule may require balancing accuracy, uncertainty and physical consistency rather than matching one known label.

For example, the desired product might be a classifier that separates possible signals from background events in particle-physics data. The benchmark cannot simply accept a script that produces syntactically valid predictions. It must ask how well the predictions distinguish the cases that matter. A score used by practitioners can encode that priority. If a model writes an elegant explanation but submits a weak classifier, the benchmark keeps the numerical outcome visible. If it generates a strong score with an unreproducible package, another part of the evaluation records that failure separately.

The paper sets out seven properties of a scientific problem that differ from a closed exercise. A task may allow multiple valid methods. It may require a program or a trained model rather than a written response. Domain knowledge can be central to beating a generic approach. The data can be too large to place directly into a prompt. The field’s metric may contain counterintuitive requirements. Some errors are silent because a plausible output can still violate scientific constraints. There may be a practical ceiling below a perfect score because the best published method itself falls short of perfection. STEMBench’s tasks attempt to include these properties in forms that remain executable.

Those choices make the benchmark narrower than the word “science” can suggest. The twelve tasks involve computational data analysis and modelling. They do not test an AI system’s ability to choose an important open question, plan and run a physical experiment, interpret every kind of scientific evidence or convince a research community that a new theory is correct. Their value lies in a well-defined subset where a program can produce an artefact, a held-out result can be calculated, and domain specialists already have a recognised way to compare methods.

Twelve tasks keep their fields intact#

Four tasks come from high-energy physics, three from astrophysics and five from neuroscience. The first group includes new-physics anomaly search, Higgs uncertainty estimation, top-quark tagging and the Dark Machines challenge. Astrophysics includes simulated and real solar Stokes-spectrum inversion and Type Ia supernova classification. Neuroscience includes autism classification across scanner sites, neural-mass model identification, intervention-response prediction, parameter recovery and BOLD signal deconvolution. The set asks for different kinds of work inside one evaluation format. STEMBench task list and descriptions

That variety is useful because an evaluation suite can accidentally reward a single habit. A model that is good at binary classification might perform well on several similar datasets and appear broadly scientific. STEMBench chooses tasks with different metric families and different hidden-target boundaries. Some concern event ranking, some forecast intervals, some reconstruct latent quantities and others test causal response or transfer across sites. The variety makes a single top score less likely to represent one reusable trick. It also makes any cross-task average harder to interpret than a leaderboard for one uniform task.

The data span materially different scales. Some task inputs are generated simulations with hundreds of pseudo-experiments. Others contain hundreds of thousands of events or jets. A task on solar spectra uses a large simulated sample, while the real-data counterpart is organised around two rasters. A brain-imaging classification task has 332 subjects and evaluates on a held-out scanner site. The benchmark reports data preparation and task units so a reader can see what one score covers. A value obtained from thousands of examples and one from a dozen interventions are not equally stable simply because both fit in a table.

The unit of evaluation matters to the conclusion. In an event search, the score is shaped by how many background cases are rejected while signal detection is preserved. In uncertainty estimation, an interval can become narrow by excluding too many true outcomes, so its coverage has to count. In a real-scanner classification task, testing on a held-out site exposes whether the model depends on acquisition patterns learned elsewhere. For the neural-mass tasks, recovering a model family, predicting an intervention and estimating hidden parameters are different problems, so the authors separated them instead of allowing one weighted composite to conceal a weak subskill.

An application-specific score brings its own interpretation. The benchmark’s authors use the metrics its source research communities use, but those metrics are not interchangeable. A score of 0.6 in one task is not necessarily better than 0.5 in another. AUC, likelihood, interval width, classification accuracy and causal-response measures each express a different goal. The table can show performance against the baseline and expert result on the same task. It cannot convert the result into a common unit of scientific ability without further choices about scale and importance.

Two anchors show the distance a solution travels#

For every task, STEMBench runs two reference submissions through the same shipped code as the model entries. The baseline is deliberately simple and uses no domain knowledge. It is not a random placeholder. Depending on the task, it may predict a training-data quantile, use a midpoint for a parameter or return a constant response. The expert anchor reproduces a published method. Both are executable artefacts rather than numbers copied from a paper’s table. Their scores establish how the task behaves under two real submissions.

The difference between them is the benchmark’s “headroom”. It gives a measured interval between a generic approach and one particular expert reference. The interval can be wide, as with a neural-mass identification task whose anchors run from 0.000 to 0.900. It can be narrow, as with autism classification across scanner sites, where the paper reports 0.000 to 0.312. A model’s gain over the baseline can therefore describe a useful move in one task and a much smaller part of the reference gap in another. The raw score and both anchors stay visible so the reader can judge that context.

The expert reference is not a guaranteed upper bound. It is one published method recreated under the benchmark’s data and code conditions. Another method may score higher, and an implementation can differ from the result originally reported. The authors explicitly treat the expert anchor as empirical rather than as a ceiling. That is an important distinction. A label such as “expert-level” would overstate what the reference proves. The test instead supplies a concrete comparator whose code can be rerun.

The baseline also deserves a careful reading. Several metrics are designed so that a generic prediction naturally receives a score around zero. A zero anchor can reflect how the metric is defined, not a broken test. The gap from zero to a reference provides a scale for that task. If researchers later change the metric, dataset, split or expert implementation, the headroom may change. Versioning these pieces is part of preserving what an older result means.

STEMBench reports the raw score, a percentage-point gain from baseline and the two anchors. The paper avoids making a normalised fraction of the interval its headline measure. Dividing the score gain by headroom can magnify small changes on tasks with a narrow interval. In this suite, some of the smallest gaps are themselves associated with the scientific challenges the authors consider most difficult. A normalised number could make those cases look deceptively large. Keeping the quantities separate leaves the reader with more work, but reduces the risk of one convenient summary hiding the task structure.

A shared harness is a scientific instrument#

The researchers put all twelve problems in a common folder contract. Each contract specifies the task, data roles, accepted submission files, checks, score implementation, dependencies and compute requirements. Public input and training material can be staged into the agent’s environment. Ground truth stays on the grading host and is read only after submission. The task therefore exposes information needed to work while keeping the answer-bearing split away from the model. Reproducibility files and checksums help establish what the grader received.

The model operates in an offline sandbox. The setup disables network access and gives it six tools, including a shell, Python, structured data exploration and local document retrieval where references exist. This avoids turning a test into unrestricted web search for a published answer. It still leaves choices about how to inspect data, write code, use the available packages and decide which experiments fit the run budget. The authors report inputs ranging from 0.1 to 12 gigabytes. Data inspection is part of the problem, and the environment constrains what inspection means.

Each episode has a limit of one hour, one million model tokens and 200 messages. The system asks the agent to get a valid submission in place early, so that a limit can prevent further improvements without erasing all output. The validation checker reports missing files and schema problems without revealing the score. This scaffolding creates a repeatable condition for different models. It also becomes part of the measured system. Another harness, retrieval setting, model interface or time budget could produce different results even for the same base model.

The paper distinguishes the numerical result from “official submission validity”. A valid entry has the required format and reproducibility files. That does not mean the science is strong. Conversely, a numerical score does not indicate on its own whether the package could be rerun in an official challenge. Of the 288 retained model runs, 168 had a positive validity flag, 64 had a negative flag and 56 had no flag available. Among 88 zero-scoring runs, only 17 had an explicitly negative flag. A zero is therefore not a single diagnosis. It may reflect a valid but poor submission, an invalid package or incomplete validity data. STEMBench evaluation and results

The rankings split across tasks and domains#

The paper tested eight frontier models with a basic agent at high reasoning effort. Each point in the headline comparison is the mean of three retained runs, including zeros. The plotted bars show the minimum and maximum of those three runs. They are not confidence intervals or standard deviations. The distinction changes how much statistical meaning a reader should assign to the ranges. Three observations provide an early view of what the benchmark can reveal.

First, winning many individual tasks does not guarantee leadership across a domain. Opus 5 and Gemini 3.1 Pro each lead five of the twelve task means. GLM 5.2 leads the other two. Yet Opus does not lead a domain average. In high-energy physics, its task wins coexist with lower scores on other tasks, and its mean trails DeepSeek V4 Pro. Gemini leads astrophysics and neuroscience, while its domain means vary substantially between the two. The pattern argues for reading the task profiles instead of extracting a single winner from the count of first places.

Second, at least one model’s mean exceeds the reproduced expert anchor on seven tasks. Those cases include new-physics search, top tagging, Dark Machines, both solar-spectrum inversions, Type Ia classification and neural-mass family identification. Every model mean remains below the expert reference on five other tasks. The gaps include Higgs uncertainty, cross-site autism classification, causal neural-mass predictions, parameter recovery and BOLD deconvolution. One task set thus contains both apparent gains beyond a published implementation and areas where the reported means still have ground to cover.

That result needs more context than “models beat experts on seven tasks”. An anchor is an implementation of one published method, and it is not described as an unbeatable expert. The score is the task’s own metric under this dataset, split, grader and sandbox. The finding says that a model’s mean crossed a particular reference score in selected tests. It does not establish that a model can replace a researcher, transfer the method to a new project or carry scientific conclusions into a paper without review. STEMBench paper

A strong average can still hide unstable work#

Three runs are a small sample for a stochastic agent. Some reported task scores vary sharply across repeats, and the paper includes a wider variance analysis for selected settings. The result for top tagging is especially illustrative. The main high-effort runs include very different outcomes. An extended-budget series clusters near 0.94, but that series also changes the API provider and omits GPU acceleration. The authors note that the experiment cannot isolate which change explains the difference. It suggests budget or runtime conditions may affect scores, yet does not prove one cause.

The reasoning-effort sweep is also not monotonic. For one scanner-site classification task, the medium-effort result is around 0.327 while the high-effort runs score zero. A BOLD deconvolution task also degrades at higher effort. The available score alone cannot establish whether that came from invalid output, a valid but weak prediction or truncation. The paper points to run traces and validity flags as information needed to distinguish explanations. More effort is not automatically more scientific performance, especially when longer reasoning competes with an execution budget.

There is a further unfinished validation step. The harness passes functional tests, and each task contract passes checksum and anchor-grading smoke tests. A blinded study asking domain experts to rank intermediate submissions between baseline and anchor is still pending. The authors set targets for agreement and describe this review as the remaining validation. The automatic metric may be technically reproducible while its ordering still needs confirmation from domain reviewers. That review matters because a benchmark can faithfully compute a score and still misrepresent what experts regard as meaningful progress.

This is why the benchmark is most useful as an inspectable research instrument at this stage. Its task definitions, grader, baseline and reference can be examined. The paper publishes the agent prompt and run settings. A reader can ask how the score behaves on a particular task and whether the reference is fair. The expert-agreement study would address a different question. It would test whether the automatic ordering matches judgments from people who know the domain, not whether the code executes.

How a team can use the numbers#

For a model evaluation team, STEMBench’s main practical lesson is to define the job before selecting a winner. A system intended to find rare high-energy physics signals needs different success conditions than one recovering solar properties from spectra. The baseline gives an initial reference for how much a solution improves on a generic approach. The expert implementation provides another point of comparison. The task’s native metric says which compromise the field is willing to make. Those facts can help a team inspect a result without combining them prematurely.

The set can also serve as a regression test for changes to a scientific coding agent. A different reasoning model, agent scaffold or tool policy might alter code quality, runtime use or experiment selection. Keeping the task version and execution conditions fixed would let an institution see how those changes affect submissions. The authors describe separating model changes from harness changes as necessary. That principle extends to local systems. A comparison of model A with tools X against model B with tools Y measures two complete configurations, not the base models in isolation.

The public assets invite rerunning and targeted examination. A lab might care about one task and adapt its lesson to its own code. It could inspect whether a score is sensitive to data sites, reference choices or budget. The publication describes its data rights and task files. Such adoption still calls for care around the specific datasets and permitted uses. For instance, the neuroscience set includes ABIDE under a public data-use agreement. Evaluation material should be used within its terms and with appropriate regard for sensitive research populations.

The task roster is not a proxy for every research area. Physics, astrophysics and neuroscience provide three computationally rich domains, but other disciplines use different evidence and method standards. Laboratory experimentation, field observation, legal or ethical reasoning and theoretical work may need other forms of evaluation. Adding tasks is not simply a matter of increasing the count. Each one needs a defensible scientific target, an appropriate metric, a reproducible reference and a clear statement of the data boundary.

For readers comparing evaluation studies, STEMBench also highlights the importance of published baselines. A high score is easier to interpret when the output is compared with a no-domain-knowledge solution and a rerunnable expert method on the exact test setup. The values help reveal whether a benchmark is trivially solved, practically difficult or bounded by an already strong technique. They can also expose when different studies use the same label for a substantially different experiment. Reference implementations and run details make a score less dependent on a headline.

What a result establishes#

The paper gives a detailed accounting of a particular benchmark and an early high-effort comparison. It reports eight models, twelve problems, two measured anchors, code-enabled work in an offline environment and scores on withheld targets. That is enough to show how task-level rankings can differ from domain averages. It is not enough to call a system a research scientist or to infer universal scientific competence from seven above-anchor means. Those broader claims require evidence outside this task set.

The strongest positive signal is structural. STEMBench makes the comparison object more visible. A reader can inspect which data the model saw, which target remained hidden, what metric applied, what baseline and reference achieved and whether the submission was formally valid. It also preserves domain differences that a blended score might flatten. That gives later evaluators a concrete starting point for questions about scientific ability and agent tooling.

The main unresolved issue is how closely the automatic ordering tracks expert judgment. The paper says that study has not yet been completed. Repeated runs, broader model coverage and independent review would improve the reading of small differences. Until then, task score, model ranking, validity, expert anchor and run variation should be read as separate pieces of evidence. The paper offers a substantial benchmark design and an instructive first pass through it. Its score table begins the discussion of scientific agents. The planned validation would establish how much weight that table should carry.