Science Prediction Universe

What SPU measures

AI agents can now search the literature, run quantum-chemistry and molecular-dynamics calculations, and write analysis code for hours on their own. Yet many questions in experimental science have no computed answer. Which electrolyte additive makes a zinc battery last longer? Which enzyme variant pushes an oxidation toward the desired product? Computation can narrow the candidates, but the answer comes from synthesis and measurement. SPU tests whether an agent can decide which evidence and which calculations matter for the quantity being measured, and then commit to a prediction without any experimental feedback.

Other benchmarks cover parts of this. Outcome-prediction benchmarks such as BrainBench ask about real experimental results, usually as a single question to a model without tools. Agentic science benchmarks let agents write and run code on data that is already in hand, and long-horizon container benchmarks grade answers with fixed tests. SPU gives an agent hours of compute and open web access and scores it against real wet-lab measurements.

What does an SPU task look like?

A task describes a set of experiments from one study: the candidates, materials and conditions, and how the result was measured. The experiments are split into comparison groups, each with one measured quantity such as a reaction yield or an ionic conductivity, and the agent orders each group from the lowest to the highest value. It works alone for up to eight hours in a container with 8 CPU cores, 16 GiB of memory, one NVIDIA H20 GPU and internet access. Each experiment carries an anonymous ID, and the agent is never told which study the task comes from.

When the agent stops, a separate container without network access compares its ranking with the measurements using Spearman correlation. Random valid rankings average 0; 1 is perfect agreement when the measurements have no ties. Ties make the attainable maximum slightly lower on some tasks. Small tasks can score well by luck, so each task also has a chance threshold: the score that only 5% of 4,000 random rankings exceed. Scoring is automatic, and no model acts as a judge.

More than ten experts built the public tasks from studies that report full measurements. They left out comparisons that simply follow a trend, such as a concentration series, and comparisons whose values are so close that the order could come from measurement error. Each task was reviewed independently against its sources, and the final 100 were chosen after pilot runs with agents. The private tasks come from four series of unpublished experiments run with partner labs, on VHH nanobodies, reaction conditions, electrolytes and variants of the antibody Magrolimab. All were measured with replicates, and no model scores were used to choose them.

Can't an agent just look the answer up?

Public tasks come from published papers, and the agents can search the web. SPU uses four safeguards. The source papers are recent. The instructions forbid using the source study or recalling its results, and an agent whose search turns up a forbidden result must stop and record it. Scoring happens in an isolated container, and every trajectory is kept for review. Finally, the private tasks cannot be found online at all.

We checked two possible shortcuts. Submitting the experiments in their listed order gives a mean score below every configuration. Runs flagged for source identifiers, such as a DOI, showed no detectable score advantage within any configuration after correction for multiple comparisons. These checks do not rule out contamination.

How do current agents compare?

We tested twelve frontier models in thirteen configurations, where a configuration is a model with its reasoning effort and agent harness. GPT-6 Astra ran in Codex at medium and at max reasoning effort. GPT-6.1 Sol also ran in Codex, and the other ten models ran in Claude Code, all at max effort. The main comparison uses one official run per configuration on every task.

SPU is far from solved. Even the best configurations beat chance on only about half of their scored tasks. The point estimates form two broad groups: every difference that holds up statistically (17 pairs after Holm correction) separates configurations in the leading six from those in the other seven, and one run per task is not enough to order the configurations within a tier. Claude Opus 5.5 and Claude Fable 5.1 were scored on fewer tasks because of frequent safety refusals, especially outside materials science, so their averages rest mostly on materials-science tasks.

Public tasks only. Configurations were scored on different sets of tasks, so each mean is adjusted for task difficulty; brackets give 95% bootstrap intervals over tasks. Above chance counts the tasks where a configuration beat the task's chance threshold.

On the 28 tasks scored by all thirteen configurations—27 public and one private—the leading six still hold the first six places, in a different order. This common set reproduces the two groups without settling the order within each.

On the 16 private tasks, whose results have never been published, 159 of 208 official runs were scored. Twelve of the thirteen configurations rank significantly better than chance. Scores depend strongly on what is being measured, and no two configurations differ detectably. Every configuration ranks the ionic conductivity of the electrolytes almost perfectly, but none of the scored configurations beats chance on the electrochemical window of the same solutions.

The average API cost of a scored run ranges 300-fold across configurations. DeepSeek-V4.1-Flash-0910, Doubao-Seed-Evolving-0909, GPT-6 Astra (medium) and Claude Opus 5.5 form the cost-performance frontier: for each of them, no other configuration is both cheaper and better. Among the five cheapest configurations, the biggest spender has the lowest score.

Why does Claude Opus 5.5 lead, and why does GLM-5.3-Flash trail?

Claude Opus 5.5 has the highest mean on every view of the results: 0.478 task-adjusted on the public tasks, 0.486 over all its scored tasks and 0.525 on the 28 tasks that every configuration completed. GLM-5.3-Flash is last over all its scored tasks and on the 28 shared tasks, at 0.237 on both, and second to last on the public tasks at 0.239, just above Qwen3.8-Max-0902 (0.227). To see where the gap comes from, we audited the trajectories of the two models on the 43 public tasks that both completed, 26 in materials science, 13 in chemistry and 4 in biology. Opus 5.5 scored higher on 31 of them and GLM-5.3 on 11, by 0.29 on average.

The lead is in the first ranking, not in the revisions.

Opus 5.5 writes a complete ranking at a median of 10% of the run and GLM-5.3 at 86%, but the timing is not what decides the score. Opus's first rankings already score 0.475, against 0.250 for GLM-5.3, and the later work moves both models only slightly (Opus 0.469 to 0.509 and GLM-5.3 0.206 to 0.219 over the runs with a visible draft). Effort is not the difference either: the median runs last 1.8 and 2.1 hours, and across all thirteen configurations run time, tool calls, web fetches and GPU use are not associated with score once task and configuration are accounted for.

Same toolbox, different execution.

The two models reach for the same methods. They use 2.7 and 2.6 method families per run, run Monte Carlo sampling in 88% and 86% of runs, physical simulation in 37% and 44%, and GPU workloads in 21% and 19%. What differs is whether the code works and what it is pointed at. Of 6,714 Python calls by Opus 5.5, 3.5% failed, mostly on missing packages and numerical library limits; per run, its failure rate is the lowest of the eleven configurations with full trajectory tables. Of 8,594 calls by GLM-5.3, 16.8% failed, the highest rate of the eleven, and most failures were bugs in its own code: wrong keys and indices (25%), type and attribute errors (21%) and wrong values or shapes (12%). Opus also reads more of the evidence it finds, fetching a median of 14 pages against 4 and consulting a scientific database in 40% of runs against 21%, while searching less.

Three runs side by side.

Fluorene dehydration (five alcohol and HBr concentrations, ranked by alkene yield). Opus 5.5 built a kinetic Monte Carlo model of ionization, bromide capture and side reactions on xTB and DFT anchors, wrote its ranking two hours in and kept it after comparing it with the runner-up order; it placed the lowest- and highest-yield conditions correctly and scored 0.9. GLM-5.3 assumed that bromide substitution consumes the acid stoichiometrically, so that alkene can only form from alcohol in excess of acid. That premise predicts almost no product for the condition with the most acid, which the experiment measured as the highest yield, and it submitted nearly the reverse order (−0.9), in a file that also used the wrong format.

Cobalt-mediated polymerization (six ligand settings, ranked by monomer conversion). Both models ranked by the strength of the cobalt–carbon bond, a weaker bond releasing more radicals. Opus 5.5 computed bond energies at PBE-D3 on xTB geometries, caught that two ligands in its own input had the wrong stereochemistry, fixed the decision rule before the decisive results came in, and scored 0.771. GLM-5.3 computed trapping free energies with xTB alone, found the two ligands that the experiment measured as slowest to be the weakest traps, ranked them fastest, and scored −0.429. The physical idea was the same; the computed quantity pointed the other way.

Copper–gold electrodes (six compositions, ranked by hydrogen incorporation in isotope-labeled acetylene reduction), where GLM-5.3 won. Opus 5.5 submitted a ranking monotone in gold content within 20 minutes and kept it, judging that breaking a near-tie would change its expected score by only about 0.001. GLM-5.3 relaxed 377 alloy slabs with a machine-learned potential, caught a lattice-constant bug from an absurd adsorption energy, and its computed differences reproduced the measured order within the copper-rich electrodes. It scored 0.543 against Opus's −0.029. When the computation targets the measured quantity, the weaker model can win.

Habits that separate them.

Across the audited runs, Opus 5.5 keeps a valid submission from the first minutes, reads the task's output contract and validates its files against it, and rechecks the numbers in its report against its raw outputs. GLM-5.3 runs sanity checks that catch gross errors in its own computations, once even catching itself padding checksums with invented characters, but it recovers requirements by searching its own transcript late in the run, and in the fluorene run its final consistency check validated the wrong schema. Opus 5.5 is also the only configuration whose submissions score above the consensus of the other twelve, while for GLM-5.3, as for each of the seven trailing configurations, the average ranking of the others predicts better than its own.

Two caveats apply. Opus 5.5's runs on 56% of the public tasks were stopped by its provider's biological-risk classifier, so the audit covers 43 tasks, mostly in materials science; the tasks it skipped are not detectably easier for the other configurations (0.023 lower, 95% CI −0.100 to 0.055), and it still ranks first on the 28 shared tasks. And the comparison is between end-to-end agents: model, reasoning effort and Claude Code harness together.

Why do predictions fall short?

We analyzed all 1,155 scored runs on the public tasks, 227,103 tool calls in total. Expert reviews of selected runs point to scientific failure modes: agents compute a quantity that is easy to calculate but does not control what the experiment measured, or revise a good early ranking based on assumptions they never checked.

The task, not the model, decides which tools an agent reaches for.

Monte Carlo sampling is the most common computational method, used in nearly three-quarters of runs, and mostly to put error bars on a ranking rather than to model the experiment. The domain-specific tools follow the discipline of the task: cheminformatics, semiempirical quantum chemistry and DFT concentrate in chemistry, sequence and structure bioinformatics in biology, and medicine tasks rarely call a physical simulation at all. Two repeated runs of GPT-6 Astra (max) on the same task use largely the same method families (mean Jaccard similarity 0.65), against 0.29 for runs on different tasks. What separates the configurations is breadth: a median of four method families per run for GPT-6 Astra (max), one for GPT-6 Astra (medium).

Almost every run checks the GPU; one in five computes on it.

Each container has one NVIDIA H20 GPU, and the instructions ask the agent to check the hardware before deciding what to run. 98% of runs probed the GPU, 89% within their first ten tool calls, but only 255 runs (22%) completed a GPU workload, from 7% of Kimi-K3.1-Pro runs to 52% of GPT-6.1 Sol (max) runs. DFT accounts for half of them, ahead of molecular dynamics, protein and sequence models and machine-learned interatomic potentials, and the workload follows the discipline of the task. Runs that used the GPU took a median of 4.9 hours against 1.0 hour for the rest, and 36% of the runs that started a GPU workload hit a GPU error, most often a CUDA library that failed to load. The return is small at best: compared with other runs of the same configuration on the same tasks, runs with a GPU workload score 0.035 higher, with a 95% interval from −0.019 to 0.090 that includes zero. GPU acceleration appears to make a chosen proxy faster to evaluate without bringing it closer to what the experiment measured.

More computation did not bring a detectable accuracy gain.

Switching GPT-6 Astra from medium to max effort makes a run 4.8 times longer and 5.3 times more expensive (medians), without a detectable improvement in score. On the 79 tasks with a recoverable first complete ranking, max effort produces that draft a median of 26% of the way into the run. That draft scores 0.469 on average, against 0.424 for the final submission; the difference is not statistically significant. Its later swaps do not detectably outperform random swaps at the same distance in the draft. Across all thirteen configurations, scores show no detectable correlation with run time, tool calls, web fetches or GPU use after correction for multiple comparisons.

Agents agree with one another more than with the experiment.

Agents lean on the same simple assumptions. When the items in a group differ in a number such as temperature or composition, agents often just sort by that number: among 95 eligible groups from 37 tasks, about a quarter of submissions follow such a monotone order, compared with 11% of measured orders. The configurations also agree with each other (0.52 Spearman on an average task) more than with the measurements (0.33).

Good runs model the quantity the experiment measured.

After correcting for multiple comparisons, none of the annotated methods or behaviors shows a detectable association with higher scores across the 1,155 scored public runs. We read one high-scoring run per discipline in detail; they were picked as examples, so they show how a good run works but not how often it happens. Each ranked the candidates by the quantity the experiment measured, and run length made little difference: the two multi-hour runs beat chance by no more than the two that finished within an hour.

Low-scoring runs often applied the same methods to the wrong quantity or the wrong molecule. On a task that ranks ligands for a cobalt-catalyzed polymerization by the conversion they give, GPT-6 Astra (medium), GLM-5.3-Flash and Qwen3.8-Max-0902 all ran semiempirical calculations and a kinetic model. They scored 0.829, −0.429 and −0.429, and they were not modeling the same chemical species: one run built the ligand with the wrong stereochemistry, and another left out hydrogen atoms even though its own tool output reported them missing.

100 public tasks, 16 private tasks

Agents need to identify what controls a measurement.

Within the eight-hour budget studied here, longer runs and heavier simulations did not bring a detectable improvement in prediction accuracy. To close the gap between computation and real experiments, agents will need to work out which mechanism controls a measured result and test their own physical assumptions before committing to a prediction.

Contributors