ScienceVision
Can AI Observe and Reason Like a Scientist?
ScienceVision is an interdisciplinary benchmark for scientific 3D understanding and reasoning. It uses expert-level visual question answering over native 3D objects to test whether a model can obtain the evidence it needs from the object itself and then reason from it. The best score is 0.53 out of 1, from GPT-6 Astra (Codex).
50 per discipline
22 subdomains
66 experts · 482.98 recorded hours
GPT-6 Astra · Harness · 3D
Best mean answer score at the 3D setting; values are the mean
answer_score on [0,1],
with ineligible trials counted zero.
Why it is hard
Visual observation in scientific research
Scientific discovery starts with observation. Galileo's telescope found the moons of Jupiter, Hooke's microscope named the cell, and Rosalind Franklin's X-ray diffraction images showed the DNA helix. A researcher has to notice what is anomalous in a three-dimensional field, work out how it is assembled, and decide what to verify next. Models can now read papers, write code and derive formulas, but interpreting visual evidence is a separate capability.
Scientific observation starts with finding the evidence needed to answer a question. A chemist may have to reveal an occluded atom before tracing a bonded path; a radiologist has to locate a lesion inside a volume. For a model, that means choosing what to inspect, obtaining the evidence, and connecting it to the question. Occlusion, repeated components, and coordinate conventions affect these observations and measurements. Research agents that generate hypotheses or run computational experiments face the same demand.
ScienceVision evaluates scientific question answering through 250 tasks across five disciplines. Each task pairs a 3D source with a domain question, a specified response format, and a reference answer. The questions require models to use domain knowledge to interpret geometric or topological evidence.
Hands on
Inspect the source yourself
ScienceVision spans five disciplines whose native 3D sources span several formats. The five samples below cover five of them: a protein is an mmCIF file, a crystal is a CIF with unit-cell symmetry, a CT study is a NIfTI volume, a CAD assembly is a STEP file, and a building is glTF. This page brings those formats into one interface and opens each in the viewer its discipline uses — Mol* for structures, NiiVue for volumes, OCCT for STEP, three.js for glTF.
Choose a discipline and inspect the source directly: orbit, zoom, section, read the unit cell, or switch the CT between a 3D render and orthogonal slices. This is active visual inspection — the reader chooses what to expose and how to look at it, rather than reading a fixed rendering. These are the expert-facing viewers, not the evaluation harness; nothing is uploaded, nothing is saved, and the page does not score your inspection. Each viewer loads only when you open it; the medical volume is the heaviest at about 61 MB, so its first view can take a few seconds.
Question
How the work is done
The reference inspection, recorded
The tools in ScienceVision are modelled on how a domain expert works a 3D source: orbit until the target is visible, isolate it from what surrounds it, cut away what is in front, switch how the structure is drawn, then measure what matters. Those inspections were captured operation by operation, so the workflow can be watched rather than described.
Five play below, one per discipline, chosen to show how differently the work divides. One inspection is almost all camera moves; another is almost all selecting and hiding; another reads a unit cell and lets symmetry show how the packing repeats. The bar under the viewer is the complete operation record, coloured by the kind of operation; click any step to jump to it, or use the discipline buttons to move between them.
measurement selection & display section camera
All 250 questions carry a recorded inspection, running to 62 operations, and every one was captured keyframe by keyframe, which is what makes the replay above possible. The proportion of operations that are measurements, selections, sections or camera moves differs sharply by discipline; counts are shown per trajectory in the timeline.
Task anatomy
What one question contains
Each task pairs a source artifact with a domain question and specifies the answer format to return. The response is evaluated against a structured reference answer.
Source artifact
A molecular coordinate file (XYZ or CIF), a CT volume (NIfTI), or a CAD or building model. The question concerns the supplied object, not a description of it.
Question
Identifies the target or explains how to locate it, and states units, coordinate conventions, or inspection conditions where they are specified.
Answer contract
The question's own contract sets the output structure. Many ask for JSON fields; others ask for a single word or category.
Reference answer
The evaluator's target for comparison. Code and tool calls are intermediate means of
obtaining evidence, not the requested result. The released task tree also includes the
reference solution under oracles/<task_id>/solution/, whose
solution.sh runs the task's oracle program; in the Harbor task format this
is the oracle solution, a reference implementation rather than a model output
(paper App. A.5; task tree oracles/).
Three answer contracts from the set:
{"sp_count": 2, "sp2_count": 6, "sp3_count": 1}
Classify every heavy atom by hybridization from its bonding and local geometry, then count each category.
{"point_vox": [77, 105, 120]}
A voxel inside a lesion. The prompt fixes array shape, axis order, voxel size, and zero-based integer indices.
{"answer": "asymmetric"}
The question asks for one word, symmetric or asymmetric. The JSON wrapper is the normalised record, not the question's own format.
Atom identities can also be nested objects carrying chain_id,
residue_name, residue_number, atom_name and
element. Field counts describe output structure, not dependent
subquestions or reasoning steps.
Access settings
Two ways to obtain the evidence, and a 2D control
The same 250 questions are posed under two access settings. What changes is not the question or the scoring, but what the solver is given. A third condition supplies only rendered images, and serves as a control against which the two settings are read.
Source access
A coding agent analyses the source artifact programmatically. It can read coordinates or other source data and write code to identify objects, compute measurements, or test geometric relationships.
Source access with inspection tools
The same source access, plus human-oriented interfaces: change views, isolate structures, inspect slices, measure geometry. The operations are modelled on expert inspection, and are available to the agent as tools.
Fixed-view control
Only rendered images accompany the question, so evidence has to come from them and the solver cannot compute from the source. The condition varies the camera plan and the number of views. The aligned plan looks along canonical directions; the orbit plan spaces views evenly around the horizontal circle at a fixed elevation. One, three, six and twelve views of each plan were tested, and six aligned views, the setting used in the comparison below, was the best of them in a GPT-6-Astra run under Codex. Medical tasks pair each volume render with a slice, so a six-view manifest contains twelve image files: six renders and six slices (paper §5).
Direct artifact access still tests visual exploration of the scientific object: the model must locate and interpret the task-relevant spatial evidence. What the score does not show is the route. The model may compute from source geometry in code rather than observe the object in a scientific application, and access to source data or tools does not by itself establish that the required evidence was acquired through a viewer.
Model comparison
How the models compare
Twelve models were run over the same 250 questions under the two 3D access settings. Both share a metric and a scale, so they are charted together and answer whether the inspection tools help. The 2D control uses the same metric on a fixed-view rendering and gets its own chart.
The questions are hard. The highest discipline cell is 0.65, and the best overall mean is 0.53.
The models separate. They run from 0.53 down to 0.16, and under both 3D settings every closed-source model stays above every open-source one.
The disciplines differ. Mechanical CAD, medical imaging and architecture have the lowest floors, with their weakest models below 0.10. Chemistry and biology stay flat and high: no model falls below 0.24 there, and biology holds the highest discipline cell in the set (0.65).
Metric and coverage. Values are the mean
answer_score on [0,1],
with ineligible trials counted zero. The two 3D settings average up to three runs per
question and then the 250 questions; Gemini's carried-over segment has two runs. The 2D
control uses the same metric over three runs; Hunyuan-4 has three questions with two
runs. All 250 questions are covered. The 2D control is drawn on its own axis, so compare its order with the 3D
charts, not its bar lengths.
Harness. The two GPT-family models were run in a Codex harness on all 250 questions and in all three settings; the other ten in Claude Code. A GPT row therefore measures a different scaffold, not only a different model. Run under Claude Code instead, GPT-6 Astra scores 0.440 on Harness · 3D and 0.426 on Harness · 3D + MCP, and GPT-6.1 Sol 0.419 and 0.416, seven to ten points below their Codex rows.
Harness cross-check. Under Codex, Claude Fable 5.1 scores 0.507 on Harness · 3D and 0.535 on Harness · 3D + MCP, versus 0.462 and 0.463 under Claude Code; Claude Opus 5.5 scores 0.545 and 0.532 under Codex versus 0.454 and 0.434 under Claude Code. The two Claude Codex rows were run on the current 250-task package at high reasoning effort, while the other rows splice the archived runs with the rerun tasks, so these gaps also include task-version and effort differences; they are not a controlled harness-only comparison.
Sources and drift. Each score splices a carried-over segment with a re-run segment, and six models take their re-run segment from a separate evaluation service. Runs were measured on different days and model aliases carry no version, so a difference here can mix a change of run with a change of model version.
Equal coverage does not establish a controlled comparison.
The set
What the 250 questions cover
Five disciplines, 50 questions each, spread over 22 sub-domains, four or five per discipline, and never an even split. Bucket size runs from 4 questions to 25, with a median of 10.
How concentrated a discipline is varies: its largest bucket takes 28% of biology, 32% of architecture, 40% of mechanical CAD, 42% of chemistry and 50% of medical imaging, where lesion pointing alone accounts for 25 of the 50. Sub-domain labels describe task categories rather than mutually exclusive cognitive abilities, and equal discipline counts do not imply uniform coverage of task types within a discipline.
These describe the delivered records. A step count does not establish a complete demonstration, expert certification, a required tool set, or a minimum reasoning length, and the 16% is an operational statistic rather than a claim about which tasks intrinsically require measurement.
Task examples
Three questions and what they demand
These describe questions in the current set, rather than inferring model behaviour from final scores.
Chemistry
Occluded atoms and bonded paths
Identify an almost fully occluded atom and a partially occluded heavy atom in the initial view, then find the shortest bonded path between them and the minimum absolute dihedral angle along it.
- task
- sv-chem-chem_conformation-0002
- reference
- [21, 8, 9, 3, 2] · 3 intermediate · 64.49°
Architecture
Continuity, extents and units
A continuous member spans two or more storeys without interruption. Count them, give the shortest and longest extents, and say whether any is discontinuous.
- task
- arch-anzj-align-a07
- reference
- 4 members · 12,000 mm and 12,200 mm · none discontinuous
Mechanical CAD
Repeated and hidden instances
Count instances of one component design, where identical shapes are marked as separate instances by colour and some are hidden behind other components.
- task
- hard-011
- reference
- 14 instances
The architecture question specifies millimetres in a Z-up source frame and metres in a Y-up viewer frame, so a correct answer depends on converting coordinates and units consistently. In the CAD task the final count alone cannot show whether a model identified the correct part, observed all instances, or obtained the number from source data.
Limits
What the scores do not settle
Several limitations bear on how the scores should be read.
Independent answer-component scores identify which subquestions fail. These verdicts do not by themselves verify the evidence used or establish whether an error came from observation or reasoning.
Three biology tasks require camera behaviour in an interactive viewer that the report says was absent from the Codex evaluation system. For one model, repairing missing biology comparators changed 11 of 12 trials across six tasks from zero to full score without changing the answers; a further 13 trials across seven tasks did not parse into the required answer fields. Cross-model comparisons therefore reflect the model, the execution system and the scoring pipeline together.
Of the 250 delivered MCP trajectories, 173 include at least one tool call and 83 include a measuring-tool call. The intersection of numeric-only answers with observed measuring-tool calls holds 40 tasks, or 16%.
Scores are reported as evaluated, and do not show that every historical run has been independently replayed. Rebuilding the aggregate tables reproduces the summary figures on this page, not a replay of the original evaluation.
Construction
From candidate questions to expert review
Automation and experts build the questions together. Neither side works alone, and each review pass feeds the next draft.
Automated checks first match an asset to a task and confirm that the object can actually be inspected, then draft candidate questions and screen them against rules before packaging each one with its assets. Experts then review the questions, inspect the 3D evidence themselves, and record reference answers and the route they took to reach them. Records that fail quality review go back for revision, by hand or by an agent, and are checked again before export.
Outlook
Where this goes next
Finding the evidence a question needs, and turning it into a checkable answer, is one of the capabilities scientific work asks of a model. ScienceVision makes it measurable on cross-disciplinary 3D tasks, with inspection tools drawn from expert workflow and answers checked against task-specific references. Expert knowledge and one annotation platform are what hold the question, the inspection and the verification together.
Next we will widen the set of disciplines the benchmark covers, and explore extending the evaluation to real production settings.
Contributors
Who built it
The expert roles were staffed across all five disciplines and coordinated through one annotation platform, so all five were built and checked the same way rather than separately and combined afterwards.
That pool spans all five disciplines: 66 people, with 482.98 hours of recorded working time between them: 74 in biology, 113.33 in chemistry, 103 in medical imaging, 115.65 in mechanical CAD and 77 in architecture. The hours cover question writing, revision, reference-answer annotation and review.
- Core contributor
- Tianyi Ma1,*, Jing Gu2,*, Haozhe Zhao1,‡, Liang Chen1,‡, Yunfan Zhang1, Haijun Su2, Jialong Wu1, Le Xue2, Arnav Das2, Dustin Tran2, Yinfei Yang2, Andrew Dai2,†, Kuan Li1,†
- Contributors
- Liangkun He1, Zihao Guo1, Xiang Gong1, Yutao Zhou1, Jiarui Sun1, Chengda Wu1, Zhao Zhang1, Qianhui Yang1, Zhenji Kang1, Yi Shen1, Wenda Wang2
1 UniPat AI. 2 Elorian AI. * Equal contribution. ‡ Project leader. † Corresponding author.