🔍 Read the full analysis: Can LLMs Reliably Assess Whether A Research Idea Is Novel? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Ai2 researchers report that six large language models produced unstable judgments of scientific idea novelty across evaluation setups. In one pairwise test, small prompt wording changes shifted accuracy by more than 50 percentage points; adding literature retrieval or extra reasoning did not consistently improve results. Ai2 released the paper, benchmark data, code and an evaluation pipeline, while warning that some benchmark labels are limited.
Researchers at Ai2 report, in the original analysis, that six large language models gave unstable judgments of scientific idea novelty when prompts and evaluation setups changed. In one pairwise experiment, changing a few words in the prompt shifted accuracy by more than 50 percentage points, raising questions about using model scores to screen research proposals. Ai2 released the study, benchmark data, code and an evaluation pipeline for further testing.
The study tested GPT-5.1, GPT-5.2 and GPT-5.4, along with Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. Researchers evaluated the systems using both absolute novelty ratings and pairwise comparisons. They also report that adding literature retrieval or giving models more reasoning did not consistently improve their judgments. The findings suggest that a score from one chosen setup may not be enough to establish a dependable evaluation method.
For one benchmark, the team mined ICLR 2026 peer reviews for explicit comments about originality. The higher-novelty group contains 154 ideas from accepted papers in the top-rated 10% of their research areas, selected where reviewers praised novelty and none disputed it. The lower-novelty group contains 145 ideas from rejected papers in the bottom 10%, selected where reviewers raised originality concerns and none praised it. An annotator agreed with the extracted novelty-signal direction in 47 of 50 checked cases; the researchers say the three disagreements did not change review classifications.
A second test paired 154 human-authored ideas with 154 ideas from a basic language-model generator. The researchers treated generated proposals as less novel, but characterize those labels as weak and not individually verified. In a separate expert check of 30 pairs, selected heavily from cases the strongest judge got wrong, the expert preferred the human idea in every pair and judged none of the generated ideas novel. Ai2 also reports that two purpose-built novelty systems performed worse than a simple prompted Claude Opus 4.6 baseline.
Why Prompt Stability Matters
Research teams may use LLM judges to filter proposals before spending time on literature reviews, experiments or expert assessment. If a judgment changes sharply after a minor prompt edit, a score may reflect the evaluation setup as much as the idea being assessed. That makes comparisons between projects or research groups harder to interpret, particularly if they use different prompts, models or retrieval tools.
The findings also bear on efforts to automate parts of scientific discovery. Systems that generate hypotheses or synthesize literature need a credible way to distinguish genuinely new proposals from familiar ones. The study does not show that every LLM novelty evaluation fails; it shows that strong performance in one configuration alone may not establish reliable judgment across configurations. Teams considering these tools need to test sensitivity to setup choices and state how their evaluations were conducted.
Popular right now
Roborock Q7 M5+ Robot Vacuum and Mop
Philips 5500 Series Fully Automatic Espresso Machine with LatteGo
Dyson Cyclone V10 Origin Cordless Vacuum
As an affiliate, we earn on qualifying purchases.
How Ai2 Built Its Test Sets
Ai2 presents the work as part of a broader effort to develop systems that assist scientific discovery, including tools for exploring structured data, generating and testing hypotheses, and synthesizing research. A novelty judge is a model asked to assess whether an idea is new relative to prior work, either on its own or in comparison with another idea.
The authors say there is no standard recipe for these evaluations: researchers can choose different models, prompts, comparison formats, reasoning budgets and literature-retrieval systems. They also argue that judges are often tested on human-written papers even when the intended use is to assess AI-generated proposals. The benchmark therefore includes a human-and-generated setting alongside ideas selected using peer-review evidence. The authors caution that acceptance or rejection alone is not a sound novelty label, because papers can succeed or fail for reasons other than originality.
““Seemingly small changes to the evaluation setup produce dramatically different judgments of the same ideas.””
— Ai2 study authors
Limits of the Novelty Labels
The benchmark’s labels are evidence-based proxies, not a definitive measurement of originality. Its human-idea groups rely on reviewer comments and selection criteria, while the generated-idea group assumes lower novelty without checking every proposal individually. The expert review covered 30 pairs, not the full benchmark, and was selected heavily from cases the strongest judge had misclassified.
The supplied study description does not provide full results for every model, prompt, reasoning budget or evaluation format. It also does not establish whether the findings extend to other disciplines, different idea generators or systems with broader literature access. The authors say some conclusions rely on disagreement between evaluation setups rather than benchmark labels, but the available description does not specify how broadly that result holds. No specific follow-up study or release timeline is announced.
Further Tests Across Research Fields
Ai2 has released the paper, benchmark data, code and an automated pipeline intended to help researchers build additional evaluation sets. Other teams can use those materials to test whether prompt sensitivity and inconsistent performance appear across disciplines, idea sources and judge designs, and to compare approaches intended to make results more stable.
The release description does not name a next study or give a timetable. Until broader testing is available, the reported results support caution rather than a blanket conclusion: researchers should treat novelty scores as setup-dependent measurements and examine whether modest changes in their evaluation method alter the verdict.
Key Questions
What did the Ai2 study find?
Ai2 reports that six LLMs gave unstable novelty judgments across evaluation setups. In one pairwise experiment, changing a few prompt words shifted accuracy by more than 50 percentage points.
Did literature retrieval or extra reasoning improve the judgments?
The researchers report that adding literature retrieval or more reasoning did not consistently improve results. The supplied description does not give complete results for every tested configuration.
How did the researchers label the benchmark ideas?
One test set used peer-review comments about originality for ideas from selected ICLR 2026 papers. A second paired human-authored ideas with basic model-generated proposals, which the researchers treated as less novel while acknowledging that those labels were not individually verified.
Does the study prove LLMs cannot judge novelty?
No. The study reports sensitivity to evaluation choices, not that all LLM novelty judgments fail. The authors say more testing across settings is needed, and the benchmark labels and expert check have limits.
What happens next?
Ai2 has released its paper, data, code and evaluation pipeline. The supplied material names no scheduled follow-up; other researchers can use the release to test results across fields, idea sources and evaluation designs.
Primary source: Hugging Face · via ThorstenMeyerAI.com
