How Interesting Are AI Research Ideas? 100 Lab Leaders Rated 4,400
Image: "Mathematics (cryptography) on a blackboard" by David Malone (CC BY 4.0) via Wikimedia Commons_on_a_blackboard.jpg), recoloured.
Can a large language model propose research ideas that senior scientists actually find interesting? Gu and Krenn built SciMuse, which pairs a 58-million-paper knowledge graph with an LLM, and asked more than 100 research group leaders to rate 4,400 personalized ideas. Ratings were modest on average, but about one in four ideas scored high, and interest could be predicted.
Testing AI ideation against expert judgment
Most claims about AI as a creative partner in science rest on small demonstrations or automated metrics. Xuemei Gu and Mario Krenn, working at the Max Planck Institute for the Science of Light and the University of Tübingen, took a different route: they generated ideas tailored to specific researchers and then asked those researchers whether the ideas were worth their attention. Their system, SciMuse, draws on a knowledge graph built from 58 million scientific papers. It selects pairs of concepts linked to a researcher's own work and hands them to a large language model, which turns them into a concrete research proposal.
The evaluation is what makes the study unusual. More than 100 research group leaders, spanning the natural sciences through to the humanities, rated over 4,400 personalized ideas on a 5-point scale of interest. This produced a large interdisciplinary dataset of expert judgments on AI-generated ideas, something the field has largely lacked.
The headline result is sobering. According to Gu and Krenn (PNAS, 2026), the mean rating was 2.40 out of 5 and the most common rating was 1. Still, 24.9% of ideas received a 4 or 5, so a meaningful minority struck experts as genuinely interesting. Perhaps more surprising, feeding the LLM concept pairs chosen with the knowledge graph did not raise expert-rated interest above a simple baseline in which GPT saw only the titles of a researcher's papers. Concept pairs selected because they were predicted to attract many citations even showed a weak tendency (1.94 σ) toward lower interest than randomly chosen pairs.
The knowledge graph was not useless, however. The authors report that graph features can be used to steer properties of the generated ideas. Using their evaluation dataset, they also show that how interesting an idea will be can be predicted, both by a supervised neural network trained on graph features and by a zero-shot ranking approach in which an LLM orders the ideas. In practice, that points toward generating many candidates and filtering them, rather than expecting every output to be strong.
Some limits follow from the design. Interest is a single subjective dimension, and the paper frames it as a difficult-to-measure quality rather than a proxy for feasibility, correctness or eventual impact. The tendency for citation-optimized pairs to score lower is reported as weak, so it should be read as a signal worth testing rather than a firm rule.
Why it matters: For researchers experimenting with LLMs as brainstorming partners, this study offers a realistic baseline: most AI-generated ideas will not excite you, but roughly a quarter might, and ranking can surface them. It also cautions against assuming that more elaborate pipelines or citation-chasing heuristics produce better ideas. For those who study science itself, the expert-rated dataset opens a path to measuring scientific interestingness directly, a quantity that matters for how we choose questions but has rarely been measured at scale.
Gu, X., & Krenn, M. (2026). Generating interesting scientific ideas using knowledge graphs and LLMs: Evaluations with 100 research group leaders. Proceedings of the National Academy of Sciences. Read the original paper. Independent editorial summary of third-party research.
Comments