ScholarCatalyst: A Benchmark for Testing AI's Intuition in Retrieving Inspiring Research Papers
ScholarCatalyst：評估 AI 是否擁有「科學家直覺」的學術文獻檢索基準
While AI progresses in solving open problems, it lacks the scientific intuition to identify which prior papers can inspire new ideas. The authors present ScholarCatalyst, a benchmark built with 184 lead authors of 207 computer science papers who labeled the prior works that actually advanced their projects. Given an initial research question, models must retrieve these inspiring papers from the literature available at the project's start. Surprisingly, agentic search reached only 0.51 Recall@20 (using Claude Fable 5.1), barely outperforming simple embedding retrieval (0.48), exposing a major gap in expert-level literature search.
Key points
Testing Scientific Intuition
Traditional retrieval focuses on surface similarity, whereas ScholarCatalyst evaluates if AI can sense which buried prior ideas can solve a new research problem.
Ground-Truth from 184 Authors
Using a scalable pipeline, the creators gathered direct annotations and rationales from 184 lead authors of 207 recent computer science papers.
Agentic Search Underperforms
Agentic search scored only 0.42 Recall@20, failing to beat a standard embedding retriever (0.48) despite having access to it as a tool.
Top Models Struggle
Even Claude Fable 5.1, which might have seen the completed papers during training, only achieved 0.51 Recall@20, highlighting a gap in expert-level training.
How it works
Why it matters
This work highlights a critical bottleneck in AI for Science. Future scientific agents cannot rely on simple keyword or semantic similarity; they must grasp the deep conceptual links that spark scientific inspiration. ScholarCatalyst provides a challenging benchmark that will push RAG, agentic search, and specialized pre-training paradigms toward deeper logical reasoning and associative thinking.
Who it affects
- AI Developer
- AI Researcher
- Enterprise Leader
How to use it
- 1Evaluating academic AI agents on cross-disciplinary literature retrieval.
- 2Developing scientific assistants that recommend potential inspiring works based on half-formed ideas.
- 3Testing and improving RAG systems on complex, long-context retrieval reasoning tasks.
Limitations & caveats
- The dataset is primarily focused on computer science, which may not fully represent literature retrieval patterns in other fields like biology or physics.
- Annotations rely on retrospective self-reporting by authors, which might introduce subjective bias or memory gaps.
Related
Ai2 Open-Sources AstaBrief: An 8B Scientific Report Generator 3.5x Faster than Claude
艾倫人工智慧研究所開源 AstaBrief：比 Claude 快 3.5 倍的 8B 科學報告生成模型
Allen Institute for AI (Ai2) has open-sourced AstaBrief 8B, a specialized model for scientific report generation that achieves a 3.5x speedup over proprietary pipelines while maintaining high citation accuracy.
GALA: Distilling 3D Gaussian Avatars into Linear Blendshapes for Real-Time Animation
GALA：用線性混合變形蒸餾技術實現 3D Gaussian 虛擬化身即時動畫
GALA distills complex neural decoding of 3D Gaussian avatars into lightweight linear blendshapes, reducing CPU animation costs by up to 1000x and enabling 60fps real-time performance on mobile devices.
SILSA: Topology-Preserving High-Resolution 3D Generation with Sliding-Window Slice Latents
SILSA：利用滑動視窗切片潛在特徵實現保持拓撲結構的高解析度 3D 生成
SILSA introduces sliding-window slice latents to replace expensive voxel tokens, reducing token count by 70% while improving structural fidelity and topological consistency in high-resolution 3D generation.