arxiv:2609.10539

IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications

Published on Sep 9

· Submitted by

YilingMa on Sep 11

Yale NLP Lab

Upvote

Authors:

Abstract

The study introduces a benchmark to evaluate whether research methods are specified clearly enough for implementation, finding that identifying missing details is the primary challenge for language models.

Generated by thinkingmachines/Inkling-Small

A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implementation. We study the codification readiness of implementation-facing research-method specifications, defined by whether they provide sufficient methodological information for a competent implementer or coding agent to construct the intended method without unsupported assumptions. We construct evidence-grounded specifications and their supported resolutions from papers, codebases, issue threads, and reproduction artifacts. We introduce IdeaAMBIG, a benchmark of 660 evidence-grounded instances: 163 real-world gaps from reproducibility reports and GitHub issues, and 497 controlled synthetic gaps injected into codification-ready references. IdeaAMBIG evaluates three capabilities: codification-readiness assessment, defect localization, and clarification action generation. Defect localization receives only the specification, whereas clarification additionally receives the annotated defect. Across 13 LLMs, the best model achieves 9.6% Macro Defect Recovery Rate on real-world instances but 80.6% Macro Clarification Action Success Rate when given the defect. In an oracle study, supplying the gold resolution raises the downstream codification-ready rate from 14% to 98%. Across all evaluated models, defect localization is the main bottleneck, with stronger clarification given the defect.

View arXiv page View PDF GitHub 0 Add to collection

Community

YilingMa

Paper submitter about 20 hours ago

Excited to share IdeaAMBIG!
We ask a simple question: when an AI system is given a research idea, can it tell whether the idea is actually specified well enough to implement?
We introduce a benchmark of 660 evidence-grounded specification gaps and evaluate 13 LLMs on detecting, localizing, and resolving implementation-critical defects.
Our most striking result: LLMs are much better at fixing a gap than finding it. On real-world cases, the best model achieves only 9.6% defect recovery, but 80.6% clarification success once the defect is given. Providing the gold resolution raises downstream codification readiness from 14% to 98%.
This points to an important bottleneck for research agents: before asking models to implement an idea, we may first need them to reliably recognize what is missing, ambiguous, or inconsistent.
Would love to hear the community’s thoughts on this!