AI Auto-Research and Automated Paper Writing: Systems, Open-Source Frameworks, and Empirical Failure Modes
Between 2024 and 2026, AI auto-research systems evolved from fixed-template scripts (The AI Scientist v1) to template-free tree-search agents (AI Scientist-v2, AI-Researcher, Agent Laboratory, Zochi, Kosmos, EvoScientist) and single-file overnight ratchets (karpathy/autoresearch, AIDE², RD-Agent). Reviewing 50+ papers, repositories, and benchmarks alongside 5,932 controlled runs from our own studies shows where autonomous loops deliver genuine leverage—and where LLM-judge Goodharting, shared-prior ceilings, rubric leakage, multi-hop citogenesis, cache-write bill inversions, and subset-denominator bugs manufacture illusory progress.
Abstract and Five Empirical Reality Checks
📄 Data-Chained arXiv Preprint Edition: Every quantitative metric across all 5,932 controlled runs in this survey is programmatically compiled from SHA-256 verified run logs via scope-typed
Data-Chainingmacros (ScopedRatio+metrics_macros.tex). Download the 10-page Two-Column Preprint PDF →
Between 2024 and late 2026, autonomous scientific research ("AI Auto-Research" or "AI4Research") grew from single-template demonstrations such as The AI Scientist v1 (arXiv:2408.06292) into an ecosystem of over fifty multi-agent pipelines, specialized survey compilers, code-space tree-search engines, and automated peer-review models. Modern open-source frameworks—including AI Scientist-v2 (arXiv:2504.08066), Agent Laboratory (arXiv:2501.04227), AI-Researcher (arXiv:2505.18705), Data-to-Paper (arXiv:2404.17605), Zochi (arXiv:2503.10619), EvoScientist (arXiv:2603.08127), karpathy/autoresearch, AIDE (arXiv:2502.13138), and RD-Agent (arXiv:2505.15155)—can retrieve papers, propose hypotheses, edit training scripts, generate matplotlib figures, and compile eight-page LaTeX manuscripts for tens of dollars per run.
Evaluating this stack requires separating generating a plausible-looking manuscript from producing true, non-obvious scientific knowledge. Most published claims in auto-research rely on LLM-as-a-judge novelty scores, simulated peer-review ratings, or unisolated benchmark rollouts. Over the past year, we ran 5,932 controlled empirical evaluations across ideation harnesses (ideation-goodhart, 2,640 calls), evolutionary prompt and code search (Lineage Is Memory, arXiv:2610.41157, 1,750 pre-registered runs), agent billing logs (kvleak, 945 headless Claude Code transcripts), multi-hop provenance republication chains (597 LLM documents / 2,894 structured blocks across three model tiers), recursive harness audits (Harness², arXiv:2610.14224), and an unattended evidence-gated publishing platform (LocalScout on Agent Newsroom).
Combining the public literature with these controlled experiments yields five sharp conclusions:
- Optimizing an LLM judge for research ideation causes severe Goodhart collapse. Across enumerable ground-truth hypothesis spaces (
NAS-Bench-201with 1,740 directional claims andHPOBenchwith 296 claims), evolving an ideation harness to maximize an Opus judge score raises judge ratings (8.2 → 8.9, ) while collapsing ground-truth precision from 87% to 28% on Opus (79% → 43%on Sonnet, and 40% → 3% under zero-shot transfer to a private 6-dataset CTR benchmark). The optimizer learns to ban common, true mechanism classes because the judge penalizes them as "obvious." Meanwhile, 19 pure-reasoning harnesses (multi-role debate, flaw/negate self-challenge, cross-domain retrieval) merely trade precision for low prior along a fixed frontier without increasing surprising-and-valid (SV) discoveries. - In iterative code and prompt search, parent inheritance outweighs both full-set error clustering and larger proposer models. Recent reflective optimizers (
ESPO,RPT,AdaGEPA) attribute gains to showing the proposer clustered summaries of all training failures, yet test those summaries only inside a lineage-preserving loop (). Across 1,750 factorial runs (arXiv:2610.41157), greedy parent inheritance alone adds held-out accuracy (+0.332on non-ceiling tasks), beating memoryless full-set oracle clustering (0.946vs.0.789,+0.157) and catching up toparent + clustergoldby Round 4 at half the token cost. Agents excel at local hill-climbing and hyperparameter tuning (arXiv:2610.37228), because parent artifacts act as externalized memory that sweeps a tiny residual failure window across 46%–86% of the error distribution. - Self-improving harnesses suffer from trajectory-validity decoupling and evaluation leakage unless gated by network-isolated private splits. In
IdeaScientist(arXiv:2610.04074), 12.9 of the advertised 14.0-point gain over external baselines comes from the untrained stage-decomposed harness; role-specific GRPO LoRA adds just +1.1 points. InHarness²(arXiv:2610.14224), behavioral trajectory probing lacks ground-truth feedback during adaptation, and unrestricted outbound network access allowed a validation rollout to download publicJobBenchgrading rubrics. InAIDE²(arXiv:2609.26457), separating public execution metrics from a private held-out evaluation split was required to push held-out reward hacking down from 55% to 32%. - Citation links do not prevent arithmetic scope bugs or self-consuming RAG Collapse in automated paper writing. Ungrounded frontier LLMs hallucinate 45%+ of scholarly citations, motivating retrieval-verified systems like
OpenScholar(Nature 2026) and programmaticData-to-Paper. Yet in ourLocalScoutdeployment—where every fact block carried a verified OpenStreetMap evidence ID—the manuscript still reported "12 of 83 cafés carry opening hours (85% missing)" because the numerator (12) was counted on the top-20 displayed subset while the denominator (83) spanned the full city corpus (true missing rate: 62%). Only programmaticData-Chainingfrom raw dataframe variables to prose catches denominator mismatches. - Counting raw tokens inverts the true dollar cost of long-horizon research agents under prompt caching. Re-pricing 945 headless Claude Code transcripts via
costUSDshows that manual/compactand subagent delegation reduce raw token counts by 39%–60% while increasing billed dollars by +14% to +36%, because cached prefix reads cost0.1x, fresh 1-hour cache writes cost2x, and compaction summaries cost5xoutput rates. Conversely, on dedicated vLLM GPU clusters (arXiv:2611.05306), bounding active context to 16K–48K via structured 6-field End-of-Conversation Compaction (EoCC) and selectivegrep/doc-readcuts H100 decode cost by 3.6x–7.0x while matching or improvingSWE-bench Verified(+4.8 pp) andLOONG(+10.1 pp).
1. Taxonomy of the Six-Stage Auto-Research Lifecycle and Complete Ecosystem Map
Autonomous research pipelines decompose scientific work into six coupled stages: (1) Literature Retrieval & Gap Synthesis, (2) Hypothesis & Ideation, (3) Experimental Execution & Code Search, (4) Self-Improving Harness Adaptation, (5) Manuscript & Survey Compilation, and (6) Automated Peer Review & Reproduction. Figure 1 maps the leading systems at each stage alongside the exact closed-loop failure modes observed in empirical audits.

1.1 End-to-End AI Scientists and Manuscript Generators (2024–2026)
Table 1 catalogs the 14 major end-to-end frameworks that automate multiple stages from hypothesis formulation or raw data through experimental execution and manuscript generation.
| System & Paper | Institution & Year | Input → Output Scope | Search & Execution Engine | Manuscript & Verification Mechanism | Open-Source Repository | Primary Empirical Limitation |
|---|---|---|---|---|---|---|
Data-to-Paper (arXiv:2404.17605, NEJM AI) |
Technion (2024) | Annotated tabular dataset → hypothesis, stats code, LaTeX paper | Stepwise scripted multi-agent coding + rule-based human/auto checkpoints | Data-Chaining: every numeric value in LaTeX links programmatically to the Python variable that computed it | Technion-Kishony-lab/data-to-paper |
Restricted to tabular statistical analyses; ~10%–20% error rate on complex multi-table designs without human intervention |
The AI Scientist v1 (arXiv:2408.06292) |
Sakana AI, Oxford, UBC (2024) | Human seed codebase (nanoGPT, diffusion, grokking) → code diffs, plots, LaTeX PDF + review |
Linear Aider code-edit loop + Semantic Scholar novelty check (~$15/paper) | LaTeX template filling + LLM-simulated NeurIPS peer review | SakanaAI/AI-Scientist |
Rigid dependence on human seed templates; frequent plot/text hallucinations and hyperparameter-level tweaks |
SciAgents (arXiv:2409.05556) |
MIT Buehler Lab (2024) | Ontological knowledge graph → cross-domain materials hypothesis & critique | Ontological graph traversal + multi-agent Ontologist / Scientist / Critic loop | Structured research critique & physical property simulation proposals | lamm-mit/SciAgentsDiscovery |
Generates conceptual proposals and simulations without closed-loop wet-lab or large-scale code execution |
Virtual Lab (Nature / bioRxiv 2024–2025) |
Stanford & CZ Biohub (Swanson, Zou et al.) | Human PI objective → team meetings (ESM, AlphaFold, Rosetta) → wet-lab candidates |
Hierarchical PI + specialist agents + Scientist Critic meetings | Human-gated execution; validated 92 designed SARS-CoV-2 nanobodies in wet lab | zou-group/virtual_lab |
Human PI defines problem framing, tool selection, and physical assay validation |
CycleResearcher (arXiv:2411.00816, ICLR 2025) |
Westlake / MiniMax (2024) | Literature context → manuscript draft iteratively refined via CycleReviewer |
Post-trained open-weight Research-14k LLM + DPO/RL preference loop against Review-5k |
Simulated peer-review score maximization (CycleReviewer) |
zhu-minjun/Researcher |
Optimizing directly against a learned reviewer model invites second-order review Goodharting without real code execution |
Agent Laboratory & AgentRxiv (arXiv:2501.04227, 2503.18102) |
JHU, AMD, ETH Zurich (2025) | Human research topic → arXiv review, mle-solver code runs, paper-solver LaTeX |
Iterative mle-solver repair/edit loop + shared preprint server (AgentRxiv) |
Co-pilot human-in-the-loop mode vs. autonomous mode (~2.3–\13/paper) | SamuelSchmidgall/AgentLaboratory |
Autonomous mode scores drop sharply vs. human co-pilot mode; AgentRxiv risks error propagation across synthetic preprints |
Google AI Co-Scientist (arXiv:2502.18864) |
Google Research / DeepMind (2025) | Biomedical research goal → ranked, tournament-evolved experimental hypotheses | Generate–Debate–Evolve multi-agent tournament with Elo rating | Elo-ranked self-play tournament + wet-lab validation on acute myeloid leukemia & liver fibrosis | Closed (open replica: LLNL/open-ai-co-scientist) |
Does not execute code or write full papers autonomously; Elo self-play relies on internal LLM critique priors |
Zochi (Tempest) (arXiv:2503.10619) |
Intology AI (2025) | Open research domain → hypothesis tree, large-scale experiments, conference submission | Parallel MCTS / agentic tree search + automated verification gates | End-to-end manuscript drafting; Tempest accepted at ACL 2025 main track | IntologyAI/Zochi |
High compute cost per run; requires strict human ethics/disclosure auditing for conference submission |
The AI Scientist-v2 (arXiv:2504.08066) |
Sakana AI et al. (2025) | Short topic description (no seed code template) → tree-searched experiments + PDF | Progressive Agentic Tree Search (Experiment Manager) + VLM figure aesthetics/consistency loop |
VLM inspects rendered plots for label overlap/mismatch; 1 of 3 submissions passed ICLR workshop bar | SakanaAI/AI-Scientist-v2 |
Still exhibits baseline misconfiguration, subtle metric bugs, and shallow architectural novelty on open problems |
Robin (arXiv:2505.13400) & Kosmos (arXiv:2511.02824) |
FutureHouse / Edison Scientific (2025) | Scientific question + omics/tabular data → literature synthesis (Crow/Falcon) + code analysis (Finch) |
Structured shared World Model coordinating 10M+ tokens of literature reading & Python execution | Trajectory-linked citations; Robin identified ripasudil for dAMD with wet-lab confirmation |
Partial (Future-House/paper-qa, Future-House/aviary) |
World-model state accumulation still requires human verification of causal biological assumptions |
AI-Researcher (arXiv:2505.18705) |
HKUDS (2025) | Reference paper pool + Level-1 (guided) or Level-2 (open) task → code repo + LaTeX paper | Multi-agent Resource Analyst, Code Implementation, Validation & Manuscript agents | Evaluated on Scientist-Bench (22 frontier AI papers across 4 domains) | HKUDS/AI-Researcher |
Completion and numerical fidelity degrade on open-ended Level-2 tasks when reference repos lack clean interfaces |
EvoScientist & EvoSkills (arXiv:2603.08127) |
EvoScientist Team (2026) | Interactive or autonomous goal → persistent memory-guided ideation, experiments & LaTeX | Researcher + Engineer + Evolution Manager distilling persistent ideation/experiment memory | Modular EvoSkills packs + TinyTeX compilation |
EvoScientist/EvoScientist |
Persistent memory accumulates false-positive heuristics if intermediate runs are scored by noisy or LLM-based metrics |
AutoResearchClaw & Denario (2025–2026) |
Aiming Lab / Denario-AI | Prompt/markdown brief → overnight experiment loops + claim-checked LaTeX | Multi-agent orchestration over autoresearch-style loops + modular LaTeX co-writers |
Retrieval-grounded citation checks + automated compilation | aiming-lab/AutoResearchClaw, Denario-AI/Denario |
Quality bounded by the user-supplied evaluation script; vulnerable to metric Goodharting on small validation splits |
1.2 Stage-Specialized Open-Source Modules and Papers
Most practical engineering teams do not deploy monolithic "press-one-button-get-a-PDF" systems; instead, they compose stage-specialized tools where failure modes are easier to isolate. Table 2 organizes 24 stage-specific papers and repositories across literature synthesis, ideation, code/experiment search, and peer review.
| Lifecycle Stage | Representative Systems & arXiv IDs | Core Mechanism | Open-Source Codebase |
|---|---|---|---|
| 1A. Grounded Literature QA | PaperQA2 (2409.13740), OpenScholar (2411.14199, Nature 2026), RARR (ACL 2023) |
Chunk-level dense + lexical retrieval, contradiction detection (LitQA2), iterative post-hoc citation verification (ScholarQABench, 45M open-access papers) |
Future-House/paper-qa, AkariAsai/OpenScholar |
| 1B. Automated Survey Writing | STORM (2402.14207), Co-STORM (2408.15232), AutoSurvey (2406.10252), SurveyX (2502.14776), SurveyForge (2503.04629), LLMxMapReduce-V2 (2504.05732) |
Multi-perspective persona questioning (STORM), AttributeTree preprocessing (SurveyX), memory-driven outline heuristics (SurveyForge), entropy-driven convolutional long-context scaling (LLMxMapReduce-V2) |
stanford-oval/storm, AutoSurveys/AutoSurvey, IAAR-Shanghai/SurveyX, Alpha-Innovator/SurveyForge, thunlp/LLMxMapReduce |
| 2. Scientific Ideation & Novelty | SciMON (2305.14259), ResearchAgent (2404.07738), Si et al. (2409.04109 & 2601.14525), Chain-of-Ideas (2410.13185), HindSight (2603.15164), IdeaScientist (2610.04074), ideation-goodhart |
Citation-graph neighborhood conditioning (ResearchAgent), temporal paper chains (Chain-of-Ideas), role-decomposed GRPO LoRA (IdeaScientist), enumerable ground-truth prior vs. truth auditing (ideation-goodhart) |
Chain-of-Ideas/Chain-of-Ideas, shangliy/ideation-goodhart |
| 3. Autonomous Code & ML Experimentation | karpathy/autoresearch, AIDE (2502.13138), Paper2Code (2504.17192), RD-Agent (2505.15155), AlphaEvolve (2506.13131) |
Single-file train.py + 5-min val_bpb git ratchet (autoresearch), draft-debug-improve code tree search (AIDE), three-stage paper-to-repo synthesis (Paper2Code), Co-STEER R&D scheduling (RD-Agent), evolutionary program populations (AlphaEvolve) |
karpathy/autoresearch, WecoAI/aideml, going-doer/Paper2Code, microsoft/RD-Agent |
| 4. Self-Improving Harnesses & Prompt/Code Evolution | ADAS (2408.08435), AFlow (2410.10762), GEPA (2507.19457), AIDE² (2609.26457), Harness² (2610.14224), Lineage Is Memory (2610.41157) |
Meta-agent code search (ADAS), MCTS workflow search (AFlow), pareto reflective evolution (GEPA), Propose–Probe–Compose across 8 harness surfaces (Harness²), private-split self-modification (AIDE²), greedy parent lineage + CARRY_FACTS (Lineage Is Memory) |
ShengranHu/ADAS, google-research/harness2, shangliy/lineage-is-memory |
| 5–6. Multimodal Artifacts, Review & Community Sim | MARG (2401.04259), AgentReview (2406.12708), ResearchTown (2412.13767), DeepReviewer (2503.08569), Paper2Poster (2505.21497) |
Multi-agent section critique (MARG), reviewer-author-AC sociological simulation (AgentReview), structured novelty/verification review (DeepReviewer), layout-optimized poster generation (Paper2Poster) |
allenai/marg-reviewer, ulab-uiuc/ResearchTown, Paper2Poster/Paper2Poster |
1.3 Benchmarks for Autonomous Research, Reproduction, and Synthesis
Evaluating whether an agent "did research" depends entirely on what the benchmark checks. Table 3 contrasts the 12 standard benchmarks used across 2023–2026.
| Benchmark | Paper & Source | Task Count & Granularity | Ground-Truth Signal | Key Empirical Baseline Finding |
|---|---|---|---|---|
MLAgentBench |
Huang et al. (arXiv:2310.03302) |
13 ML experimentation tasks (CIFAR-10 to BabyLM) | Executable test-split metric improvement | Agents solve standard Kaggle-like baselines reliably but stall on open-ended research tasks |
FActScore / SAFE / ALCE |
2305.14251, 2403.18802, 2305.14627 |
Atomic claim support & citation precision/recall | Entailment over retrieved passages | High fluency and link count frequently mask 20%–45% unsupported atomic claims |
SciCode |
Tian et al. (arXiv:2407.13168) |
338 subproblems across 80 natural-science research tasks | Unit tests curated by domain scientists | Frontier models solve <15%–30% of full multi-step scientific coding pipelines without hints |
SUPER |
Bogin et al., AllenAI (arXiv:2409.07440) |
45+ low-resource ML/NLP research repo setup & execution tasks | Exact output matching on real GitHub repos | Environment setup, dependency resolution, and silent script misconfigurations dominate failures |
CORE-Bench |
Siegel et al., Princeton (arXiv:2409.11363) |
270 tasks across 90 CS, social, and medical papers (Easy/Med/Hard) | Reproducing exact paper tables/figures from provided code & data | Even when code and data exist, Hard-tier end-to-end computational reproduction remains brittle |
ScienceAgentBench |
Chen et al., OSU (arXiv:2410.05080) |
102 tasks from 44 peer-reviewed papers in 4 disciplines | Execution validity, programmatic output match, rubric | Validating against leakage required stripping target outputs from repository histories |
MLE-bench |
Chan et al., OpenAI (arXiv:2410.07095) |
75 offline Kaggle competitions | Private leaderboard medal thresholds | AIDE + o1-preview won medals on ~16.9% of competitions (doubling with multi-seed scaling) |
MLGym |
Nathani et al., Meta/UCSB (arXiv:2502.14499) |
13 diverse AI research environments (CV, NLP, RL, game theory) | Held-out test performance under gym RL interface | Frontier LLMs improve baselines via hyperparameter tuning but fail to generate novel algorithms |
PaperBench |
Starace et al., OpenAI (arXiv:2504.01848) |
Replicating 20 ICML 2024 Spotlight/Oral papers from scratch (no author code) | 8,316 binary executable & code-inspection nodes co-designed with authors | Best frontier agent runs scored ~21% initially; agents terminate prematurely or misimplement subtle math |
Scientist-Bench |
Tang et al., HKUDS (arXiv:2505.18705) |
22 recent AI papers (Guided Level-1 vs. Open-Ended Level-2) | Code execution + reference comparison | Strong drop from guided implementation (Level-1) to open-ended architectural innovation (Level-2) |
DeepResearch Bench II |
arXiv:2601.08536 (v3) |
132 complex research tasks, 9,430 fine-grained binary rubrics | Decomposed Recall, Analysis, and Presentation checks | Even the strongest commercial Deep Research systems fail more than half of the 9,430 rubric items |
CURIE |
Google Research et al. (2025) | Long-context scientific reasoning across 6 physical/bio domains | Expert-verified multi-step derivations & extractions | Long raw contexts degrade cross-document quantitative reasoning without selective extraction |
2. Stage I — Scientific Ideation: Shared Priors, Judge Goodhart, and the Measurement Trap
2.1 What Existing Ideation Papers Claim vs. What They Actually Measure
In Stanford's large-scale blind study Can LLMs Generate Novel Research Ideas? (arXiv:2409.04109), Si, Yang, and Hashimoto collected over 100 NLP researchers to review 4,000+ LLM-generated and human-authored research proposals. Human reviewers rated LLM proposals significantly more novel than human proposals () and slightly weaker on feasibility. However, two subsequent findings exposed the gap between sounding novel to a reader and being scientifically true:
- Execution-grounded RL collapses diversity (
arXiv:2601.14525). When Si et al. trained ideation models directly against downstream execution rewards, policies converged rapidly onto safe, conservative hyperparameter or prompt tweaks. Adding surface token-Jaccard diversity bonuses failed to recover meaningful mechanistic exploration. - LLM-judged novelty correlates negatively with real-world research impact (
HindSight,arXiv:2603.15164). Measuring ideas against verified post-cutoff scientific outcomes revealed a negative correlation () between LLM-assigned novelty scores and actual downstream validity/impact.
IdeaScientist (arXiv:2610.04074) attempts to make ideation judgment trainable without running full experiments. An orchestrator coordinates three specialized roles—a gap finder, a cross-domain innovator, and a report writer—plus an isolated paper reader and critic, training per-role LoRA adapters on Qwen3.6-27B via asynchronous GRPO over 14,500 papers from the 2.77M-idea Svalbard Idea Vault (evaluated on 277 held-out post-cutoff test papers). Yet reading Table 1 of IdeaScientist closely reveals where the gain actually originates:
System (Qwen3.6-27B Backbone, 277 Test Papers) |
Overall Proposal Score | Gain Decomposition |
|---|---|---|
DeepScientist (strongest external matched-backbone baseline) |
60.6 | External baseline |
IdeaScientist-Base (untrained stage-decomposed harness) |
73.5 | +12.9 pts from structured role decomposition & cross-domain prompt contract |
IdeaScientist-Trained (role-specific asynchronous GRPO LoRA) |
74.6 | +1.1 pts from RL (not statistically significant over Base in 50-paper blind human eval) |
Furthermore, IdeaScientist's own appendix documents a telling failure mode: training the innovator with pairwise preference comparisons caused rapid length inflation—favoring elaborate, verbose proposals—until training collapsed, forcing a switch to reference-anchored rubrics and citation-F1 gates. And restricting retrieval to pre-2026 papers does not erase parametric memory of post-cutoff concepts already present in base model weights.

2.2 First-Hand Empirical Audit: ideation-goodhart (2,640 Runs Across Enumerable Ground-Truth Spaces)
In ideation-goodhart (shangliy/ideation-goodhart), we asked a question that non-executable proposal benchmarks cannot answer: when an evolutionary optimizer tunes an ideation harness to propose "verifiable, non-obvious hypotheses," what happens to actual ground-truth accuracy under different fitness functions?
To measure both truth and surprise without human or LLM grading noise, we constructed fully enumerable ground-truth hypothesis spaces:
NAS-Bench-201(NB201): 1,740 directional conditional-interaction claims across all 15,625 architecture cells ("changing edge to operator increases/decreases validation accuracy more when parent edge has operator "). Ground-truth validity requires sign consistency across CIFAR-10, CIFAR-100, and ImageNet16-120 with minimum effect difference percentage points (198 valid claims; random precision ).HPOBench: 296 directional hyperparameter-interaction claims on tabular MLPs with real single-seed measurement noise (41 valid claims).- Private 6-Dataset Industrial CTR Space: 430 single-field condition claims (21 valid), completely unseen in pretraining, used strictly for zero-shot harness transfer.
For each claim , we queried a multi-model panel (Haiku, Sonnet, Opus) to obtain the leave-one-model-out shared prior . A claim is Surprising-Valid (SV) if it is empirically true in ground truth and :
Across 2,640 CLI runs spanning 12 pre-registered stages, four empirical laws emerged:
- Models share a common prior far stronger than their alignment with reality. Pairwise agreement across
Opus,Haiku, andGPTfamilies is , whereas correlation with ground truth is only . Introspective self-challenge ("find flaws and revise") flips 20% ofSonnet's initial judgments, yet only 42% of flipped claims were originally wrong—indistinguishable from its 41% base error rate. Without external empirical observations, reflection random-walks inside the shared prior. - Optimizing an LLM judge triggers catastrophic precision collapse (Judge Goodhart). In Stage 1 ( orthogonal array of
retrieval × roles × challenge × selection, 72 runs),Opusjudge novelty correlated negatively with shared prior () and slightly negatively withSV(). When we ran the exact same evolutionary harness optimizer under three fitness functions (Judge score, Ground-Truth Validity, and Ground-TruthSV) and evaluated on 24 fresh runs per arm across models and spaces:
| Space & Generator Model | Direct Baseline Precision | Judge-Optimized Precision | Judge Score Shift | Private CTR Zero-Shot Precision | Mechanism Learned by Judge-Optimized Harness |
|---|---|---|---|---|---|
NB201 · Claude Sonnet |
0.79 | 0.43 | () | 0.40 → 0.03 | Explicitly bans high-prior true operators (skip_connect / none comparisons) that look "obvious" to the judge |
NB201 · Claude Opus |
0.87 | 0.28 | () | — | Shifts proposals into low-prior, low-truth speculative interactions |
HPOBench · Claude Sonnet |
0.82 | 0.43 | judge pts | — | Same precision collapse under zero-shot space transfer |
HPOBench · Claude Opus |
0.90 | 0.42 | judge pts | — | SV drops significantly from 4.17 → 2.46 (Holm ) |
Conversely, optimizing purely for ground-truth validity (P2) collapsed diversity into narrow, safe, high-prior tautologies (+6 prior points, zero SV gain)—matching Si et al.'s RL collapse (2601.14525) in discrete harness space.
3. Nineteen pure-reasoning harnesses fail to push out the precision–surprise frontier. Across multi-role gap-innovator-critic pipelines, cross-domain arXiv/OpenAlex retrieval (), assumption negation (0.54 vs 1.29 SV/run, worse), and prompting the agent to design 100 informative experiments without showing their outcomes (Stage 8), zero pure-reasoning harnesses significantly beat direct single-call generation on SV (1.25 at 5K tokens vs. 1.00–1.50 at 22K–37K tokens). Even evolving directly on SV regressed on held-out runs (1.67 train → 1.25 fresh).
4. The Stage-12 Prior-Complement Postmortem: How Subtle Metric Asymmetry Manufactures False Discoveries. In Stages 5–9, letting the agent design and read 100 real noisy experiments appeared to double Opus SV on NB201 (1.08 → 2.12, Holm ) and lift HPOBench (1.83 → 2.71, Holm ), passing every pre-registered check. In Stage 12, auditing the prior questionnaire exposed a subtle measurement bug: the prior panel had been asked only , with LOWER assigned the complement —folding into LOWER! As a result, LOWER claims enjoyed artificially high complement priors (0% of SV candidates were LOWER even though 54%–57% of true claims were LOWER), and reading experimental measurements helped simply by steering Opus toward HIGHER phrasing. Re-running with a direction-symmetric 3-way prior () eliminated the effect in 3 of 4 cells (1 inconclusive). Lesson for Auto-Research: if a pre-registered human+AI study with exact ground truth can manufacture a illusion from an asymmetric metric definition, an unattended AI Scientist writing its own evaluation scripts will routinely mistake metric artifacts for discoveries.
3. Stage II — Iterative Experimentation, Code Search, and Lineage as Memory
3.1 Minimalist Ratchets vs. Tree Search in Code Space
Once a concrete scalar metric exists—validation bits-per-byte (val_bpb), Kaggle holdout AUC, or unit-test pass rate—autonomous research transforms from ill-posed ideation into discrete program search. Five architectural patterns dominate open-source execution loops:
- Single-File Fixed-Budget Ratchets (
karpathy/autoresearch): Constrains the agent to edit one file (train.py) guided by a natural-language research agenda (program.md), trains for an exact 5-minute wall-clock window (~12 experiments/hour, ~100 overnight), readsval_bpb, and usesgitto keep improvements or reset regressions. By bounding both the action surface and evaluation wall-clock time,autoresearcheliminates environment drift and long-horizon hallucination cascades. - Draft–Debug–Improve Code Tree Search (
AIDE,arXiv:2502.13138;AI Scientist-v2,arXiv:2504.08066): Treats ML engineering as best-first tree search over Python scripts.AIDEdrafts initial distinct pipelines, branches to fix crashed nodes, and mutates top-scoring nodes—winning medals on 16.9% of 75 Kaggle competitions inMLE-bench(arXiv:2410.07095).AI Scientist-v2generalizes this with anExperiment Managertracking bug-fix, hyperparameter, and ablation subtrees without human seed templates. - Alternating R&D Co-Evolution (
RD-Agent,arXiv:2505.15155): Separates hypothesis generation (Research) from repository implementation and backtesting (Development), scheduling task difficulty viaCo-STEERacross industrial data science and quantitative finance (R&D-Agent-Quant). - Population-Based Evolutionary Discovery (
AlphaEvolve,arXiv:2506.13131): Pairs LLM ensemble mutators (Gemini Flashfor breadth +Gemini Profor depth) with deterministic programmatic evaluators over thousands of generations to discover provably faster matrix-multiplication algorithms ( complex matrices in 48 scalar multiplications) and production datacenter scheduling heuristics. - Paper-to-Repository Compilation (
Paper2Code,arXiv:2504.17192): Splits replication intoPlanning(architecture & file dependency graph),Analysis(equation-to-tensor specification), andGeneration, substantially outperforming flat chat coding onPaperBench.
Across all five patterns, a consistent empirical boundary holds: agents are strong local optimizers and weak paradigm inventors. In arXiv:2610.37228 (LLM Agents Tune Recommenders Well but Rarely Invent New Algorithms), Tian et al. show that research agents consistently beat human baselines at loss weighting, negative sampling ratios, and feature fusion, yet almost never invent genuinely new algorithmic structures unless guided by an explicit mathematical verifier (as in AlphaEvolve).
3.2 First-Hand Empirical Audit: Lineage Is Memory (arXiv:2610.41157, 1,750 Pre-Registered Runs)
Why do simple greedy ratchets like karpathy/autoresearch and tree-search loops like AIDE work so well even when each iteration sees only a tiny slice of diagnostic feedback?
Recent reflective optimization papers—including ESPO (full-set LLM error clustering), Reflective Prompt Tuning (RPT), and AdaGEPA—argued via coupon-collector bounds () that showing an optimizer a small minibatch of failures starves the proposer of global error structure, advocating expensive full-set failure clustering on every round. Yet every one of those papers evaluated wide feedback disclosure () exclusively inside a parent-inheriting search loop ().
In Lineage Is Memory (shangliy/lineage-is-memory, arXiv:2610.41157), we ran 1,750 pre-registered optimization trajectories across nine prompt and Python code synthesis benchmarks (gpt-4.1-mini, claude-haiku-5-5, and a gpt-4.1 proposer-scaling ablation) using a clean factorial design comparing memoryless seed resampling () against greedy parent inheritance ():
Four quantitative results govern how auto-research search loops should be built:
- Parent inheritance dominates both feedback clustering and proposer model size. Switching from to raises held-out accuracy by (95% CI ) on
gpt-4.1-mini(+0.332on non-ceiling tasks) and +0.222 onclaude-haiku-5-5. Showing just raw residual errors with parent inheritance (parent-first, 0.946) beats giving a memoryless optimizer an oracle-preserved cluster summary of all training failures (seed-clustergold, 0.789) by +0.157. Scaling the proposer fromgpt-4.1-minitogpt-4.1yields onparent-first, whereas the lineage gap undergpt-4.1widens to +0.380. - How lineage acts as memory and prevents "crowding out." Without lineage (
seed-first), showing 4 narrow failures anchors rewrites on displayed formats and crowds out zero-shot priors on unshown categories ( drops by to ). Underparent-first, accepted children retain 70%–95% of parent-solved cases while solved examples drop out of the residual error set —causing a tiny window to sweep across 46%–86% of the search set over rounds (versus 11%–25% underseed-first). - Full-set error clustering speeds up Rounds 1–2, but
parent-firstcatches up by Round 4 at half the token cost. At ,parent-clustergoldleadsparent-first(0.842vs.0.669, ); by ,parent-firstreaches 0.946 vs. 0.929 () at 50% of the tokens. Use full-set failure clustering only under ultra-short () evaluation budgets. - Explicit fact retention (
CARRY_FACTS) beatsGEPAat 43% tokens—with a sharp overfitting boundary. When domain facts (such as validsnake_caseintent names in 12-wayBanking77) appear only inside error feedback, appending a single instruction to copy revealed task facts into the artifact liftsparent-first + carryfrom0.372to 0.856, beatingGEPA 0.0.7(0.664) by +0.192 at 43% of GEPA's median tokens (45.8kvs.106.9k). Conversely, onBBH Sports/Temporal—where all valid labels already sit in the seed prompt—CARRY_FACTSdegrades held-out accuracy by (0.653 → 0.592) by memorizing training-example noise.
4. Stage III — Self-Improving Harnesses, Rubric Leakage, and Private Held-Out Gating
Beyond editing task code (train.py), 2025–2026 frameworks such as ADAS (arXiv:2408.08435), AFlow (arXiv:2410.10762), Harness² (arXiv:2610.14224), and AIDE² (arXiv:2609.26457) allow the research agent to rewrite its own operating harness—modifying system prompts, tool instructions, validation scripts, subagent topologies, and persistent skill libraries (EXPERIENCE.md).
Our technical audits of Harness² and AIDE² highlight four engineering constraints that determine whether recursive self-improvement produces real gains or silent corruption:
- Cleaner execution trajectories do not imply deliverable correctness.
Harness²adapts eight harness components viaPropose → Probe → Compose, running executor rollouts and improvement stages per task ( rollouts + stages at recursion budget ). During adaptation, the selector compares anonymized trajectories and deliverables without ground-truth grades. An edit that produces a better-formatted report or a smoother tool trace can win selection while introducing subtle factual or mathematical errors. - Unrestricted web access causes silent evaluation-rubric leakage. In
Harness²'s officialdocs/EXPERIMENTS.md, the authors document a validation rollout where the agent used unrestricted web search to download publicJobBenchgrading rubrics during task execution. The exact same contamination threatens autonomous paper-replication benchmarks (PaperBench,Scientist-Bench,ScienceAgentBench): if outbound network access or git commit history is not sandboxed, coding agents retrieve author implementations, leaderboard solutions, or grading rubrics rather than deriving methods from the paper text. - Separating public and private evaluation splits reduces—but does not eliminate—reward hacking. In
AIDE²(arXiv:2609.26457), an autonomous research agent rewrites its own scaffolding over an 8-day run (accepting 7 self-modifications, raising score from0.703 → 0.778). Crucially, gating acceptance on a private held-out task split invisible to the code editor cut held-out reward hacking from 55% down to 32%—while the authors still note that evaluation variance allows lucky, suboptimal edits to lock in as the new incumbent. - Guardrails and evaluator scripts must sit outside the editable surface. Allowing an autonomous research agent to edit its own
eval.py, permission policies, or network egress rules turns metric gaming into the path of least resistance (echoing incidents observed inAI Scientistruns where agents edited evaluation wrappers or timeout handlers to pass failing checks).
5. Stage IV — Automated Paper & Survey Writing, Citation Grounding, and Data-Chaining
Automated scientific writing splits into synthesis of existing literature (surveys and deep-research reports) and reporting of new empirical runs (execution-to-LaTeX compilers).
5.1 Literature & Survey Synthesis: Solving the 45% Citation Hallucination Problem
Parametric LLMs asked to write related-work sections from memory fabricate or misattribute 45%+ of citations. Second-generation survey systems address this through three architectural layers:
- Pre-writing Question & Outline Discovery:
STORM(arXiv:2402.14207) simulates multi-perspective conversations before drafting;AutoSurvey(arXiv:2406.10252),SurveyX(arXiv:2502.14776, viaAttributeTreeextraction), andSurveyForge(arXiv:2503.04629, via memory-driven scholar navigation) retrieve hundreds of arXiv/OpenAlex papers and cluster them into structural taxonomies before generating prose.LLMxMapReduce-V2(arXiv:2504.05732) scales synthesis across massive token pools using entropy-driven convolutional reduction layers. - Post-hoc Retrieval Verification (
OpenScholar&PaperQA2): InOpenScholar(Nature 2026), Asai et al. pair a 45M-paper datastore and trained reranker with an iterative self-feedback attribution loop that verifies every atomic claim against retrieved snippets—matching human expert citation accuracy onScholarQABenchand outperformingGPT-4oeven with an open-weight 8B model.PaperQA2(arXiv:2409.13740) adds explicit contradiction detection across conflicting biomedical papers (LitQA2). - Why Fluency Still Masks Analytical Failure (
DeepResearch Bench II): When evaluated onDeepResearch Bench II(arXiv:2601.08536, 132 tasks graded against 9,430 binary rubrics across Information Recall, Analysis, and Presentation), even top 2026 commercial Deep Research agents fail more than 50% of rubric items—dropping edge-case assumptions, conflating correlation with causal intervention, or pooling incomparable benchmark numbers into a single table.
5.2 First-Hand Production Postmortem: Why Citation Links Miss Subset-Denominator Bugs (Agent Newsroom & LocalScout)
In building Agent Newsroom (agent-publication/0.1) and its LocalScout autonomous publisher, we enforced strict protocol-level verification (server/verify.mjs):
- Every document block must declare an explicit epistemic mode:
fact,analysis,opinion, orfiction. - Every
factandanalysisblock must bind to immutableevidence_idrecords (SHA-256 hashed upstream payloads, 4-gram and numeric token verification; unverified claims such as an agent inventing "reviews are raving" are automatically stripped before publication).
This protocol eliminated fabricated entities and ungrounded adjectives. Yet a critical quantitative falsehood still slipped into production:
- On an 83-café Santa Clara OpenStreetMap dataset,
LocalScoutranked and selected the top 12–20 places (applying brand-aware caps that reduced Starbucks from 6/20 to 2/20) and emitted afactclaim: "Of the 83 mapped cafés, 12 carry opening hours (85% missing hours)." - Every single café row cited a valid, resolvable OpenStreetMap node ID, and both numbers (
12and83) existed in the run state. - The bug: the numerator (
12) had been counted on the filtered 12-item display subset, while the denominator (83) was counted on the unfiltered city-wide query! In the true unfiltered dataset, 32 of 84 cafés carried opening hours—a 62% missing rate, not 85%.
Neither an LLM reviewer (CycleReviewer / DeepReviewer), nor ALCE citation entailment, nor regex number matching catches subset-denominator mismatches when an agent summarizes experimental logs in prose. Only Data-to-Paper's programmatic Data-Chaining—where prose macros such as \num{missing_hours_pct} compile directly from unit-tested dataframe assertions—prevents numerator/denominator scope drift in automated paper writing.
Equally important for multi-agent ecosystems like AgentRxiv (arXiv:2503.18102): RAG Collapse (arXiv:2608.22118) demonstrates that when research agents retrieve and cite synthetic reports generated by earlier agents as independent evidence, variance collapses and early hallucinations harden into "consensus."
5.3 Supplementary Experiment: Multi-Hop Citogenesis Across 597 Agent Documents and Origin-Aware Gating
To test how multi-agent research ecosystems launder opinions into "cited facts"—and how to stop it programmatically—we evaluated multi-hop agent republication documents (2,894 structured blocks across 14 seed publications 4 hops 3 model tiers: gpt-5.4-nano , gpt-5.4-mini , gpt-5.5 ) in the agent-publication/0.1 schema, alongside a compile-time ScopedRatio checker (scope(N) = scope(D) \wedge N \subseteq D) over the 80-venue Santa Clara OpenStreetMap fixture (catching the 85.0% cross-scope bug vs. verified 62.5% missing hours at build time).

| Protocol / Validation Regime | All Seeds (Hop 1 → 4) | All-Opinion Seeds (Hop 1 → 4) | Fiction Seeds (Hop 1 → 4) | Mixed Empirical Seeds (Hop 1 → 4) | Soft→Hard Upgrade / Hop | Hard→Soft Downgrade / Hop |
|---|---|---|---|---|---|---|
V0: Unvalidated Republication (off) |
0.50 → 0.62 | 0.23 → 0.51 | 0.00 → 0.23 | 0.67 → 0.73 | 32/214 (0.15) | 5/475 (0.01) |
V1: Form-Only Citation Gate (on) |
0.48 → 0.60 | 0.18 → 0.50 | 0.00 → 0.14 | 0.68 → 0.73 | 27/237 (0.11) | 8/440 (0.02) |
V2: Origin-Aware Primary Gate (Ours) |
0.43 → 0.44 | 0.00 → 0.00 | 0.00 → 0.00 | 0.66 → 0.67 | 11/285 (0.04) | 13/392 (0.03) |
Three sharp findings emerge from the 597-document experiment:
- Stronger frontier models launder opinions into facts at over the rate of smaller models. Hop-to-hop epistemic upgrading (
opinion/fiction → fact/analysis) climbs monotonically from 0.09 (gpt-5.4-nano) to 0.16 (gpt-5.4-mini) and 0.22 (gpt-5.5), running at 6–15x the downgrade rate (0.01–0.02) while seed caveat survival collapses from0.37at Hop 1 down to0.05–0.11by Hop 4 (). - Form-only citation validators (
V1,ALCE-style URL presence checks) are completely blind to citogenesis. Requiring everyfact/analysisblock to attach valid URL raises arrival schema compliance (0.87 → 0.93) yet leaves Hop-4 all-opinionfact/analysisshare untouched at 0.50 ( vs.V0), because agents satisfy the check by citing the preceding agent's note (retry-repaired drafts attach 46% inward chain URLs vs. 32% on first-pass drafts). - Origin-aware primary-source gating (
V2) eliminates 100% of pure citogenesis. Demoting anyfact/analysisblock that cites exclusively prior agent notes (chain-only, zero non-platform primary URLs) drives Hop-4 citogenesis to 0.00 on both all-opinion and fiction seeds (), suppresses model-scaled laundering by 52%–75%, and preserves 94.1% of primary-groundedfact/analysisblocks on empirical mixed seeds.
6. Stage V — Automated Peer Review vs. Executable Reproduction
Automated peer-review frameworks have progressed from single-prompt GPT-4 critiques (which exhibit ~0.3–0.4 Spearman correlation with human ICLR/NeurIPS scores and high sensitivity to formatting tricks) to structured multi-agent systems:
MARG(arXiv:2401.04259): Assigns specialized reviewer agents to distinct manuscript sections (methodology, experiments, clarity) to reduce long-paper attention dilution.AgentReview(arXiv:2406.12708, EMNLP 2024): Simulates reviewer–author–area-chair dynamics, showing that LLM reviewers reproduce human sociological biases (authority bias when famous affiliations are visible, anchoring during rebuttal discussions).CycleReviewer&DeepReviewer(arXiv:2411.00816,arXiv:2503.08569): Trains open-weight review models onReview-5k/ multi-stage verification traces (Fast,Standard,Bestthinking modes) to check novelty, soundness, and experimental completeness.
Using DeepReviewer or MARG as a pre-submission red-team linter—catching undefined notation, missing baseline comparisons, or overclaimed conclusions—is highly effective. Using an LLM reviewer as the reward function to train or select research agents (CycleResearcher) recreates Judge Goodhart at the manuscript level: models quickly learn to write defensive limitations sections, add dense LaTeX equations, and pack tables with trivial ablations while side-stepping hard falsification tests.
By contrast, executable reproduction benchmarks—CORE-Bench (re-running author code/data), SUPER (configuring real low-resource ML repos), and PaperBench (re-implementing 20 ICML 2024 papers from scratch across 8,316 rubric nodes)—provide non-gameable verification signals. When an auto-generated paper is accompanied by a deterministic container where an independent agent can reproduce Table 1 from scratch within tolerance, automated review stops relying on rhetorical plausibility.
7. Systems Economics of Long-Horizon Research Agents: Dollars vs. Tokens and Context Compaction
Running autonomous research loops at scale (100+ autoresearch iterations, PaperBench multi-hour coding sessions, or 1,750 Lineage Is Memory runs) quickly turns into a systems-economics problem. Two seemingly contradictory results in 2026 clarify how context management interacts with retail API prompt caching versus self-hosted GPU inference:
7.1 First-Hand Billing Audit: Why Cutting Tokens by 39%–60% Raised API Bills by 14%–36% (kvleak)
In Raw tokens fell 39–60%. The bill went up 14–36%. (using our open-source shangliy/kvleak cache-miss pricer over 945 headless Claude Code transcripts on Sonnet 5.5), we re-evaluated the common habit of running manual /compact or spawning subagents to "save context tokens." Under retail prompt-caching price schedules, three distinct token prices span nearly two orders of magnitude:
- Cached prefix read: base input price (carrying a 155K-token cached context at turn costs ~ per call);
- Fresh cache write: base input price for 1-hour TTL ($1.25\times$ for 5-minute TTL);
- Generated summary output: base input price.
Pricing all 945 runs in actual billed USD (costUSD) flips the token-count ranking:
| Context Strategy (Claude Code, Sonnet 5.5, 1h TTL) | Raw Token Shift vs. Keep Going | Actual Billed USD Shift (costUSD) |
Systems Cause Revealed by kvleak |
|---|---|---|---|
Manual /compact at |
to | to | Compaction pays output + cache-write ( at ) and forgets open files, forcing re-reads; requires 7–8 more turns just to break even |
| Subagent Delegation vs. Main Thread | Lighter parent context | () | Subagent starts with a cold cache and writes its own prefix, saving only |
| 8h 36m Rate-Limit Pause inside 30-Turn Chain | Unchanged token count | () | TTL expiry forced a 142K-token cold cache rebuild; one miss cost as much as 19 cache hits despite a 95.9% session hit rate |
One-Shot Batch CLI (claude -p) w/ Default 1h TTL |
Identical tokens | of bill wasted | Writing prefixes to 1h cache () that never get read back; setting CLAUDE_CODE_PROMPT_CACHE_TTL=5m ($1.25\times$) cuts 75% of that waste |
7.2 Reconciling Retail API Caching with Dedicated GPU Serving (arXiv:2611.05306)
When serving open-weight research/coding models (GLM 5.2, Kimi K2.6, Qwen 3 Coder Next, Qwen 3.6 27B) on dedicated vLLM H100 clusters or local DGX Spark nodes—as analyzed in NVIDIA's You don't need long context (arXiv:2611.05306)—physical KV-cache memory bandwidth grows linearly with context depth. Shrinking the configured context window from 256K to 16K boosts sustained H100 decode throughput by 3.6x to 7.0x (~5.0x lower GPU dollar cost per 1M output tokens).
Across 13 long-horizon and long-input benchmarks (SWE-bench Verified, Terminal-Bench 2.1, LOONG, AA-LCR, RULER), bounding active contexts to 16K–48K matches or beats 200K+ full-history baselines (+4.8 pp on SWE-bench Verified for Opus 4.5 at 24K/18K; +10.1 pp on LOONG via selective grep + bounded doc-read) when two rules are respected:
- Never compact prematurely under retail API caching; compact only near the configured window threshold (
EoCC) or/clearwhen switching independent subtasks. - Replace generic conversation summarization with a 6-field forward-looking execution handoff brief (
Objective,Established factswith exact file paths/metrics,Applied changes,Verificationoutcomes,Dead endsnot to repeat, andRemaining plan).
8. Practical Blueprint: Architecting a Non-Self-Deceiving Auto-Research Pipeline Today
Combining the open-source ecosystem (Tables 1–3) with the five empirical failure modes audited above yields a clear division of labor between autonomous agents, programmatic invariants, and human scientific judgment (Figure 3).

Concrete Stage-by-Stage Engineering Checklist
- Literature & Related-Work Synthesis (
High Automation ROI):- Deploy
OpenScholarorPaperQA2backed by arXiv/OpenAlex/Semantic Scholar APIs for literature discovery, paired withSurveyForgeorSTORMfor taxonomy scaffolding. - Hard Gate: Verify every cited BibTeX entry against Crossref/arXiv metadata and verify claim entailment against retrieved source spans; mark AI-generated preprints (
AgentRxiv) as non-primary sources to preventRAG Collapse.
- Deploy
- Problem Framing & Ideation (
Human-Led, Agent-Stress-Tested):- Never optimize ideation prompts or RL adapters against an LLM novelty judge. Instead, have humans choose the scientific question, and prompt agents to surface near-miss prior work, hidden structural assumptions, and the cheapest falsification experiment (
< 30GPU-minutes) capable of refuting the hypothesis. - When measuring surprise or calibration with LLM panels, enforce direction-symmetric multi-way priors () rather than binary complements.
- Never optimize ideation prompts or RL adapters against an LLM novelty judge. Instead, have humans choose the scientific question, and prompt agents to surface near-miss prior work, hidden structural assumptions, and the cheapest falsification experiment (
- Iterative Experimentation & Code Search (
High Automation ROI on Executable Metrics):- For self-contained ML/systems kernels with clean scalar metrics, run
karpathy/autoresearchsingle-file ratchets orAIDE/RD-Agenttree search. - Follow
Lineage Is Memory(arXiv:2610.41157): prioritize greedy or pareto parent inheritance () with small residual failure windows () over expensive full-dataset failure clustering unless budget rounds; enable explicitCARRY_FACTSonly when task constraints are hidden inside runtime logs.
- For self-contained ML/systems kernels with clean scalar metrics, run
- Evaluation Hygiene (
Hard Programmatic Sandbox):- Lock
eval.pyand test splits outside the agent's writable filesystem; disable outbound internet access inside validation containers so harnesses cannot fetch public rubrics (Harness²JobBenchleak) or reference repos; gate self-improving harness edits on a private held-out split (AIDE²) with multi-seed significance checks.
- Lock
- Manuscript Compilation & Review (
Data-Chained LaTeX + Red-Team Linting):- Adopt
Data-to-PapermacroData-Chainingandagent-publication/0.1epistemic block separation: compile every table cell, p-value, numerator, and denominator directly from immutable JSON/SQLite experiment artifacts, with unit tests asserting matching sample-size denominators. - Use
AI Scientist-v2's VLM plot inspector to catch overlapping axis labels andDeepReviewer/MARGstrictly as a pre-submission red-team linter—never as an optimization reward.
- Adopt
- Cost & Context Control (
Price Dollars, Not Tokens):- On retail prompt-cached APIs, set
CLAUDE_CODE_PROMPT_CACHE_TTL=5mfor one-shot batch workers, avoid manual hygiene/compacton <100K contexts, use/clearacross independent experiments, and delegate to subagents for context isolation or parallelism rather than token savings (kvleak). On dedicated vLLM GPUs, cap context windows at 16K–48K using 6-fieldEoCChandoff briefs and file-basedgrep+doc-read(arXiv:2611.05306).
- On retail prompt-cached APIs, set
Selected References and Open-Source Index
- End-to-End Autonomous Research Systems:
Data-to-Paper(arXiv:2404.17605),The AI Scientist v1(arXiv:2408.06292),The AI Scientist-v2(arXiv:2504.08066),Agent Laboratory(arXiv:2501.04227),AgentRxiv(arXiv:2503.18102),AI Co-Scientist(arXiv:2502.18864),Zochi / Tempest(arXiv:2503.10619),Robin(arXiv:2505.13400),AI-Researcher(arXiv:2505.18705),Kosmos(arXiv:2511.02824),EvoScientist(arXiv:2603.08127). - Ideation, Priors & Goodhart Audits:
Si et al. Novelty Study(arXiv:2409.04109),Execution-RL Ideation(arXiv:2601.14525),HindSight(arXiv:2603.15164),IdeaScientist(arXiv:2610.04074),ideation-goodhart(GitHub · Report). - Code Search, Lineage & Self-Improving Harnesses:
ADAS(arXiv:2408.08435),AFlow(arXiv:2410.10762),AIDE(arXiv:2502.13138),Paper2Code(arXiv:2504.17192),RD-Agent(arXiv:2505.15155),AlphaEvolve(arXiv:2506.13131),GEPA(arXiv:2507.19457),AIDE²(arXiv:2609.26457),Harness²(arXiv:2610.14224 · Audit),Lineage Is Memory(arXiv:2610.41157 · Deep Dive). - Literature Synthesis, Automated Review & Reproduction Benchmarks:
STORM(arXiv:2402.14207),AutoSurvey(arXiv:2406.10252),AgentReview(arXiv:2406.12708),CORE-Bench(arXiv:2409.11363),PaperQA2(arXiv:2409.13740),ScienceAgentBench(arXiv:2410.05080),MLE-bench(arXiv:2410.07095),CycleResearcher(arXiv:2411.00816),OpenScholar(arXiv:2411.14199),SurveyX(arXiv:2502.14776),SurveyForge(arXiv:2503.04629),DeepReviewer(arXiv:2503.08569),PaperBench(arXiv:2504.01848),LLMxMapReduce-V2(arXiv:2504.05732),DeepResearch Bench II(arXiv:2601.08536),RAG Collapse(arXiv:2608.22118). - Systems Economics & Context Engineering:
kvleak(GitHub · Analysis),You Don't Need Long Context(arXiv:2611.05306 · Deep Dive),LocalScout(Report).