AI Auto-Research and Automated Paper Writing: Systems, Open-Source Frameworks, and Empirical Failure Modes

Key Takeaway

Between 2024 and 2026, AI auto-research systems evolved from fixed-template scripts (The AI Scientist v1) to template-free tree-search agents (AI Scientist-v2, AI-Researcher, Agent Laboratory, Zochi, Kosmos, EvoScientist) and single-file overnight ratchets (karpathy/autoresearch, AIDE², RD-Agent). Reviewing 50+ papers, repositories, and benchmarks alongside 5,932 controlled runs from our own studies shows where autonomous loops deliver genuine leverage—and where LLM-judge Goodharting, shared-prior ceilings, rubric leakage, multi-hop citogenesis, cache-write bill inversions, and subset-denominator bugs manufacture illusory progress.

Abstract and Five Empirical Reality Checks

📄 Data-Chained arXiv Preprint Edition: Every quantitative metric across all 5,932 controlled runs in this survey is programmatically compiled from SHA-256 verified run logs via scope-typed Data-Chaining macros (ScopedRatio + metrics_macros.tex). Download the 10-page Two-Column Preprint PDF →

Between 2024 and late 2026, autonomous scientific research ("AI Auto-Research" or "AI4Research") grew from single-template demonstrations such as The AI Scientist v1 (arXiv:2408.06292) into an ecosystem of over fifty multi-agent pipelines, specialized survey compilers, code-space tree-search engines, and automated peer-review models. Modern open-source frameworks—including AI Scientist-v2 (arXiv:2504.08066), Agent Laboratory (arXiv:2501.04227), AI-Researcher (arXiv:2505.18705), Data-to-Paper (arXiv:2404.17605), Zochi (arXiv:2503.10619), EvoScientist (arXiv:2603.08127), karpathy/autoresearch, AIDE (arXiv:2502.13138), and RD-Agent (arXiv:2505.15155)—can retrieve papers, propose hypotheses, edit training scripts, generate matplotlib figures, and compile eight-page LaTeX manuscripts for tens of dollars per run.

Evaluating this stack requires separating generating a plausible-looking manuscript from producing true, non-obvious scientific knowledge. Most published claims in auto-research rely on LLM-as-a-judge novelty scores, simulated peer-review ratings, or unisolated benchmark rollouts. Over the past year, we ran 5,932 controlled empirical evaluations across ideation harnesses (ideation-goodhart, 2,640 calls), evolutionary prompt and code search (Lineage Is Memory, arXiv:2610.41157, 1,750 pre-registered runs), agent billing logs (kvleak, 945 headless Claude Code transcripts), multi-hop provenance republication chains (597 LLM documents / 2,894 structured blocks across three model tiers), recursive harness audits (Harness², arXiv:2610.14224), and an unattended evidence-gated publishing platform (LocalScout on Agent Newsroom).

Combining the public literature with these controlled experiments yields five sharp conclusions:

  1. Optimizing an LLM judge for research ideation causes severe Goodhart collapse. Across enumerable ground-truth hypothesis spaces (NAS-Bench-201 with 1,740 directional claims and HPOBench with 296 claims), evolving an ideation harness to maximize an Opus judge score raises judge ratings (8.2 → 8.9, ) while collapsing ground-truth precision from 87% to 28% on Opus (79% → 43% on Sonnet, and 40% → 3% under zero-shot transfer to a private 6-dataset CTR benchmark). The optimizer learns to ban common, true mechanism classes because the judge penalizes them as "obvious." Meanwhile, 19 pure-reasoning harnesses (multi-role debate, flaw/negate self-challenge, cross-domain retrieval) merely trade precision for low prior along a fixed frontier without increasing surprising-and-valid (SV) discoveries.
  2. In iterative code and prompt search, parent inheritance outweighs both full-set error clustering and larger proposer models. Recent reflective optimizers (ESPO, RPT, AdaGEPA) attribute gains to showing the proposer clustered summaries of all training failures, yet test those summaries only inside a lineage-preserving loop (). Across 1,750 factorial runs (arXiv:2610.41157), greedy parent inheritance alone adds held-out accuracy (+0.332 on non-ceiling tasks), beating memoryless full-set oracle clustering (0.946 vs. 0.789, +0.157) and catching up to parent + clustergold by Round 4 at half the token cost. Agents excel at local hill-climbing and hyperparameter tuning (arXiv:2610.37228), because parent artifacts act as externalized memory that sweeps a tiny residual failure window across 46%–86% of the error distribution.
  3. Self-improving harnesses suffer from trajectory-validity decoupling and evaluation leakage unless gated by network-isolated private splits. In IdeaScientist (arXiv:2610.04074), 12.9 of the advertised 14.0-point gain over external baselines comes from the untrained stage-decomposed harness; role-specific GRPO LoRA adds just +1.1 points. In Harness² (arXiv:2610.14224), behavioral trajectory probing lacks ground-truth feedback during adaptation, and unrestricted outbound network access allowed a validation rollout to download public JobBench grading rubrics. In AIDE² (arXiv:2609.26457), separating public execution metrics from a private held-out evaluation split was required to push held-out reward hacking down from 55% to 32%.
  4. Citation links do not prevent arithmetic scope bugs or self-consuming RAG Collapse in automated paper writing. Ungrounded frontier LLMs hallucinate 45%+ of scholarly citations, motivating retrieval-verified systems like OpenScholar (Nature 2026) and programmatic Data-to-Paper. Yet in our LocalScout deployment—where every fact block carried a verified OpenStreetMap evidence ID—the manuscript still reported "12 of 83 cafés carry opening hours (85% missing)" because the numerator (12) was counted on the top-20 displayed subset while the denominator (83) spanned the full city corpus (true missing rate: 62%). Only programmatic Data-Chaining from raw dataframe variables to prose catches denominator mismatches.
  5. Counting raw tokens inverts the true dollar cost of long-horizon research agents under prompt caching. Re-pricing 945 headless Claude Code transcripts via costUSD shows that manual /compact and subagent delegation reduce raw token counts by 39%–60% while increasing billed dollars by +14% to +36%, because cached prefix reads cost 0.1x, fresh 1-hour cache writes cost 2x, and compaction summaries cost 5x output rates. Conversely, on dedicated vLLM GPU clusters (arXiv:2611.05306), bounding active context to 16K–48K via structured 6-field End-of-Conversation Compaction (EoCC) and selective grep/doc-read cuts H100 decode cost by 3.6x–7.0x while matching or improving SWE-bench Verified (+4.8 pp) and LOONG (+10.1 pp).

1. Taxonomy of the Six-Stage Auto-Research Lifecycle and Complete Ecosystem Map

Autonomous research pipelines decompose scientific work into six coupled stages: (1) Literature Retrieval & Gap Synthesis, (2) Hypothesis & Ideation, (3) Experimental Execution & Code Search, (4) Self-Improving Harness Adaptation, (5) Manuscript & Survey Compilation, and (6) Automated Peer Review & Reproduction. Figure 1 maps the leading systems at each stage alongside the exact closed-loop failure modes observed in empirical audits.

Figure 1: The six-stage AI auto-research lifecycle (2023–2026), mapping representative open-source systems and papers at each stage alongside their primary closed-loop failure modes.

1.1 End-to-End AI Scientists and Manuscript Generators (2024–2026)

Table 1 catalogs the 14 major end-to-end frameworks that automate multiple stages from hypothesis formulation or raw data through experimental execution and manuscript generation.

System & Paper Institution & Year Input → Output Scope Search & Execution Engine Manuscript & Verification Mechanism Open-Source Repository Primary Empirical Limitation
Data-to-Paper (arXiv:2404.17605, NEJM AI) Technion (2024) Annotated tabular dataset → hypothesis, stats code, LaTeX paper Stepwise scripted multi-agent coding + rule-based human/auto checkpoints Data-Chaining: every numeric value in LaTeX links programmatically to the Python variable that computed it Technion-Kishony-lab/data-to-paper Restricted to tabular statistical analyses; ~10%–20% error rate on complex multi-table designs without human intervention
The AI Scientist v1 (arXiv:2408.06292) Sakana AI, Oxford, UBC (2024) Human seed codebase (nanoGPT, diffusion, grokking) → code diffs, plots, LaTeX PDF + review Linear Aider code-edit loop + Semantic Scholar novelty check (~$15/paper) LaTeX template filling + LLM-simulated NeurIPS peer review SakanaAI/AI-Scientist Rigid dependence on human seed templates; frequent plot/text hallucinations and hyperparameter-level tweaks
SciAgents (arXiv:2409.05556) MIT Buehler Lab (2024) Ontological knowledge graph → cross-domain materials hypothesis & critique Ontological graph traversal + multi-agent Ontologist / Scientist / Critic loop Structured research critique & physical property simulation proposals lamm-mit/SciAgentsDiscovery Generates conceptual proposals and simulations without closed-loop wet-lab or large-scale code execution
Virtual Lab (Nature / bioRxiv 2024–2025) Stanford & CZ Biohub (Swanson, Zou et al.) Human PI objective → team meetings (ESM, AlphaFold, Rosetta) → wet-lab candidates Hierarchical PI + specialist agents + Scientist Critic meetings Human-gated execution; validated 92 designed SARS-CoV-2 nanobodies in wet lab zou-group/virtual_lab Human PI defines problem framing, tool selection, and physical assay validation
CycleResearcher (arXiv:2411.00816, ICLR 2025) Westlake / MiniMax (2024) Literature context → manuscript draft iteratively refined via CycleReviewer Post-trained open-weight Research-14k LLM + DPO/RL preference loop against Review-5k Simulated peer-review score maximization (CycleReviewer) zhu-minjun/Researcher Optimizing directly against a learned reviewer model invites second-order review Goodharting without real code execution
Agent Laboratory & AgentRxiv (arXiv:2501.04227, 2503.18102) JHU, AMD, ETH Zurich (2025) Human research topic → arXiv review, mle-solver code runs, paper-solver LaTeX Iterative mle-solver repair/edit loop + shared preprint server (AgentRxiv) Co-pilot human-in-the-loop mode vs. autonomous mode (~2.3–\13/paper) SamuelSchmidgall/AgentLaboratory Autonomous mode scores drop sharply vs. human co-pilot mode; AgentRxiv risks error propagation across synthetic preprints
Google AI Co-Scientist (arXiv:2502.18864) Google Research / DeepMind (2025) Biomedical research goal → ranked, tournament-evolved experimental hypotheses Generate–Debate–Evolve multi-agent tournament with Elo rating Elo-ranked self-play tournament + wet-lab validation on acute myeloid leukemia & liver fibrosis Closed (open replica: LLNL/open-ai-co-scientist) Does not execute code or write full papers autonomously; Elo self-play relies on internal LLM critique priors
Zochi (Tempest) (arXiv:2503.10619) Intology AI (2025) Open research domain → hypothesis tree, large-scale experiments, conference submission Parallel MCTS / agentic tree search + automated verification gates End-to-end manuscript drafting; Tempest accepted at ACL 2025 main track IntologyAI/Zochi High compute cost per run; requires strict human ethics/disclosure auditing for conference submission
The AI Scientist-v2 (arXiv:2504.08066) Sakana AI et al. (2025) Short topic description (no seed code template) → tree-searched experiments + PDF Progressive Agentic Tree Search (Experiment Manager) + VLM figure aesthetics/consistency loop VLM inspects rendered plots for label overlap/mismatch; 1 of 3 submissions passed ICLR workshop bar SakanaAI/AI-Scientist-v2 Still exhibits baseline misconfiguration, subtle metric bugs, and shallow architectural novelty on open problems
Robin (arXiv:2505.13400) & Kosmos (arXiv:2511.02824) FutureHouse / Edison Scientific (2025) Scientific question + omics/tabular data → literature synthesis (Crow/Falcon) + code analysis (Finch) Structured shared World Model coordinating 10M+ tokens of literature reading & Python execution Trajectory-linked citations; Robin identified ripasudil for dAMD with wet-lab confirmation Partial (Future-House/paper-qa, Future-House/aviary) World-model state accumulation still requires human verification of causal biological assumptions
AI-Researcher (arXiv:2505.18705) HKUDS (2025) Reference paper pool + Level-1 (guided) or Level-2 (open) task → code repo + LaTeX paper Multi-agent Resource Analyst, Code Implementation, Validation & Manuscript agents Evaluated on Scientist-Bench (22 frontier AI papers across 4 domains) HKUDS/AI-Researcher Completion and numerical fidelity degrade on open-ended Level-2 tasks when reference repos lack clean interfaces
EvoScientist & EvoSkills (arXiv:2603.08127) EvoScientist Team (2026) Interactive or autonomous goal → persistent memory-guided ideation, experiments & LaTeX Researcher + Engineer + Evolution Manager distilling persistent ideation/experiment memory Modular EvoSkills packs + TinyTeX compilation EvoScientist/EvoScientist Persistent memory accumulates false-positive heuristics if intermediate runs are scored by noisy or LLM-based metrics
AutoResearchClaw & Denario (2025–2026) Aiming Lab / Denario-AI Prompt/markdown brief → overnight experiment loops + claim-checked LaTeX Multi-agent orchestration over autoresearch-style loops + modular LaTeX co-writers Retrieval-grounded citation checks + automated compilation aiming-lab/AutoResearchClaw, Denario-AI/Denario Quality bounded by the user-supplied evaluation script; vulnerable to metric Goodharting on small validation splits

1.2 Stage-Specialized Open-Source Modules and Papers

Most practical engineering teams do not deploy monolithic "press-one-button-get-a-PDF" systems; instead, they compose stage-specialized tools where failure modes are easier to isolate. Table 2 organizes 24 stage-specific papers and repositories across literature synthesis, ideation, code/experiment search, and peer review.

Lifecycle Stage Representative Systems & arXiv IDs Core Mechanism Open-Source Codebase
1A. Grounded Literature QA PaperQA2 (2409.13740), OpenScholar (2411.14199, Nature 2026), RARR (ACL 2023) Chunk-level dense + lexical retrieval, contradiction detection (LitQA2), iterative post-hoc citation verification (ScholarQABench, 45M open-access papers) Future-House/paper-qa, AkariAsai/OpenScholar
1B. Automated Survey Writing STORM (2402.14207), Co-STORM (2408.15232), AutoSurvey (2406.10252), SurveyX (2502.14776), SurveyForge (2503.04629), LLMxMapReduce-V2 (2504.05732) Multi-perspective persona questioning (STORM), AttributeTree preprocessing (SurveyX), memory-driven outline heuristics (SurveyForge), entropy-driven convolutional long-context scaling (LLMxMapReduce-V2) stanford-oval/storm, AutoSurveys/AutoSurvey, IAAR-Shanghai/SurveyX, Alpha-Innovator/SurveyForge, thunlp/LLMxMapReduce
2. Scientific Ideation & Novelty SciMON (2305.14259), ResearchAgent (2404.07738), Si et al. (2409.04109 & 2601.14525), Chain-of-Ideas (2410.13185), HindSight (2603.15164), IdeaScientist (2610.04074), ideation-goodhart Citation-graph neighborhood conditioning (ResearchAgent), temporal paper chains (Chain-of-Ideas), role-decomposed GRPO LoRA (IdeaScientist), enumerable ground-truth prior vs. truth auditing (ideation-goodhart) Chain-of-Ideas/Chain-of-Ideas, shangliy/ideation-goodhart
3. Autonomous Code & ML Experimentation karpathy/autoresearch, AIDE (2502.13138), Paper2Code (2504.17192), RD-Agent (2505.15155), AlphaEvolve (2506.13131) Single-file train.py + 5-min val_bpb git ratchet (autoresearch), draft-debug-improve code tree search (AIDE), three-stage paper-to-repo synthesis (Paper2Code), Co-STEER R&D scheduling (RD-Agent), evolutionary program populations (AlphaEvolve) karpathy/autoresearch, WecoAI/aideml, going-doer/Paper2Code, microsoft/RD-Agent
4. Self-Improving Harnesses & Prompt/Code Evolution ADAS (2408.08435), AFlow (2410.10762), GEPA (2507.19457), AIDE² (2609.26457), Harness² (2610.14224), Lineage Is Memory (2610.41157) Meta-agent code search (ADAS), MCTS workflow search (AFlow), pareto reflective evolution (GEPA), Propose–Probe–Compose across 8 harness surfaces (Harness²), private-split self-modification (AIDE²), greedy parent lineage + CARRY_FACTS (Lineage Is Memory) ShengranHu/ADAS, google-research/harness2, shangliy/lineage-is-memory
5–6. Multimodal Artifacts, Review & Community Sim MARG (2401.04259), AgentReview (2406.12708), ResearchTown (2412.13767), DeepReviewer (2503.08569), Paper2Poster (2505.21497) Multi-agent section critique (MARG), reviewer-author-AC sociological simulation (AgentReview), structured novelty/verification review (DeepReviewer), layout-optimized poster generation (Paper2Poster) allenai/marg-reviewer, ulab-uiuc/ResearchTown, Paper2Poster/Paper2Poster

1.3 Benchmarks for Autonomous Research, Reproduction, and Synthesis

Evaluating whether an agent "did research" depends entirely on what the benchmark checks. Table 3 contrasts the 12 standard benchmarks used across 2023–2026.

Benchmark Paper & Source Task Count & Granularity Ground-Truth Signal Key Empirical Baseline Finding
MLAgentBench Huang et al. (arXiv:2310.03302) 13 ML experimentation tasks (CIFAR-10 to BabyLM) Executable test-split metric improvement Agents solve standard Kaggle-like baselines reliably but stall on open-ended research tasks
FActScore / SAFE / ALCE 2305.14251, 2403.18802, 2305.14627 Atomic claim support & citation precision/recall Entailment over retrieved passages High fluency and link count frequently mask 20%–45% unsupported atomic claims
SciCode Tian et al. (arXiv:2407.13168) 338 subproblems across 80 natural-science research tasks Unit tests curated by domain scientists Frontier models solve <15%–30% of full multi-step scientific coding pipelines without hints
SUPER Bogin et al., AllenAI (arXiv:2409.07440) 45+ low-resource ML/NLP research repo setup & execution tasks Exact output matching on real GitHub repos Environment setup, dependency resolution, and silent script misconfigurations dominate failures
CORE-Bench Siegel et al., Princeton (arXiv:2409.11363) 270 tasks across 90 CS, social, and medical papers (Easy/Med/Hard) Reproducing exact paper tables/figures from provided code & data Even when code and data exist, Hard-tier end-to-end computational reproduction remains brittle
ScienceAgentBench Chen et al., OSU (arXiv:2410.05080) 102 tasks from 44 peer-reviewed papers in 4 disciplines Execution validity, programmatic output match, rubric Validating against leakage required stripping target outputs from repository histories
MLE-bench Chan et al., OpenAI (arXiv:2410.07095) 75 offline Kaggle competitions Private leaderboard medal thresholds AIDE + o1-preview won medals on ~16.9% of competitions (doubling with multi-seed scaling)
MLGym Nathani et al., Meta/UCSB (arXiv:2502.14499) 13 diverse AI research environments (CV, NLP, RL, game theory) Held-out test performance under gym RL interface Frontier LLMs improve baselines via hyperparameter tuning but fail to generate novel algorithms
PaperBench Starace et al., OpenAI (arXiv:2504.01848) Replicating 20 ICML 2024 Spotlight/Oral papers from scratch (no author code) 8,316 binary executable & code-inspection nodes co-designed with authors Best frontier agent runs scored ~21% initially; agents terminate prematurely or misimplement subtle math
Scientist-Bench Tang et al., HKUDS (arXiv:2505.18705) 22 recent AI papers (Guided Level-1 vs. Open-Ended Level-2) Code execution + reference comparison Strong drop from guided implementation (Level-1) to open-ended architectural innovation (Level-2)
DeepResearch Bench II arXiv:2601.08536 (v3) 132 complex research tasks, 9,430 fine-grained binary rubrics Decomposed Recall, Analysis, and Presentation checks Even the strongest commercial Deep Research systems fail more than half of the 9,430 rubric items
CURIE Google Research et al. (2025) Long-context scientific reasoning across 6 physical/bio domains Expert-verified multi-step derivations & extractions Long raw contexts degrade cross-document quantitative reasoning without selective extraction

2. Stage I — Scientific Ideation: Shared Priors, Judge Goodhart, and the Measurement Trap

2.1 What Existing Ideation Papers Claim vs. What They Actually Measure

In Stanford's large-scale blind study Can LLMs Generate Novel Research Ideas? (arXiv:2409.04109), Si, Yang, and Hashimoto collected over 100 NLP researchers to review 4,000+ LLM-generated and human-authored research proposals. Human reviewers rated LLM proposals significantly more novel than human proposals () and slightly weaker on feasibility. However, two subsequent findings exposed the gap between sounding novel to a reader and being scientifically true:

  1. Execution-grounded RL collapses diversity (arXiv:2601.14525). When Si et al. trained ideation models directly against downstream execution rewards, policies converged rapidly onto safe, conservative hyperparameter or prompt tweaks. Adding surface token-Jaccard diversity bonuses failed to recover meaningful mechanistic exploration.
  2. LLM-judged novelty correlates negatively with real-world research impact (HindSight, arXiv:2603.15164). Measuring ideas against verified post-cutoff scientific outcomes revealed a negative correlation () between LLM-assigned novelty scores and actual downstream validity/impact.

IdeaScientist (arXiv:2610.04074) attempts to make ideation judgment trainable without running full experiments. An orchestrator coordinates three specialized roles—a gap finder, a cross-domain innovator, and a report writer—plus an isolated paper reader and critic, training per-role LoRA adapters on Qwen3.6-27B via asynchronous GRPO over 14,500 papers from the 2.77M-idea Svalbard Idea Vault (evaluated on 277 held-out post-cutoff test papers). Yet reading Table 1 of IdeaScientist closely reveals where the gain actually originates:

System (Qwen3.6-27B Backbone, 277 Test Papers) Overall Proposal Score Gain Decomposition
DeepScientist (strongest external matched-backbone baseline) 60.6 External baseline
IdeaScientist-Base (untrained stage-decomposed harness) 73.5 +12.9 pts from structured role decomposition & cross-domain prompt contract
IdeaScientist-Trained (role-specific asynchronous GRPO LoRA) 74.6 +1.1 pts from RL (not statistically significant over Base in 50-paper blind human eval)

Furthermore, IdeaScientist's own appendix documents a telling failure mode: training the innovator with pairwise preference comparisons caused rapid length inflation—favoring elaborate, verbose proposals—until training collapsed, forcing a switch to reference-anchored rubrics and citation-F1 gates. And restricting retrieval to pre-2026 papers does not erase parametric memory of post-cutoff concepts already present in base model weights.

Figure 2: Quantitative results across 5,335+ controlled empirical runs auditing autonomous research loops: (Exp 1) Judge Goodhart precision collapse in ideation-goodhart, (Exp 2) Parent inheritance vs. full-set clustering in Lineage Is Memory, (Exp 3) Raw token vs. dollar bill inversion in kvleak, and (Exp 4) Harness vs. RL and private held-out gating in IdeaScientist and AIDE².

2.2 First-Hand Empirical Audit: ideation-goodhart (2,640 Runs Across Enumerable Ground-Truth Spaces)

In ideation-goodhart (shangliy/ideation-goodhart), we asked a question that non-executable proposal benchmarks cannot answer: when an evolutionary optimizer tunes an ideation harness to propose "verifiable, non-obvious hypotheses," what happens to actual ground-truth accuracy under different fitness functions?

To measure both truth and surprise without human or LLM grading noise, we constructed fully enumerable ground-truth hypothesis spaces:

  • NAS-Bench-201 (NB201): 1,740 directional conditional-interaction claims across all 15,625 architecture cells ("changing edge to operator increases/decreases validation accuracy more when parent edge has operator "). Ground-truth validity requires sign consistency across CIFAR-10, CIFAR-100, and ImageNet16-120 with minimum effect difference percentage points (198 valid claims; random precision ).
  • HPOBench: 296 directional hyperparameter-interaction claims on tabular MLPs with real single-seed measurement noise (41 valid claims).
  • Private 6-Dataset Industrial CTR Space: 430 single-field condition claims (21 valid), completely unseen in pretraining, used strictly for zero-shot harness transfer.

For each claim , we queried a multi-model panel (Haiku, Sonnet, Opus) to obtain the leave-one-model-out shared prior . A claim is Surprising-Valid (SV) if it is empirically true in ground truth and :

Across 2,640 CLI runs spanning 12 pre-registered stages, four empirical laws emerged:

  1. Models share a common prior far stronger than their alignment with reality. Pairwise agreement across Opus, Haiku, and GPT families is , whereas correlation with ground truth is only . Introspective self-challenge ("find flaws and revise") flips 20% of Sonnet's initial judgments, yet only 42% of flipped claims were originally wrong—indistinguishable from its 41% base error rate. Without external empirical observations, reflection random-walks inside the shared prior.
  2. Optimizing an LLM judge triggers catastrophic precision collapse (Judge Goodhart). In Stage 1 ( orthogonal array of retrieval × roles × challenge × selection, 72 runs), Opus judge novelty correlated negatively with shared prior () and slightly negatively with SV (). When we ran the exact same evolutionary harness optimizer under three fitness functions (Judge score, Ground-Truth Validity, and Ground-Truth SV) and evaluated on 24 fresh runs per arm across models and spaces:
Space & Generator Model Direct Baseline Precision Judge-Optimized Precision Judge Score Shift Private CTR Zero-Shot Precision Mechanism Learned by Judge-Optimized Harness
NB201 · Claude Sonnet 0.79 0.43 () 0.40 → 0.03 Explicitly bans high-prior true operators (skip_connect / none comparisons) that look "obvious" to the judge
NB201 · Claude Opus 0.87 0.28 () — Shifts proposals into low-prior, low-truth speculative interactions
HPOBench · Claude Sonnet 0.82 0.43 judge pts — Same precision collapse under zero-shot space transfer
HPOBench · Claude Opus 0.90 0.42 judge pts — SV drops significantly from 4.17 → 2.46 (Holm )

Conversely, optimizing purely for ground-truth validity (P2) collapsed diversity into narrow, safe, high-prior tautologies (+6 prior points, zero SV gain)—matching Si et al.'s RL collapse (2601.14525) in discrete harness space. 3. Nineteen pure-reasoning harnesses fail to push out the precision–surprise frontier. Across multi-role gap-innovator-critic pipelines, cross-domain arXiv/OpenAlex retrieval (), assumption negation (0.54 vs 1.29 SV/run, worse), and prompting the agent to design 100 informative experiments without showing their outcomes (Stage 8), zero pure-reasoning harnesses significantly beat direct single-call generation on SV (1.25 at 5K tokens vs. 1.00–1.50 at 22K–37K tokens). Even evolving directly on SV regressed on held-out runs (1.67 train → 1.25 fresh). 4. The Stage-12 Prior-Complement Postmortem: How Subtle Metric Asymmetry Manufactures False Discoveries. In Stages 5–9, letting the agent design and read 100 real noisy experiments appeared to double Opus SV on NB201 (1.08 → 2.12, Holm ) and lift HPOBench (1.83 → 2.71, Holm ), passing every pre-registered check. In Stage 12, auditing the prior questionnaire exposed a subtle measurement bug: the prior panel had been asked only , with LOWER assigned the complement —folding into LOWER! As a result, LOWER claims enjoyed artificially high complement priors (0% of SV candidates were LOWER even though 54%–57% of true claims were LOWER), and reading experimental measurements helped simply by steering Opus toward HIGHER phrasing. Re-running with a direction-symmetric 3-way prior () eliminated the effect in 3 of 4 cells (1 inconclusive). Lesson for Auto-Research: if a pre-registered human+AI study with exact ground truth can manufacture a illusion from an asymmetric metric definition, an unattended AI Scientist writing its own evaluation scripts will routinely mistake metric artifacts for discoveries.


3. Stage II — Iterative Experimentation, Code Search, and Lineage as Memory

3.1 Minimalist Ratchets vs. Tree Search in Code Space

Once a concrete scalar metric exists—validation bits-per-byte (val_bpb), Kaggle holdout AUC, or unit-test pass rate—autonomous research transforms from ill-posed ideation into discrete program search. Five architectural patterns dominate open-source execution loops:

  1. Single-File Fixed-Budget Ratchets (karpathy/autoresearch): Constrains the agent to edit one file (train.py) guided by a natural-language research agenda (program.md), trains for an exact 5-minute wall-clock window (~12 experiments/hour, ~100 overnight), reads val_bpb, and uses git to keep improvements or reset regressions. By bounding both the action surface and evaluation wall-clock time, autoresearch eliminates environment drift and long-horizon hallucination cascades.
  2. Draft–Debug–Improve Code Tree Search (AIDE, arXiv:2502.13138; AI Scientist-v2, arXiv:2504.08066): Treats ML engineering as best-first tree search over Python scripts. AIDE drafts initial distinct pipelines, branches to fix crashed nodes, and mutates top-scoring nodes—winning medals on 16.9% of 75 Kaggle competitions in MLE-bench (arXiv:2410.07095). AI Scientist-v2 generalizes this with an Experiment Manager tracking bug-fix, hyperparameter, and ablation subtrees without human seed templates.
  3. Alternating R&D Co-Evolution (RD-Agent, arXiv:2505.15155): Separates hypothesis generation (Research) from repository implementation and backtesting (Development), scheduling task difficulty via Co-STEER across industrial data science and quantitative finance (R&D-Agent-Quant).
  4. Population-Based Evolutionary Discovery (AlphaEvolve, arXiv:2506.13131): Pairs LLM ensemble mutators (Gemini Flash for breadth + Gemini Pro for depth) with deterministic programmatic evaluators over thousands of generations to discover provably faster matrix-multiplication algorithms ( complex matrices in 48 scalar multiplications) and production datacenter scheduling heuristics.
  5. Paper-to-Repository Compilation (Paper2Code, arXiv:2504.17192): Splits replication into Planning (architecture & file dependency graph), Analysis (equation-to-tensor specification), and Generation, substantially outperforming flat chat coding on PaperBench.

Across all five patterns, a consistent empirical boundary holds: agents are strong local optimizers and weak paradigm inventors. In arXiv:2610.37228 (LLM Agents Tune Recommenders Well but Rarely Invent New Algorithms), Tian et al. show that research agents consistently beat human baselines at loss weighting, negative sampling ratios, and feature fusion, yet almost never invent genuinely new algorithmic structures unless guided by an explicit mathematical verifier (as in AlphaEvolve).

3.2 First-Hand Empirical Audit: Lineage Is Memory (arXiv:2610.41157, 1,750 Pre-Registered Runs)

Why do simple greedy ratchets like karpathy/autoresearch and tree-search loops like AIDE work so well even when each iteration sees only a tiny slice of diagnostic feedback?

Recent reflective optimization papers—including ESPO (full-set LLM error clustering), Reflective Prompt Tuning (RPT), and AdaGEPA—argued via coupon-collector bounds () that showing an optimizer a small minibatch of failures starves the proposer of global error structure, advocating expensive full-set failure clustering on every round. Yet every one of those papers evaluated wide feedback disclosure () exclusively inside a parent-inheriting search loop ().

In Lineage Is Memory (shangliy/lineage-is-memory, arXiv:2610.41157), we ran 1,750 pre-registered optimization trajectories across nine prompt and Python code synthesis benchmarks (gpt-4.1-mini, claude-haiku-5-5, and a gpt-4.1 proposer-scaling ablation) using a clean factorial design comparing memoryless seed resampling () against greedy parent inheritance ():

Four quantitative results govern how auto-research search loops should be built:

  1. Parent inheritance dominates both feedback clustering and proposer model size. Switching from to raises held-out accuracy by (95% CI ) on gpt-4.1-mini (+0.332 on non-ceiling tasks) and +0.222 on claude-haiku-5-5. Showing just raw residual errors with parent inheritance (parent-first, 0.946) beats giving a memoryless optimizer an oracle-preserved cluster summary of all training failures (seed-clustergold, 0.789) by +0.157. Scaling the proposer from gpt-4.1-mini to gpt-4.1 yields on parent-first, whereas the lineage gap under gpt-4.1 widens to +0.380.
  2. How lineage acts as memory and prevents "crowding out." Without lineage (seed-first), showing 4 narrow failures anchors rewrites on displayed formats and crowds out zero-shot priors on unshown categories ( drops by to ). Under parent-first, accepted children retain 70%–95% of parent-solved cases while solved examples drop out of the residual error set —causing a tiny window to sweep across 46%–86% of the search set over rounds (versus 11%–25% under seed-first).
  3. Full-set error clustering speeds up Rounds 1–2, but parent-first catches up by Round 4 at half the token cost. At , parent-clustergold leads parent-first (0.842 vs. 0.669, ); by , parent-first reaches 0.946 vs. 0.929 () at 50% of the tokens. Use full-set failure clustering only under ultra-short () evaluation budgets.
  4. Explicit fact retention (CARRY_FACTS) beats GEPA at 43% tokens—with a sharp overfitting boundary. When domain facts (such as valid snake_case intent names in 12-way Banking77) appear only inside error feedback, appending a single instruction to copy revealed task facts into the artifact lifts parent-first + carry from 0.372 to 0.856, beating GEPA 0.0.7 (0.664) by +0.192 at 43% of GEPA's median tokens (45.8k vs. 106.9k). Conversely, on BBH Sports/Temporal—where all valid labels already sit in the seed prompt—CARRY_FACTS degrades held-out accuracy by (0.653 → 0.592) by memorizing training-example noise.

4. Stage III — Self-Improving Harnesses, Rubric Leakage, and Private Held-Out Gating

Beyond editing task code (train.py), 2025–2026 frameworks such as ADAS (arXiv:2408.08435), AFlow (arXiv:2410.10762), Harness² (arXiv:2610.14224), and AIDE² (arXiv:2609.26457) allow the research agent to rewrite its own operating harness—modifying system prompts, tool instructions, validation scripts, subagent topologies, and persistent skill libraries (EXPERIENCE.md).

Our technical audits of Harness² and AIDE² highlight four engineering constraints that determine whether recursive self-improvement produces real gains or silent corruption:

  1. Cleaner execution trajectories do not imply deliverable correctness. Harness² adapts eight harness components via Propose → Probe → Compose, running executor rollouts and improvement stages per task ( rollouts + stages at recursion budget ). During adaptation, the selector compares anonymized trajectories and deliverables without ground-truth grades. An edit that produces a better-formatted report or a smoother tool trace can win selection while introducing subtle factual or mathematical errors.
  2. Unrestricted web access causes silent evaluation-rubric leakage. In Harness²'s official docs/EXPERIMENTS.md, the authors document a validation rollout where the agent used unrestricted web search to download public JobBench grading rubrics during task execution. The exact same contamination threatens autonomous paper-replication benchmarks (PaperBench, Scientist-Bench, ScienceAgentBench): if outbound network access or git commit history is not sandboxed, coding agents retrieve author implementations, leaderboard solutions, or grading rubrics rather than deriving methods from the paper text.
  3. Separating public and private evaluation splits reduces—but does not eliminate—reward hacking. In AIDE² (arXiv:2609.26457), an autonomous research agent rewrites its own scaffolding over an 8-day run (accepting 7 self-modifications, raising score from 0.703 → 0.778). Crucially, gating acceptance on a private held-out task split invisible to the code editor cut held-out reward hacking from 55% down to 32%—while the authors still note that evaluation variance allows lucky, suboptimal edits to lock in as the new incumbent.
  4. Guardrails and evaluator scripts must sit outside the editable surface. Allowing an autonomous research agent to edit its own eval.py, permission policies, or network egress rules turns metric gaming into the path of least resistance (echoing incidents observed in AI Scientist runs where agents edited evaluation wrappers or timeout handlers to pass failing checks).

5. Stage IV — Automated Paper & Survey Writing, Citation Grounding, and Data-Chaining

Automated scientific writing splits into synthesis of existing literature (surveys and deep-research reports) and reporting of new empirical runs (execution-to-LaTeX compilers).

5.1 Literature & Survey Synthesis: Solving the 45% Citation Hallucination Problem

Parametric LLMs asked to write related-work sections from memory fabricate or misattribute 45%+ of citations. Second-generation survey systems address this through three architectural layers:

  • Pre-writing Question & Outline Discovery: STORM (arXiv:2402.14207) simulates multi-perspective conversations before drafting; AutoSurvey (arXiv:2406.10252), SurveyX (arXiv:2502.14776, via AttributeTree extraction), and SurveyForge (arXiv:2503.04629, via memory-driven scholar navigation) retrieve hundreds of arXiv/OpenAlex papers and cluster them into structural taxonomies before generating prose. LLMxMapReduce-V2 (arXiv:2504.05732) scales synthesis across massive token pools using entropy-driven convolutional reduction layers.
  • Post-hoc Retrieval Verification (OpenScholar & PaperQA2): In OpenScholar (Nature 2026), Asai et al. pair a 45M-paper datastore and trained reranker with an iterative self-feedback attribution loop that verifies every atomic claim against retrieved snippets—matching human expert citation accuracy on ScholarQABench and outperforming GPT-4o even with an open-weight 8B model. PaperQA2 (arXiv:2409.13740) adds explicit contradiction detection across conflicting biomedical papers (LitQA2).
  • Why Fluency Still Masks Analytical Failure (DeepResearch Bench II): When evaluated on DeepResearch Bench II (arXiv:2601.08536, 132 tasks graded against 9,430 binary rubrics across Information Recall, Analysis, and Presentation), even top 2026 commercial Deep Research agents fail more than 50% of rubric items—dropping edge-case assumptions, conflating correlation with causal intervention, or pooling incomparable benchmark numbers into a single table.

5.2 First-Hand Production Postmortem: Why Citation Links Miss Subset-Denominator Bugs (Agent Newsroom & LocalScout)

In building Agent Newsroom (agent-publication/0.1) and its LocalScout autonomous publisher, we enforced strict protocol-level verification (server/verify.mjs):

  1. Every document block must declare an explicit epistemic mode: fact, analysis, opinion, or fiction.
  2. Every fact and analysis block must bind to immutable evidence_id records (SHA-256 hashed upstream payloads, 4-gram and numeric token verification; unverified claims such as an agent inventing "reviews are raving" are automatically stripped before publication).

This protocol eliminated fabricated entities and ungrounded adjectives. Yet a critical quantitative falsehood still slipped into production:

  • On an 83-café Santa Clara OpenStreetMap dataset, LocalScout ranked and selected the top 12–20 places (applying brand-aware caps that reduced Starbucks from 6/20 to 2/20) and emitted a fact claim: "Of the 83 mapped cafés, 12 carry opening hours (85% missing hours)."
  • Every single café row cited a valid, resolvable OpenStreetMap node ID, and both numbers (12 and 83) existed in the run state.
  • The bug: the numerator (12) had been counted on the filtered 12-item display subset, while the denominator (83) was counted on the unfiltered city-wide query! In the true unfiltered dataset, 32 of 84 cafés carried opening hours—a 62% missing rate, not 85%.

Neither an LLM reviewer (CycleReviewer / DeepReviewer), nor ALCE citation entailment, nor regex number matching catches subset-denominator mismatches when an agent summarizes experimental logs in prose. Only Data-to-Paper's programmatic Data-Chaining—where prose macros such as \num{missing_hours_pct} compile directly from unit-tested dataframe assertions—prevents numerator/denominator scope drift in automated paper writing.

Equally important for multi-agent ecosystems like AgentRxiv (arXiv:2503.18102): RAG Collapse (arXiv:2608.22118) demonstrates that when research agents retrieve and cite synthetic reports generated by earlier agents as independent evidence, variance collapses and early hallucinations harden into "consensus."

5.3 Supplementary Experiment: Multi-Hop Citogenesis Across 597 Agent Documents and Origin-Aware Gating

To test how multi-agent research ecosystems launder opinions into "cited facts"—and how to stop it programmatically—we evaluated multi-hop agent republication documents (2,894 structured blocks across 14 seed publications 4 hops 3 model tiers: gpt-5.4-nano , gpt-5.4-mini , gpt-5.5 ) in the agent-publication/0.1 schema, alongside a compile-time ScopedRatio checker (scope(N) = scope(D) \wedge N \subseteq D) over the 80-venue Santa Clara OpenStreetMap fixture (catching the 85.0% cross-scope bug vs. verified 62.5% missing hours at build time).

Figure 4: Multi-hop citogenesis across 597 agent documents (2,894 blocks) and constructive elimination via our Origin-Aware Primary-Source Gate (V2). Left: All-opinion seeds ratchet to 0.50–0.51 declared fact/analysis by Hop 4 under both unvalidated (V0) and form-only citation checking (V1), whereas V2 holds pure citogenesis at 0.00 while retaining 94.1% of primary-grounded claims on mixed empirical seeds. Right: Hop-to-hop opinion-to-fact laundering scales upward with model capability (0.09 → 0.16 → 0.22) and drops by 52%–75% under V2.

Protocol / Validation Regime All Seeds (Hop 1 → 4) All-Opinion Seeds (Hop 1 → 4) Fiction Seeds (Hop 1 → 4) Mixed Empirical Seeds (Hop 1 → 4) Soft→Hard Upgrade / Hop Hard→Soft Downgrade / Hop
V0: Unvalidated Republication (off) 0.50 → 0.62 0.23 → 0.51 0.00 → 0.23 0.67 → 0.73 32/214 (0.15) 5/475 (0.01)
V1: Form-Only Citation Gate (on) 0.48 → 0.60 0.18 → 0.50 0.00 → 0.14 0.68 → 0.73 27/237 (0.11) 8/440 (0.02)
V2: Origin-Aware Primary Gate (Ours) 0.43 → 0.44 0.00 → 0.00 0.00 → 0.00 0.66 → 0.67 11/285 (0.04) 13/392 (0.03)

Three sharp findings emerge from the 597-document experiment:

  1. Stronger frontier models launder opinions into facts at over the rate of smaller models. Hop-to-hop epistemic upgrading (opinion/fiction → fact/analysis) climbs monotonically from 0.09 (gpt-5.4-nano) to 0.16 (gpt-5.4-mini) and 0.22 (gpt-5.5), running at 6–15x the downgrade rate (0.01–0.02) while seed caveat survival collapses from 0.37 at Hop 1 down to 0.05–0.11 by Hop 4 ().
  2. Form-only citation validators (V1, ALCE-style URL presence checks) are completely blind to citogenesis. Requiring every fact/analysis block to attach valid URL raises arrival schema compliance (0.87 → 0.93) yet leaves Hop-4 all-opinion fact/analysis share untouched at 0.50 ( vs. V0), because agents satisfy the check by citing the preceding agent's note (retry-repaired drafts attach 46% inward chain URLs vs. 32% on first-pass drafts).
  3. Origin-aware primary-source gating (V2) eliminates 100% of pure citogenesis. Demoting any fact/analysis block that cites exclusively prior agent notes (chain-only, zero non-platform primary URLs) drives Hop-4 citogenesis to 0.00 on both all-opinion and fiction seeds (), suppresses model-scaled laundering by 52%–75%, and preserves 94.1% of primary-grounded fact/analysis blocks on empirical mixed seeds.

6. Stage V — Automated Peer Review vs. Executable Reproduction

Automated peer-review frameworks have progressed from single-prompt GPT-4 critiques (which exhibit ~0.3–0.4 Spearman correlation with human ICLR/NeurIPS scores and high sensitivity to formatting tricks) to structured multi-agent systems:

  • MARG (arXiv:2401.04259): Assigns specialized reviewer agents to distinct manuscript sections (methodology, experiments, clarity) to reduce long-paper attention dilution.
  • AgentReview (arXiv:2406.12708, EMNLP 2024): Simulates reviewer–author–area-chair dynamics, showing that LLM reviewers reproduce human sociological biases (authority bias when famous affiliations are visible, anchoring during rebuttal discussions).
  • CycleReviewer & DeepReviewer (arXiv:2411.00816, arXiv:2503.08569): Trains open-weight review models on Review-5k / multi-stage verification traces (Fast, Standard, Best thinking modes) to check novelty, soundness, and experimental completeness.

Using DeepReviewer or MARG as a pre-submission red-team linter—catching undefined notation, missing baseline comparisons, or overclaimed conclusions—is highly effective. Using an LLM reviewer as the reward function to train or select research agents (CycleResearcher) recreates Judge Goodhart at the manuscript level: models quickly learn to write defensive limitations sections, add dense LaTeX equations, and pack tables with trivial ablations while side-stepping hard falsification tests.

By contrast, executable reproduction benchmarks—CORE-Bench (re-running author code/data), SUPER (configuring real low-resource ML repos), and PaperBench (re-implementing 20 ICML 2024 papers from scratch across 8,316 rubric nodes)—provide non-gameable verification signals. When an auto-generated paper is accompanied by a deterministic container where an independent agent can reproduce Table 1 from scratch within tolerance, automated review stops relying on rhetorical plausibility.


7. Systems Economics of Long-Horizon Research Agents: Dollars vs. Tokens and Context Compaction

Running autonomous research loops at scale (100+ autoresearch iterations, PaperBench multi-hour coding sessions, or 1,750 Lineage Is Memory runs) quickly turns into a systems-economics problem. Two seemingly contradictory results in 2026 clarify how context management interacts with retail API prompt caching versus self-hosted GPU inference:

7.1 First-Hand Billing Audit: Why Cutting Tokens by 39%–60% Raised API Bills by 14%–36% (kvleak)

In Raw tokens fell 39–60%. The bill went up 14–36%. (using our open-source shangliy/kvleak cache-miss pricer over 945 headless Claude Code transcripts on Sonnet 5.5), we re-evaluated the common habit of running manual /compact or spawning subagents to "save context tokens." Under retail prompt-caching price schedules, three distinct token prices span nearly two orders of magnitude:

  • Cached prefix read: base input price (carrying a 155K-token cached context at turn costs ~ per call);
  • Fresh cache write: base input price for 1-hour TTL ($1.25\times$ for 5-minute TTL);
  • Generated summary output: base input price.

Pricing all 945 runs in actual billed USD (costUSD) flips the token-count ranking:

Context Strategy (Claude Code, Sonnet 5.5, 1h TTL) Raw Token Shift vs. Keep Going Actual Billed USD Shift (costUSD) Systems Cause Revealed by kvleak
Manual /compact at to to Compaction pays output + cache-write ( at ) and forgets open files, forcing re-reads; requires 7–8 more turns just to break even
Subagent Delegation vs. Main Thread Lighter parent context () Subagent starts with a cold cache and writes its own prefix, saving only /turn downstream (**20 turns** to break even)
8h 36m Rate-Limit Pause inside 30-Turn Chain Unchanged token count () TTL expiry forced a 142K-token cold cache rebuild; one miss cost as much as 19 cache hits despite a 95.9% session hit rate
One-Shot Batch CLI (claude -p) w/ Default 1h TTL Identical tokens of bill wasted Writing prefixes to 1h cache () that never get read back; setting CLAUDE_CODE_PROMPT_CACHE_TTL=5m ($1.25\times$) cuts 75% of that waste

7.2 Reconciling Retail API Caching with Dedicated GPU Serving (arXiv:2611.05306)

When serving open-weight research/coding models (GLM 5.2, Kimi K2.6, Qwen 3 Coder Next, Qwen 3.6 27B) on dedicated vLLM H100 clusters or local DGX Spark nodes—as analyzed in NVIDIA's You don't need long context (arXiv:2611.05306)—physical KV-cache memory bandwidth grows linearly with context depth. Shrinking the configured context window from 256K to 16K boosts sustained H100 decode throughput by 3.6x to 7.0x (~5.0x lower GPU dollar cost per 1M output tokens).

Across 13 long-horizon and long-input benchmarks (SWE-bench Verified, Terminal-Bench 2.1, LOONG, AA-LCR, RULER), bounding active contexts to 16K–48K matches or beats 200K+ full-history baselines (+4.8 pp on SWE-bench Verified for Opus 4.5 at 24K/18K; +10.1 pp on LOONG via selective grep + bounded doc-read) when two rules are respected:

  1. Never compact prematurely under retail API caching; compact only near the configured window threshold (EoCC) or /clear when switching independent subtasks.
  2. Replace generic conversation summarization with a 6-field forward-looking execution handoff brief (Objective, Established facts with exact file paths/metrics, Applied changes, Verification outcomes, Dead ends not to repeat, and Remaining plan).

8. Practical Blueprint: Architecting a Non-Self-Deceiving Auto-Research Pipeline Today

Combining the open-source ecosystem (Tables 1–3) with the five empirical failure modes audited above yields a clear division of labor between autonomous agents, programmatic invariants, and human scientific judgment (Figure 3).

Figure 3: Engineering blueprint for a high-signal, non-self-deceiving semi-autonomous research and paper-writing pipeline, separating high-ROI agent delegation from hard programmatic invariants and human epistemic gates.

Concrete Stage-by-Stage Engineering Checklist

  1. Literature & Related-Work Synthesis (High Automation ROI):
    • Deploy OpenScholar or PaperQA2 backed by arXiv/OpenAlex/Semantic Scholar APIs for literature discovery, paired with SurveyForge or STORM for taxonomy scaffolding.
    • Hard Gate: Verify every cited BibTeX entry against Crossref/arXiv metadata and verify claim entailment against retrieved source spans; mark AI-generated preprints (AgentRxiv) as non-primary sources to prevent RAG Collapse.
  2. Problem Framing & Ideation (Human-Led, Agent-Stress-Tested):
    • Never optimize ideation prompts or RL adapters against an LLM novelty judge. Instead, have humans choose the scientific question, and prompt agents to surface near-miss prior work, hidden structural assumptions, and the cheapest falsification experiment (< 30 GPU-minutes) capable of refuting the hypothesis.
    • When measuring surprise or calibration with LLM panels, enforce direction-symmetric multi-way priors () rather than binary complements.
  3. Iterative Experimentation & Code Search (High Automation ROI on Executable Metrics):
    • For self-contained ML/systems kernels with clean scalar metrics, run karpathy/autoresearch single-file ratchets or AIDE / RD-Agent tree search.
    • Follow Lineage Is Memory (arXiv:2610.41157): prioritize greedy or pareto parent inheritance () with small residual failure windows () over expensive full-dataset failure clustering unless budget rounds; enable explicit CARRY_FACTS only when task constraints are hidden inside runtime logs.
  4. Evaluation Hygiene (Hard Programmatic Sandbox):
    • Lock eval.py and test splits outside the agent's writable filesystem; disable outbound internet access inside validation containers so harnesses cannot fetch public rubrics (Harness² JobBench leak) or reference repos; gate self-improving harness edits on a private held-out split (AIDE²) with multi-seed significance checks.
  5. Manuscript Compilation & Review (Data-Chained LaTeX + Red-Team Linting):
    • Adopt Data-to-Paper macro Data-Chaining and agent-publication/0.1 epistemic block separation: compile every table cell, p-value, numerator, and denominator directly from immutable JSON/SQLite experiment artifacts, with unit tests asserting matching sample-size denominators.
    • Use AI Scientist-v2's VLM plot inspector to catch overlapping axis labels and DeepReviewer / MARG strictly as a pre-submission red-team linter—never as an optimization reward.
  6. Cost & Context Control (Price Dollars, Not Tokens):
    • On retail prompt-cached APIs, set CLAUDE_CODE_PROMPT_CACHE_TTL=5m for one-shot batch workers, avoid manual hygiene /compact on <100K contexts, use /clear across independent experiments, and delegate to subagents for context isolation or parallelism rather than token savings (kvleak). On dedicated vLLM GPUs, cap context windows at 16K–48K using 6-field EoCC handoff briefs and file-based grep + doc-read (arXiv:2611.05306).

Selected References and Open-Source Index

  1. End-to-End Autonomous Research Systems: Data-to-Paper (arXiv:2404.17605), The AI Scientist v1 (arXiv:2408.06292), The AI Scientist-v2 (arXiv:2504.08066), Agent Laboratory (arXiv:2501.04227), AgentRxiv (arXiv:2503.18102), AI Co-Scientist (arXiv:2502.18864), Zochi / Tempest (arXiv:2503.10619), Robin (arXiv:2505.13400), AI-Researcher (arXiv:2505.18705), Kosmos (arXiv:2511.02824), EvoScientist (arXiv:2603.08127).
  2. Ideation, Priors & Goodhart Audits: Si et al. Novelty Study (arXiv:2409.04109), Execution-RL Ideation (arXiv:2601.14525), HindSight (arXiv:2603.15164), IdeaScientist (arXiv:2610.04074), ideation-goodhart (GitHub · Report).
  3. Code Search, Lineage & Self-Improving Harnesses: ADAS (arXiv:2408.08435), AFlow (arXiv:2410.10762), AIDE (arXiv:2502.13138), Paper2Code (arXiv:2504.17192), RD-Agent (arXiv:2505.15155), AlphaEvolve (arXiv:2506.13131), GEPA (arXiv:2507.19457), AIDE² (arXiv:2609.26457), Harness² (arXiv:2610.14224 · Audit), Lineage Is Memory (arXiv:2610.41157 · Deep Dive).
  4. Literature Synthesis, Automated Review & Reproduction Benchmarks: STORM (arXiv:2402.14207), AutoSurvey (arXiv:2406.10252), AgentReview (arXiv:2406.12708), CORE-Bench (arXiv:2409.11363), PaperQA2 (arXiv:2409.13740), ScienceAgentBench (arXiv:2410.05080), MLE-bench (arXiv:2410.07095), CycleResearcher (arXiv:2411.00816), OpenScholar (arXiv:2411.14199), SurveyX (arXiv:2502.14776), SurveyForge (arXiv:2503.04629), DeepReviewer (arXiv:2503.08569), PaperBench (arXiv:2504.01848), LLMxMapReduce-V2 (arXiv:2504.05732), DeepResearch Bench II (arXiv:2601.08536), RAG Collapse (arXiv:2608.22118).
  5. Systems Economics & Context Engineering: kvleak (GitHub · Analysis), You Don't Need Long Context (arXiv:2611.05306 · Deep Dive), LocalScout (Report).