Lineage Is Memory: Why Parent Inheritance Drives LLM-Guided Prompt and Code Optimization
Across 1,750 pre-registered runs on nine prompt and code optimization benchmarks (arXiv:2610.41157), greedy parent inheritance adds +0.238 held-out accuracy, closes the gap with full-set error clustering by round 4, and reaches 0.856 vs. GEPA's 0.664 on Banking77 at 43% of the token cost when instructed to copy revealed facts into the prompt.
Abstract and Core Thesis
Iterative large language model (LLM) optimizers refine prompts and programs by repeating a three-step loop: evaluate a candidate on a search set, show diagnostic failures to a proposer model, and retain candidates that improve validation performance. Recent work on reflective prompt optimization has focused almost entirely on feedback disclosure — arguing that small random minibatches provide too narrow a view of the error distribution, and proposing full-set error clustering, contrastive exemplars, or adaptive sample allocation to widen what the proposer sees in a single round.
In Lineage Is Memory: Parent Inheritance and Feedback Disclosure in Short-Budget LLM-Guided Optimization (arXiv:2610.41157), Shanglin Yang and Zhan Shi show that this diagnosis confounds single-round feedback disclosure () with multi-round parent inheritance (). Every recent reflection-scope method evaluates its wider feedback window exclusively inside a lineage-preserving search loop (), never testing whether wide disclosure can substitute for building on the best prior candidate, or whether parent inheritance renders single-round clustering redundant once search runs for a few rounds.
To separate these mechanisms, the study pre-registers and executes 1,750 optimization runs across nine prompt and Python code synthesis benchmarks, two model families (gpt-4.1-mini and claude-haiku-5-5), a larger proposer ablation (gpt-4.1), and five sequential pre-registered batches (A.1–A.5). Five findings stand out:
- Parent inheritance is the primary driver of short-budget optimization. Switching from memoryless seed resampling () to greedy parent inheritance () raises held-out accuracy by ( bootstrap CI ) across all seven primary tasks on
gpt-4.1-mini( on the five non-ceiling tasks) and () onclaude-haiku-5-5. - Lineage outweighs both full-set error clustering and a larger proposer. Showing just raw failures to a lineage-preserving optimizer (
parent-first, ) beats giving a memoryless optimizer a target-preserving cluster summary of all search-set failures (seed-clustergold, ) by (). Upgrading the proposer and clusterer fromgpt-4.1-minitogpt-4.1yields onparent-first, whereas the lineage gap undergpt-4.1reaches . - Lineage works through three coupled mechanisms that prevent "crowding out." Under
seed-first, showing four narrow failures anchors rewrites on displayed formats and crowds out zero-feedback priors on unshown categories ( drops by ondurationand onsize). Underparent-first, child candidates retain 70%–95% of parent-solved cases while solved errors drop out of the failure set, shifting the feedback window across 46%–86% of the search set over rounds (versus 11%–25% underseed-first). - Full-set clustering accelerates rounds 1–2, but
parent-firstcatches up by round 4. When target rubrics are preserved (clustergold), clustering all failures helpsseed() and givesparent-clustergoldan early lead at round ( vs. , ). By round ,parent-firstreaches vs. and finishes tied at half the token cost, yielding an attenuation interaction of . - Lineage acts as externalized memory when paired with explicit fact retention (
CARRY_FACTS) — with a sharp boundary condition. On 12-intentbanking77, where valid snake_case intent labels appear only inside error feedback, baselineparent-firststalls at (retaining intents) behindGEPA 0.0.7(, intents). Appending one sentence instructing the proposer to copy revealed facts into the prompt (CARRY_FACTS) liftsparent-first + carryto ( intents), beating GEPA by at 43% of GEPA's median tokens ( vs. ). Meanwhile,seed-first + carrysees the same four initial errors every round, retains intents, and stalls at . Conversely, onbbh_ste— where all six option labels are already present in the seed prompt —CARRY_FACTSreduces accuracy by () by overfitting to training-example details.
1. Background and the Confound in Reflective Optimization
How Iterative LLM Optimizers Work
Discrete prompt and program optimization treats an instruction or source file as a mutable artifact improved through black-box evaluations on a search split . Across the literature, four families of algorithms have emerged:
| Optimization Family | Representative Methods | How Candidates Are Updated |
|---|---|---|
| Trajectory / Meta-Prompt Resampling | OPRO (Yang et al., 2023), Promptbreeder (Fernando et al., 2023) | Prompt an LLM with a scored history of prior candidates to sample higher-scoring instructions. |
| Evolutionary Operators | EvoPrompt (Guo et al., 2023), FunSearch (Romera-Paredes et al., 2023), AlphaEvolve (Novikov et al., 2025), ImprovEvolve (Kravatskiy et al., 2026) | Maintain an island or population of candidates; apply LLM-driven mutation, crossover, and basin-hopping. |
| Textual Gradients & Auto-Differentiation | ProTeGi (Pryzant et al., 2023), TextGrad (Yuksekgonul et al., 2024), LLM-AutoDiff (Yin & Wang, 2025), Momentum TextGrad (Ding et al., 2026) | Compute natural-language critiques on failing examples ("textual gradients") and apply an editing step to the current prompt. |
| Reflective Evolution & Agent Harnesses | GEPA (Agrawal et al., 2025), ESPO (Liu et al., 2026), RPT (Bayat et al., 2026), AdaGEPA (Chen et al., 2026), CASD (Singh et al., 2026) | Reflect on minibatch or clustered execution traces, propose targeted edits, and gate acceptance via validation checks. |
Why Recent Critique Focused on Single-Round Disclosure ()
Standard reflective optimizers such as GEPA (Agrawal et al., 2025) sample a tiny random minibatch of examples per round, reflect on failures, and accept a child candidate only if it improves on that minibatch (followed by periodic full-set pareto tracking). A wave of 2026 papers argued that narrow minibatches starve the proposer of global error structure:
- ESPO (Liu et al., 2026) invoked a coupon-collector argument: if a prompt fails on distinct error categories, random minibatches of size require draws just to observe every category once, and local fixes for one minibatch can regress previously solved categories. ESPO therefore evaluates the full training split and uses an LLM to cluster all errors into 3–7 diagnostic failure patterns before proposing an edit.
- Reflective Prompt Tuning (RPT) (Bayat et al., 2026) exposes the training error table to the proposer via tool calls (
inspect_errors,cluster_failures). - AdaGEPA (Chen et al., 2026), Contrastive Reflection (Koh et al., 2026), and Coding Agents as Prompt Optimizers (CASD) (Singh et al., 2026) similarly widen or structure the diagnostic context fed to the proposer.
The Hidden Confound: Lineage () vs. Disclosure ()
Notice what all of these studies hold fixed: every single one runs inside a lineage-preserving loop (). When an optimizer edits the best candidate found so far () rather than re-editing the initial seed (), the candidate text itself carries forward rules accumulated in earlier rounds. Furthermore, when evaluation filters for residual failures , every example solved in round disappears from the failure list in round .
Consequently, a small fixed window of failures under is not static across rounds — it automatically sweeps across the residual error distribution as earlier failure modes are eliminated. Testing wide disclosure () only under makes it impossible to tell:
- How much of the optimization gain comes from parent inheritance () versus feedback disclosure ()?
- Does wide single-round disclosure () allow memoryless resampling () to match lineage-preserving search?
- Does multi-round lineage () attenuate the advantage of full-set error clustering once search runs beyond the first two rounds?
2. The Factorial Framework and Estimands
To disentangle these factors, Yang and Shi factorize an iterative LLM optimizer into three orthogonal design choices at each round :
1. Lineage Policy ()
Given a seed candidate and a search set with mean per-example score , the best-so-far candidate updates via strict inequality:
Strict inequality () is essential: if ties () were accepted, neutral edits could drift away from verified instructions. The parent candidate handed to the proposer at round is then determined by :
2. Feedback Disclosure Policy ()
Let denote the ordered list of failing search examples on parent . The factorial crosses with six disclosure rules:
| Disclosure Policy | What the Proposer Sees at Round | Extra LLM Calls |
|---|---|---|
none |
Zero-feedback baseline: only the task objective and parent candidate (score=None, feedback="(no feedback)"). |
0 |
first |
The first failing examples from in fixed dataset index order: input -> got, expected. |
0 |
diverse |
failing examples selected via greedy farthest-point sampling under normalized character Levenshtein distance on input strings (Pu et al., 2017). | 0 |
minibatch |
examples drawn uniformly at random from with round seed , showing both PASS and FAIL items (matching GEPA's unfiltered minibatch view). |
0 |
cluster |
All failing examples grouped by one clusterer LLM call into 3–7 failure patterns with count and one representative wrong answer (Batch A.1). | 1 clusterer call / round |
clustergold |
Target-preserving clustering: all failures grouped into 3–7 failure patterns, explicitly preserving the input, wrong output, and full expected answer/rubric (Batches A.2–A.4). | 1 clusterer call / round |
3. Evaluation Scheduling () and External Baseline (GEPA 0.0.7)
All factorial cells evaluate once on and run up to proposal rounds on prompt tasks (, maximum search evaluations) and rounds on Python code synthesis tasks (, maximum evaluations), stopping early as soon as . On the seven prompt tasks, the factorial is compared at matched max_metric_calls = 180 against external GEPA 0.0.7 (reflection_minibatch_size = 3) both without early stopping (GEPA (mb3)) and with an early-stop wrapper (GEPA (mb3 + stop)).
Pre-Registered Estimands
Every cell is run for independent seeds (), and all confidence intervals use task-stratified bootstrap resamples. Three formal estimands govern the analysis:
- Lineage Main Effect (): The average gain of parent inheritance over seed resampling across the four non-
nonedisclosure rules : - Lineage Attenuation of Disclosure Breadth (): For a wider disclosure policy , how much larger its benefit over
firstis under memorylessseedthan underparent:A positive proves that multi-round lineage substitutes for single-round feedback breadth. - Unshown-Category Coverage (): For tasks with partitioned format categories , let be the set of categories that ever appeared inside the feedback window during run . Coverage on held-out examples belonging to unshown categories tests whether narrow feedback crowds out zero-feedback model priors:
3. Finding 1: Parent Inheritance Outweighs Disclosure Breadth and Proposer Size
Across all five pre-registered batches (1,750 runs total), parent inheritance () is the dominant factor, dwarfing both feedback disclosure strategy () and proposer model scale.

Cross-Task Factorial Results (gpt-4.1-mini and claude-haiku-5-5)
Of the seven primary tasks, two (dates and scone) are zero-feedback ceiling tasks: even with F = none (zero examples shown), rewriting the seed prompt raises held-out accuracy to on dates and on scone because the task model already knows ISO-8601 date rules and negation entailment once prompted clearly. On the remaining five feedback-dependent tasks (labels, routing, replies, duration, size), feedback and lineage are essential:
| Cell ( – ) | labels |
dates |
routing |
replies |
scone |
duration |
size |
7-Task Mean (4.1-mini) |
5-Task Non-Ceiling (4.1-mini) |
5-Task Non-Ceiling (Haiku 5.5) |
|---|---|---|---|---|---|---|---|---|---|---|
seed – none |
0.277 | 1.000 | 0.000 | 0.057 | 0.900 | 0.633 | 0.738 | 0.515 | 0.341 | 0.335 |
seed – first |
0.410 | 1.000 | 0.500 | 0.290 | 0.900 | 0.562 | 0.767 | 0.633 | 0.506 | 0.504 |
seed – diverse |
0.633 | 0.993 | 0.440 | 0.420 | 0.903 | 0.729 | 0.738 | 0.694 | 0.592 | 0.626 |
seed – minibatch |
0.553 | 0.990 | 0.500 | 0.433 | 0.890 | 0.625 | 0.804 | 0.685 | 0.583 | 0.559 |
seed – clustergold |
0.637 | 0.990 | 0.520 | 0.867 | 0.900 | 0.675 | 0.838 | 0.775 | 0.707 | 0.653 |
parent – none |
0.280 | 1.000 | 0.000 | 0.053 | 0.897 | 0.633 | 0.738 | 0.514 | 0.341 | 0.332 |
parent – first |
0.923 | 1.000 | 0.933 | 0.980 | 0.910 | 0.962 | 0.925 | 0.948 | 0.945 | 0.906 |
parent – diverse |
0.947 | 1.000 | 0.857 | 0.907 | 0.913 | 0.971 | 0.988 | 0.940 | 0.934 | 0.907 |
parent – minibatch |
0.883 | 0.993 | 0.750 | 0.917 | 0.893 | 0.862 | 0.900 | 0.886 | 0.862 | 0.839 |
parent – clustergold |
0.893 | 0.990 | 0.793 | 0.893 | 0.913 | 0.979 | 0.938 | 0.914 | 0.899 | 0.888 |
Look closely at the contrast between parent-first and seed-clustergold:
- Even when
seed-clustergoldinspects every failure in and summarizes them with full expected answers via an extra clusterer call per round, it reaches on the 5 non-ceiling tasks ( in the dedicated Batch A.2 replication). - Meanwhile,
parent-first— which shows only the first failing examples with zero clustering calls — achieves ( in Batch A.2), outperformingseed-clustergoldby () ongpt-4.1-miniand () onclaude-haiku-5-5.
Why a Bigger Proposer (gpt-4.1) Does Not Replace Lineage
A natural objection is that gpt-4.1-mini and claude-haiku-5-5 might be too weak as proposers to synthesize a global prompt from clustered failures in a single shot. Batch A.4 tests this directly by upgrading both the proposer and clusterer to gpt-4.1 while holding the task model fixed at gpt-4.1-mini:
| Policy ( – ) | gpt-4.1-mini Proposer |
gpt-4.1 Proposer |
Proposer Size Effect |
|---|---|---|---|
seed – first |
0.506 | 0.525 | |
seed – clustergold |
0.789 | 0.720 | |
parent – first |
0.946 | 0.927 | |
parent – clustergold |
0.928 | 0.922 |
Upgrading the proposer to gpt-4.1 yields a slightly negative shift on both parent-first () and seed-clustergold (), while the lineage advantage under gpt-4.1 widens to (). Why? Without lineage, a stronger proposer still only sees a compressed summary or 4 failures per round and over-tailors its single-step rewrite to those examples. Bottlenecks in iterative prompt optimization are structural (lack of state accumulation across steps), not proposer reasoning capacity.
4. Finding 2: How Lineage Works as Memory (and Prevents Crowding Out)
Why does editing with failures work so well, while editing with failures stalls — or even degrades below the zero-feedback baseline? The paper traces this to three coupled mechanisms.
Mechanism A: Narrow Feedback Without Lineage Causes "Crowding Out" ()
Look at the Python code synthesis task duration in Figure 1:
- Under
seed-none(zero feedback shown), askinggpt-4.1-minito improveparse_duration(text)from its task description alone raises held-out accuracy from (seed ) to , because the LLM draws on its pretrained knowledge of common duration formats (HH:MM:SS,1h30m, ISO-8601PT..., word units). - Under
seed-first(showing the first failing examples), held-out accuracy drops to ( belowseed-none!).
Measuring unshown-category coverage across the 8 format categories of duration and size explains why:
| Code Synthesis Task | seed-none |
seed-first |
Crowding-Out Drop () | parent-first |
|---|---|---|---|---|
duration (8 format categories) |
0.483 | 0.429 | 0.875 | |
size (8 format categories) |
0.676 | 0.563 | 0.750 |
When seed-first displays four specific failures (e.g., whitespace-padded " 45 s " and "2.5h"), the proposer writes a narrow patch tailored to those four strings and omits general branches it would have written under seed-none. And because seed-first resets to every round, never changes — so the exact same 4 failures are shown in all 12 rounds, locking the optimizer into a narrow local patch.
Mechanism B: High Parent-to-Child Retention + Residual-Error Exposure Shift
Under , two dynamics break this lock:
- Text Inheritance Preserves Solved Cases: On every accepted step (), the child candidate inherits the parent's prompt/code text and retains 70% to 95% of examples already solved by the parent ( on
duration, onsize, onlabels, onrouting). - Residual-Error Exposure Shift: Because solved examples disappear from , the deterministic
firstwindow () automatically advances to the next unsolved failure modes in . Over a run,parent-firstexposes 46% to 86% of the entire search set (86.5%onrouting,77.5%onlabels,67.1%onsize), compared to just 11% to 25% underseed-first. - Multi-Step Edit Decomposition: Instead of forcing the proposer to synthesize a 12-rule system prompt or an 8-branch parser in one shot,
parent-firstlets the proposer add 1–2 rules per round while strict selection () rejects any edit that breaks previously solved categories.
5. Finding 3: Error Clustering Speeds Up Early Rounds, But Lineage Catches Up by Round 4
The Batch A.1 A.2 Lesson: Why Naive Error Clustering Fails (cluster vs. clustergold)
In Batch A.1, the authors noticed an unexpected failure in parent-cluster on the customer-support policy task replies: while parent-first scored , parent-cluster collapsed to (seed-cluster scored ). Auditing the 156 clusterer outputs revealed a critical failure mode of naive LLM error summarization:
- In
replies, each failing example has a multi-clause policy rubric as its expected targetwant(e.g., "Must apologize for damaged item, offer replacement or refund without requiring return shipping on orders under $50, and not promise a specific delivery date"). - The Batch A.1
CLUSTER_PROMPTasked the clusterer for "what goes wrong, how many of the wrong answers it covers, and one representative wrong answer copied verbatim." - In 125 of 156 calls (80.1%), the clusterer summarized the candidate's wrong reply but dropped the expected policy rubric! Without the exact target rubric, the proposer guessed at Acme's support policies and regressed.
In Batch A.2, clustergold modified seven words of the clusterer prompt to require "one representative failure copied verbatim with its input, the wrong answer and the expected answer in full." That single change raised rubric retention on replies from 19.9% to 100.0%, lifting seed-clustergold from and parent-clustergold from .
Anytime Trajectory () and the Attenuation Interaction
With target-preserving clustergold in place, how do the six policies compare across rounds? Tracking held-out accuracy of the search-best candidate after proposal rounds across the five feedback-dependent tasks reveals a clean crossover:
| Policy ( – ) | Round | Round | Round | Final () | Median Total Tokens / Run |
|---|---|---|---|---|---|
seed – first |
0.471 | 0.491 | 0.500 | 0.506 | 35.1k |
seed – diverse |
0.531 | 0.562 | 0.580 | 0.592 | 35.4k |
seed – clustergold |
0.674 | 0.728 | 0.764 | 0.789 | 61.8k |
parent – minibatch |
0.543 | 0.662 | 0.785 | 0.862 | 38.9k |
parent – clustergold |
0.712 | 0.842 | 0.929 | 0.928 | 49.4k |
parent – first |
0.547 | 0.669 | 0.946 | 0.946 | 26.7k |
Three takeaways emerge from this trajectory:
- At ultra-tiny budgets ( or rounds),
parent-clustergoldleadsparent-firstby to ( vs. at ) because clustering exposes all failure families immediately in round 1. - By round ,
parent-firstcompletely closes the gap and overtakesparent-clustergold( vs. ), finishing at vs. while using roughly half the total tokens ( vs. ) because it pays zero clusterer LLM calls and triggers early stopping more often. - Lineage strongly attenuates the value of wide single-round disclosure: The pre-registered interaction is () on
gpt-4.1-miniand () onclaude-haiku-5-5( and ). Furthermore,parent-minibatchtrailsparent-firstby () because random minibatches waste feedback slots on passing examples (PASS), starving the proposer of failure signal as accuracy climbs above 70%.
6. Finding 4: Fact Retention in the Prompt (CARRY_FACTS) and Its Boundary Condition
Why Baseline parent-first Trailed GEPA on banking77
To test out-of-distribution generalization on standard public benchmarks, Batch A.5 evaluated 12-intent banking77 (Casanueva et al., 2020) and BIG-Bench Hard Salient Translation Error Detection (bbh_ste) (Suzgun et al., 2022). On banking77, the seed prompt is simply "Classify the banking customer query." with output format "Reply with the intent label only." Crucially, none of the 12 exact snake_case target labels (such as beneficiary_not_allowed, declined_cash_withdrawal, pending_top_up, or visa_or_mastercard) appear in the seed prompt — they are only revealed one by one when an example fails (got 'wire_transfer_error', expected 'beneficiary_not_allowed').
On this task, baseline parent-first scored only , whereas external GEPA 0.0.7 (mb3) scored . Why did GEPA beat baseline parent-first here despite losing or tying on the primary tasks?
Inspecting the prompts evolved by both methods revealed the exact mechanism:
- GEPA's built-in reflection prompt explicitly instructs the proposer: "Include all task-specific factual information revealed in the execution trace (such as domain labels, formatting rules, or edge cases) directly in the new instruction." Across 10 runs, GEPA's final prompts explicitly listed 9.0 of the 12 target intent names.
- Baseline
parent-firstused a minimal 4-line meta-prompt ("Improve the prompt so it fixes the failures without breaking what already works."). Without an instruction to copy target label names into the prompt, the proposer often wrote generic semantic advice ("Use concise snake_case banking terms") and listed only 5.6 of the 12 intents on average.
The One-Sentence CARRY_FACTS Intervention ()
To test whether explicit instruction-level fact retention accounts for the entire gap, Batch A.5 appended a single pre-registered sentence (CARRY_FACTS) to the proposer objective:
"The task model sees only this prompt, never the feedback: copy every task-specific fact the feedback reveals (for example the exact expected labels or formats) into the prompt."

The results on banking77 and bbh_ste (Table 5 of the paper) demonstrate both the power of pairing lineage with fact retention and its exact boundary condition:
| Optimizer Configuration | banking77 Held-Out Acc |
banking77 Intents Retained (of 12) |
banking77 Median Tokens |
bbh_ste Held-Out Acc |
bbh_ste Median Tokens |
|---|---|---|---|---|---|
seed-first (baseline) |
0.225 | 3.2 / 12 | 25.0k | 0.620 | 64.9k |
parent-first (baseline) |
0.372 | 5.6 / 12 | 31.4k | 0.653 | 71.1k |
GEPA 0.0.7 (mb3) |
0.664 | 9.0 / 12 | 106.9k | 0.633 | 127.8k |
seed-first + carry |
0.317 | 4.0 / 12 | 30.8k | 0.587 | 72.4k |
parent-first + carry |
0.856 | 11.7 / 12 | 45.8k | 0.592 | 82.5k |
Look at why seed-first + carry fails while parent-first + carry dominates on banking77:
- In round 1, the seed prompt fails on all 20 search examples (
0.000). The first failing examples happen to belong to 4 distinct intents. - Under
seed-first + carry, the proposer dutifully copies those 4 intent names into — scoring intents (). Next round, because resets the parent to , fails on the exact same examples,firstdisplays the exact same 4 failures, and the optimizer is trapped at 4.0/12 intents in every single run ( CI )! - Under
parent-first + carry, round 1's candidate solves the first 4 intents and becomes . In round 2, examples from those 4 intents pass and drop out of , sofirstreveals 3–4 new intents. By round 4–5, the prompt has accumulated 11.7 of the 12 intents and reaches held-out accuracy — beatingparent-firstby (), beatingseed-first + carryby (), and beatingGEPA 0.0.7by () at 43% of GEPA's token budget ( vs. ).
This is the literal meaning of "Lineage Is Memory": the evolving prompt text serves as an externalized, append-and-refine memory buffer across rounds, while residual-error filtering acts as a read-pointer that advances to unmemorized facts each step.
The Boundary Condition (): When CARRY_FACTS Hurts (bbh_ste)
Why did CARRY_FACTS lower parent-first accuracy on bbh_ste from to (, CI )?
- In
bbh_ste, the seed prompt already lists all six multiple-choice options(A)–(F)and their definitions. There are no hidden label names to discover from feedback. - When instructed by
CARRY_FACTSto "copy every task-specific fact the feedback reveals into the prompt," the proposer copies instance-specific German-to-English phrases and entities from the 20 search examples into the system instructions, which overfits the search split and hurts generalization on the 30 held-out translations. - Design rule: Enable
CARRY_FACTSwhen target labels, schemas, or policy rules are absent or underspecified in and must be discovered from feedback; disable it when already enumerates the complete label space and task rules.
7. Finding 5: Where GEPA's Tokens Actually Go ()
Across the five primary prompt tasks, parent-first matches or beats GEPA 0.0.7 (mb3) on held-out accuracy ( vs. mean across the 5 prompt tasks) while consuming to fewer tokens. Where do GEPA's tokens go? Decomposing every GEPA API call across the 7 prompt tasks into three buckets — (1) Proposer Reflection, (2) Full Search-Set Validation, and (3) Minibatch Gating Evaluations — reveals a striking imbalance:
| Prompt Task | parent-first Median Tokens |
GEPA (mb3) Median Tokens |
GEPA Reflection Share | GEPA Full-Set Validation Share | GEPA Minibatch Gating Share |
|---|---|---|---|---|---|
labels |
24.7k | 54.4k () | 9.8% | 60.5% | 29.7% |
dates |
5.2k | 17.8k () | 2.1% | 21.3% | 76.6% |
routing |
29.4k | 68.2k () | 11.4% | 56.8% | 31.8% |
replies |
64.1k | 189.3k () | 4.6% | 73.9% | 21.5% |
scone |
14.8k | 41.2k () | 10.9% | 51.2% | 37.9% |
banking77 |
45.8k (+carry) |
106.9k () | 12.6% | 58.1% | 29.3% |
bbh_ste |
71.1k | 127.8k () | 14.8% | 66.8% | 18.4% |
Proposer reflection accounts for only 2% to 15% of GEPA's total token spend. The remaining 85% to 98% is spent on task-model evaluations (minibatch gating checks that reject proposals, plus full-set validations whenever a noisy 3-example minibatch passes). Because a 3-example random minibatch has high variance, many proposals that tie or pass on 3 examples fail to improve the full 20-example search set, triggering expensive full-set validations without advancing .
In contrast, parent-first evaluates the 20-example search set directly, conditions only on confirmed failures so every proposal targets real errors, and stops immediately once . This aligns closely with concurrent findings by Zhou & Cheng (2026) (Explore More, Validate Less) and Oved et al. (2026) (Evolution or Illusion?).
8. Related Literature and How the Pieces Fit Together
Understanding Lineage Is Memory also clarifies how recent threads in prompt optimization, program synthesis, and agent harness engineering connect:
- Coupon-Collector Bounds vs. State-Conditional Exposure Shift:
ESPO (Liu et al., 2026) modeled error discovery as sampling with replacement from a static failure distribution, where seeing all error types in one round requires large or clustered feedback. Lineage Is Memory shows that under and strict improvement , failure sampling is state-conditional and without replacement on solved categories: once category is fixed in , its examples drop out of , shifting the deterministic
firstwindow to unsolved categories in round . - Representative Example Selection in Program Synthesis:
In classic programming-by-example (Pu et al., 2017), selecting diverse input shapes helps synthesize a program in a single call. The factorial confirms that input-diverse selection (
diverse) helps one-step/memorylessseed( ongpt-4.1-mini, on Haiku), just asclustergoldhelpsseed( to ). Yet once multi-step lineage () is enabled, residual-error filtering provides diversity across time rather than within a single prompt. - Prompt Overfitting and Goodhart's Law on Small Search Splits:
Lumer et al. (2026) (Reflective Prompt Optimization of a Deep-Research Agent) and Yang (2026) (Ideation Goodhart) document how aggressive iterative optimization can overfit small search sets when the optimizer memorizes training examples rather than transferable rules. Batch A.5's
bbh_steresult ( underCARRY_FACTS) provides a controlled demonstration of this boundary: copying revealed facts is beneficial when the task specification is missing label definitions (banking77, ), and harmful when the specification is already complete (bbh_ste, ). - Connection to Agent Harness Engineering (
Harness²): Just asHarness²(arXiv:2610.14224) improves an agent by proposing, probing, and composing verified changes to its surrounding system harness,parent-first + carrytreats the system prompt as a verified, monotonically improving state artifact () that accumulates domain rules across rounds.
9. Practical Engineering Guidelines and AdalFlow Implementation
The parent-first + carry algorithm from this paper ships as the default prompt optimizer in AdalFlow (adalflow.optim.evolution.optimize_prompt). Based on the 1,750-run empirical audit, four engineering rules apply when building or tuning an LLM-guided optimizer:
Rule 1: Default to Greedy Lineage + Deterministic First- Failures (parent-first)
Do not spend extra LLM calls clustering errors or sampling random pass/fail minibatches on budgets of rounds. Evaluate on a fixed 20–25 example search split, keep only strictly improving candidates (), and show the first failing examples (input -> got, expected).
Rule 2: Enable carry_facts=True When Labels or Formats Live in the Data
Whenever your seed prompt is a brief objective and the exact label vocabulary, output schema, or policy edge cases are only visible inside ground-truth answers or evaluator rubrics, enable carry_facts=True (the default in AdalFlow). Turn carry_facts=False if your seed prompt already contains the full closed-book specification and option definitions.
Rule 3: Use parent-clustergold Only for Ultra-Short Budgets () — and Never Drop Target Rubrics
If latency or evaluation cost restricts you to or proposal rounds, clustering all failures speeds up early convergence ( vs. at ). If you use an LLM clusterer, always copy the full expected answer or grading rubric verbatim (CLUSTER_GOLD_PROMPT); omitting expected rubrics caused an 80% rubric loss and a accuracy collapse on replies.
Rule 4: Minimal Usage in Python (adalflow & lineage-is-memory)
from adalflow.optim.evolution import optimize_prompt
# Default configuration runs parent-first with carry_facts=True (k=4 failures, strict > acceptance)
result = optimize_prompt(
objective=(
"A system prompt that makes the model classify an online-banking "
"customer query into its intent. Intent labels are lower-case snake_case names."
),
seed="Classify the banking customer query.",
output_format="Reply with the intent label only.",
search_data=banking77_search_split, # 20 (input, expected_label) pairs
rounds=8,
k_failures=4,
carry_facts=True, # Appends CARRY_FACTS; set False if seed already lists all labels
)
print("Best search score:", result.best_score)
print("Optimized system prompt:\n", result.best_prompt)
To reproduce all five pre-registered batches (A.1–A.5), bootstrap confidence intervals, and figures directly from the open-source repository:
git clone https://github.com/shangliy/lineage-is-memory.git
cd lineage-is-memory
# Run the 2x5 factorial on the 7 primary tasks (Batch A.1 / A.2)
uv run python -m benchmarks.evolution.factorial --rounds 8 --code-rounds 12 --seeds 10
# Run the Banking77 & BBH-STE CARRY_FACTS experiments (Batch A.5)
uv run python -m benchmarks.evolution.carry_facts --tasks banking77 bbh_ste --seeds 10
References
- Yang, S., & Shi, Z. (2026). Lineage Is Memory: Parent Inheritance and Feedback Disclosure in Short-Budget LLM-Guided Optimization.
arXiv:2610.41157. Code & pre-registered raw data: github.com/shangliy/lineage-is-memory. - Agrawal, L. A., Tan, S., Soylu, D., et al. (2025). GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning.
arXiv:2507.19457. - Liu, L.-H., Tang, P., Singh, K. Y., & Ghadar, S. (2026). ESPO: Error-Structured Prompt Optimization via Diagnose, Diversify, and Stabilize.
arXiv preprint, 2026. - Bayat, F. F., Aminnaseri, M., Pezeshkpour, P., & Hruschka, E. (2026). Reflective Prompt Tuning through Language Model Function-Calling.
arXiv:2605.21781. - Chen, J., Wang, Z., & Chen, J. (2026). AdaGEPA: Adaptive Feedback Allocation for Reflective Prompt Optimization.
arXiv:2609.39927. - Koh, D., Mo, J., Le, B. H., et al. (2026). Contrastive Reflection for Iterative Prompt Optimization.
arXiv:2606.30840. - Singh, A., Gautam, S., Gupta, P., et al. (2026). Coding Agents Are Strong Prompt Optimizers.
arXiv:2609.26261. - Zhou, X., & Cheng, L. (2026). Explore More, Validate Less: Efficient Reflective Prompt Optimization.
arXiv:2610.27367. - Oved, T., Pony, R., Naparstek, O., & Barzelay, U. (2026). Evolution or Illusion? Rethinking Evaluation in LLM Evolutionary Search.
arXiv:2609.19799. - Yuksekgonul, M., Bianchi, F., Boen, J., et al. (2024). TextGrad: Automatic "Differentiation" via Text.
arXiv:2406.07496. - Yang, C., Wang, X., Lu, Y., et al. (2023). Large Language Models as Optimizers (OPRO).
arXiv:2309.03409. - Guo, Q., Wang, R., Guo, J., et al. (2023). EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers.
ICLR 2024. - Romera-Paredes, B., Barekatain, M., Novikov, A., et al. (2023). Mathematical Discoveries from Program Search with Large Language Models (FunSearch).
Nature, 625, 468–475. - Novikov, A., Vũ, N., Eisenberger, M., et al. (2025). AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery.
arXiv:2506.13131. - Lumer, E., Sen, S., Subbiah, V. K., & Paul, K. (2026). Reflective Prompt Optimization of a Deep-Research Agent: A Topic-Disjoint, Multi-Seed Protocol for Measuring Transferable Gains and Prompt-Level Overfitting.
arXiv:2610.12601. - Yang, S. (2026). What Should You Optimize When You Optimize an LLM Research-Ideation Harness? (Ideation Goodhart). github.com/shangliy/ideation-goodhart.