Harness²: What an Agent Can Learn by Changing Its Harness

An agent can fail because it has no procedure for checking an output file, forgets a constraint halfway through a task, or delegates work without checking what comes back. Those failures suggest changes to the system around the model: add a validation script, revise a tool instruction, or change how subagents report their results.

Harness² makes that surrounding system editable during task execution. It runs alternative harnesses on the same task, examines what changed, and retains experience that can guide later edits. The executor’s model weights stay fixed. [1]

The difficult part is deciding which edits deserve to survive. A cleaner trajectory can still produce a wrong answer. Harness²’s execution comparisons provide evidence about behavior, but the system does not receive the task’s ground-truth grade while adapting. That distinction determines how I would evaluate it.

The editable object is larger than a prompt

The project exposes eight harness components: system prompt, guardrails, memory, tool guidance, subagents, skills, scripts, and plugins. Together, they shape which context the model receives, how it calls tools, what it delegates, and what happens inside the execution loop. [1]

These components can solve different kinds of problems. A prompt can remind an agent to inspect an artifact. A script can perform a repeatable check. A skill can describe a domain procedure. A subagent configuration can change who performs an independent review.

Consider a spreadsheet task. If an agent returns a workbook without checking its formulas, a harness edit could require reopening the saved file and checking calculated cells. That is a hypothetical example of a useful intervention, not a reproduced Harness² result. The important question is whether executing the edited harness reveals that the check actually happened and affected the deliverable.

The released code separates the main responsibilities. `core/recursion.py` schedules proposal, probe, and composition work. `core/context.py` assembles the evidence available to the editor. `core/library.py` stores reusable harnesses and experience, while `core/selection.py` handles output selection and adoption. The official repository documents these implementation details; the paper supplies the experimental claims. [1]

Compare executions before composing an edit

A basic Harness² step has three parts.

Propose. First run the current harness to obtain a reference trajectory and deliverable. An editor then proposes two candidate harnesses. One addresses a deficiency that might recur across domains, such as planning or delegation. The other addresses the current domain’s procedures and constraints. These are executable edits to harness components. [1]

Probe. Run both candidates on the same task under matched model and environment configurations. Compare their trajectories and deliverables with the reference. Did the proposed check execute? Did it expose a missing input? Did the new instruction cause the agent to spend its budget on an irrelevant detour?

Compose. Read the candidate diffs and execution evidence, then form a successor harness. Composition can merge changes, revise conflicts, select one candidate, or preserve the current harness. Execute that successor as well. The result includes the revised harness, its actual execution, and the observations used to justify the changes. [1]

Figure 3 from Harness². The left side adapts the execution harness for one task through proposals, probes and composition. The right side records supported experience for future tasks; it does not update the executor’s model weights. Source: arXiv:2610.14224, CC BY 4.0.

Read the figure from left to right. The candidate probes are the point where a proposed improvement is exposed to execution. The return path matters too: experience changes how future harness edits are made. It does not mean the next task automatically inherits everything that happened on this one.

This design gives composition more information than a free-form request to “improve the prompt.” It still leaves a causal ambiguity. Agent executions vary. If a candidate performs better once, the harness change may have helped, or the executor may simply have taken a more fortunate path. The paper explicitly identifies stochasticity and shared model biases as limits on these behavioral contrasts. [1]

A library carries lessons across tasks, not a permanently mutated agent

After a task, a separate reflection step inspects the improvement record. The library retains an adopted executable harness together with an `EXPERIENCE.md` entry describing the applicable situation, lesson, and evidence. If the evidence is insufficient, reflection need not add an entry. [1]

The next task starts from the common base execution harness. Earlier components can be reused or adapted during proposal, guided by the library. They are not unconditionally installed as the new default.

That reset is useful. A procedure that helps generate a legal document may be irrelevant to a coding task. Even within a domain, a successful edit may depend on a particular file format or tool. Storing the conditions under which an edit helped gives a future editor something more useful than a generic instruction to “be careful.”

There are therefore two changing objects: the execution harness being tested on the current task, and the improvement library that informs later refinement. The improvement process does not freely rewrite every part of itself. In the paper’s formulation, model weights, stage instructions, and the decision rule stay fixed while the library evolves. [1]

The recursion budget buys width or depth

The budget parameter k controls two different schedules.

Parallel recursion collects k base executions, proposes k invariant and k domain-specific candidates, probes them, and performs k independent compositions using the shared candidate pool. It increases the width of a population update.

Sequential recursion begins with the base evidence and chains k refinement steps. Each step can use the preceding harness and accumulated execution history. It increases the depth of refinement. At k=1, both reduce to the basic contrastive step. [1]

Figure 4 from Harness². Parallel recursion explores candidate edits and compositions across a shared pool. Sequential recursion carries the revised harness and accumulated evidence into the following step. Equal k does not imply equal wall-clock latency. Source: arXiv:2610.14224, CC BY 4.0.

The paper’s implementation schedules 4k executor rollouts and 3k improvement stages, including the base executions. At k=1, that is four executor rollouts plus three improvement stages. At k=5, it is twenty rollouts plus fifteen improvement stages, before accounting for selection and post-task reflection. Individual sessions may contain many model calls.

For k>1, a selector compares anonymized deliverables with the request and source materials. It returns an existing output; it does not synthesize a fresh answer. Adoption here means “chosen by the selector,” not “verified correct.” [1]

Parallel execution may reduce elapsed time when enough workers are available. It does not remove the computation or API charges. A deployment decision needs total tokens, cost, and latency across the whole pipeline.

What the reported results establish

The paper evaluates frozen Gemini 3.5 Flash and GLM-5 executors under OpenCode and Codex, giving four model and initial-harness configurations. Claude Opus 4.8 is the default harness editor. The main results report means and standard deviations over three task streams with shuffled task order. [1]

The test scope is specific:

WorkBuddy-Bench uses 40 Code, 35 Web, and 25 Office test tasks. Its task scores combine deterministic checks with model-based judging.
JobBench reports 34 held-out professional tasks, scored against weighted rubrics.
LAB reports 100 held-out legal tasks. Its Partial metric is a per-criterion pass rate from a model judge, not a binary measure of legally correct deliverables.

For one concrete configuration, Gemini 3.5 Flash with OpenCode on WorkBuddy-Bench, Table 1 reports the following macro-average across the three domains: [1]

Setting · Reported average score

Base harness · 72.4, SD 0.9

Harness², k = 1 · 77.5, SD 2.5

Harness², parallel k = 5 · 80.8, derived from the reported +3.3 change in mean

The single-step gain is 5.1 score points. The larger budget adds 3.3 more. The recursion row reports a change in mean, so I have not attached an invented standard deviation to the derived 80.8.

The largest one-step gains across the reported configurations are 11.5 WorkBuddy-Bench points, 4.5 JobBench points, and 11.8 percentage points on LAB Partial. Those are different metrics and configurations. They should not be combined into a single “percent improvement.” Harness² also does not lead every metric: Meta-Harness is ahead on LAB Partial for Gemini under Codex before the extra recursion budget. [1]

Ablations provide some evidence for the design choices. For Gemini/OpenCode at k=1, removing the library reduces WorkBuddy average from 77.5 to 74.4 and JobBench from 36.0 to 34.2. Removing composition yields 75.9 and 34.1 respectively. Those results support the usefulness of these components in that setup; they do not show that every retained lesson transfers.

How this relates to reflective memory

Reflexion supplies a useful earlier comparison. Its Actor generates actions, an Evaluator supplies feedback, and a Self-Reflection component turns experience into text retained in episodic memory. Later trials receive those reflections without updating the actor’s weights. Its feedback can include tests or environment signals, so it should not be described as universally label-free. [2]

Agentic Context Engineering, or ACE, organizes context adaptation through a Generator, Reflector, and Curator. It emphasizes structured incremental updates to a playbook so useful detail is not lost through repeated rewriting. I see it as a relevant comparison for maintaining experience, not as evidence that Harness²’s full workflow works. [3]

Harness² extends the editable surface to an executable harness and tests proposed edits through new executions. It also retains experience about the refinement process. The practical distinction is what persists: a reflection about a failed attempt, an evolving context playbook, or executable components accompanied by observations about when they helped.

For a related deployment question, my SearchJev analysis considers when an agent should delegate or stop. Harness² adds a prior decision: whether to spend more computation changing the agent’s operating procedure at all.

The limits I would test before deployment

Behavioral improvement is not correctness feedback. The improver sees trajectories and deliverables, while benchmark graders evaluate the results separately. A convincing explanation or a more complete-looking document can still be wrong. The absence of ground-truth feedback during adaptation is a constraint, not a guarantee that evaluation is unnecessary. [1]

Public evaluation material can leak into execution. The release’s experiment guide warns that JobBench rubrics are public and reports a validation rollout downloading its task’s rubric. The release does not restrict network access by default. This is a documented evaluation-hygiene risk; it does not by itself establish that the paper’s reported scores are contaminated.

Executor steps do not measure pipeline cost. A shorter selected trajectory can coexist with a larger total bill because proposals, probes, composition, selection, and reflection all consume resources. I would not infer cost savings from a reduction in reported executor turns.

Editable guardrails need an external boundary. Guardrails are among the harness components the framework can edit. In a production system, permission policy, secrets, network egress, and deployment approval should remain outside the editable experiment. That is my deployment recommendation, not a feature I have verified in the release.

The repository is Apache-2.0, but benchmark materials have separate terms. Its local deterministic demo can illustrate both recursion schedules without model credentials. That demonstration would test orchestration, not reproduce the paper’s model-backed benchmark gains. I have not run either here.

An equal-cost experiment would decide whether I would use it

I would begin with a small, non-production task stream with independently checkable deliverables. Compare a fixed harness, repeated execution with output selection, Harness² without persistent experience, and Harness² with its online library. Match total model spending across arms, including the editor and reflector.

Keep task order, available tools, source documents, and evaluation access controlled. Record library growth, artifact correctness, regressions, total dollars, and wall-clock time. A useful edit should improve the deliverable under an independent check, not merely make the trajectory easier for another model to praise.

Then test transfer on tasks the library has not seen. Include a deliberately mismatched library from another domain to measure whether experience helps because it is relevant or merely because it adds instructions.

The outcome I would care about is a measured advantage over equal-cost retries, with a library that helps on later tasks without accumulating regressions. That would justify using Harness² as a harness-engineering process. Until then, the code and paper offer a concrete way to investigate how much agent reliability can be improved outside the model weights.