Harness Engineering: How Agents Improve the Systems Around Them

A failed run can reveal a defect in the machinery around a model. Recent systems let agents propose repairs, but independent tests must decide whether those repairs count as progress.

A coding agent starts a development server in the background. Twenty minutes later it needs to check the process, but the terminal returned only a line of text. There is no durable process identifier, the logs have left the context, and a second launch collides with the occupied port.

A better tool records the process ID, command, working directory, status, and log path. The agent can poll or stop the job. The model has not changed; its harness has.

The harness turns model predictions into a working process. It constructs context, exposes actions, preserves state, runs retries and delegation, and applies permissions. It also connects the work to tests and human oversight. What once looked like glue around a prompt is now a major determinant of agent performance.

Recent research goes further: an agent reads failed trajectories, proposes edits to this machinery, and tests them. Self-Harness reports a rise from 40.5% to 61.9% for one tested model on its regression split. Agentic Harness Engineering (AHE) reports 69.7% to 77.0% on Terminal-Bench 2 after ten iterations. The protocols differ, so these figures are not a ranking.

This is useful self-improvement only if the editable surface has a firm boundary. An agent that can rewrite its evaluator, expand its permissions, or raise its budget can improve a score without improving the work.

Capability Belongs to the Whole Stack

A model predicts a continuation. An agent adds a loop through an environment:

goal → context → model → action → environment → observation
             ↑                                  │
             └────────── next turn ─────────────┘

Each arrow contains design choices. The harness decides which files enter context, how compiler errors return, whether background jobs remain visible, and what happens after a failed test. A model may know that it should inspect an existing server but still fail because the tool provides no process handle.

OpenAI calls the part of Codex that coordinates the user, model, and tools the agent loop. In an internal agent-first experiment, it reports that Codex wrote roughly one million lines of code in about one-tenth the estimated human development time. Human engineers concentrated on making the repository legible, encoding architectural constraints, and building feedback loops. (agent loop, experiment)

Anthropic reached a related conclusion for work spanning several context windows. Its long-running harness leaves plans, progress files, and verifiable increments for later sessions. Continuity comes from durable artifacts rather than new neural memory. (Effective harnesses for long-running agents)

base model
├── message and tool format
├── context construction and compression
├── persistent memory and artifacts
├── workflow, retries, and parallelism
├── evaluators and observability
└── permissions, sandbox, and oversight

A benchmark measures this stack even when its leaderboard displays only a model name. A harness can reveal capability or hide it, and one tuned to a model's habits may work poorly with its successor.

This complicates model comparisons. A score can rise because the model reasons better, because the tool description became clearer, because context retrieval improved, or because the system made more attempts. All are legitimate engineering gains, but they support different claims. Reporting the harness version, budget, and evaluation protocol is therefore necessary to interpret an agent result.

The coupling also explains why apparently small interface defects have broad effects. An ambiguous edit tool does not fail on only one task; it repeatedly turns correct intentions into malformed patches. Missing process state makes every asynchronous workflow harder. Conversely, a durable fix can transfer across tasks because it changes the environment in which many decisions become actions.

The harness is not a substitute for a capable model. It cannot supply knowledge the model lacks, and elaborate orchestration can add latency or new failure modes. Its role is narrower: preserve relevant state, make consequences observable, and prevent avoidable interface errors from consuming the model's reasoning budget. That distinction matters when reading claims of self-improvement. The system may become better at completing a benchmark without the underlying model acquiring a new general ability.

Three Durable Patterns

First, the outcome of an action must be visible. Andrej Karpathy's autoresearch illustrates the minimum loop: modify a small training setup, run it for five minutes, inspect the metric, then keep or discard the edit. Its credibility comes from the fixed measurement and budget, not from autonomy. A trace saying “timeout” is weak evidence; a trace showing a successful launch, lost process state, and a later port collision identifies a mechanism.

Second, state must outlive the prompt. Logs, diffs, screenshots, and test reports do not all belong in active context. They can remain in addressable files while the prompt holds an index and the current state. Memory is the material the system can retrieve; context is the subset the model sees now. Durable artifacts also allow another person to reconstruct a conclusion.

Third, parallel work needs receipts. A parent agent should be able to start, inspect, interrupt, and collect a worker. Each worker needs bounded ownership, visible status, and a durable output. The same contract applies to any background process.

Parallelism helps only where assignments can be separated. Several workers editing the same files create integration costs and uncertain ownership; several workers checking independent sources can return distinct evidence. The harness must retain which worker produced each artifact and whether its result was accepted. Starting more model sessions without collection and attribution is additional concurrency, not a better workflow.

These patterns share a deliberately plain objective. The runtime does not need to imitate human memory or management. It needs stable identifiers, retrievable artifacts, explicit lifecycle states, and verifier outputs that point back to evidence. Those primitives make a failure available to the next experiment rather than allowing it to disappear in a transcript.

From Failed Run to Harness Edit

Prompt engineering changes instructions. Context engineering changes the evidence presented to the model. Workflow engineering changes sequencing, verification, retries, and delegation. Harness engineering can modify the runtime itself: tools, parsers, middleware, memory, and recovery.

Research increasingly treats all these layers as searchable. Agentic Context Engineering curates lessons from successful and failed trajectories; Meta Context Engineering evolves the function that retrieves and formats them. Automated Design of Agentic Systems and AFlow search executable workflows.

A credible loop follows a short sequence:

read failed runs
→ identify a recurring mechanism
→ propose one bounded change and a prediction
→ test it against separate evidence
→ keep or revert it

Meta-Harness: Preserve the Trace

Meta-Harness argues that text optimizers summarize failures too early. It gives a coding agent a directory of earlier implementations, scores, and trajectories, which the agent explores with file and shell tools.

The paper reports a 7.7-point gain over a strong context-management system in online text classification with four times fewer context tokens. In retrieval-augmented mathematical reasoning, one discovered harness gains an average 4.7 points on 200 IMO-level problems across five held-out models. Candidates also beat the hand-engineered TerminalBench-2 baselines considered in the paper.

The boundary matters. TerminalBench search and final evaluation use the same 89 tasks. The authors check for task-string leakage, but this is not held-out generalization. Only the mathematical retrieval experiment tests transfer to five unseen models.

Self-Harness: Make One Bounded Change

Self-Harness groups failures by causal mechanism, not merely by verifier label. A timeout caused by an invisible process needs a different repair from one caused by aimless exploration. The proposer sees a bounded editable surface and earlier attempts, then generates narrow alternatives. Candidates run on a held-in set and a regression split hidden from the proposer.

The paper reports 40.5% → 61.9% for MiniMax M2.5, 23.8% → 38.1% for Qwen3.5-35B-A3B, and 42.9% → 57.1% for GLM-5 on Terminal-Bench-2. Scores from the regression split are consulted in every promotion round, so it is a selection set rather than an independent final test. The results apply to the tested models, tasks, and starting harness.

AHE: Record the Prediction

Agentic Harness Engineering requires every edit to predict the failure it should correct, the behavior it should preserve, and possible regressions. Summaries remain linked to raw evidence, and editable components live in inspectable files.

Ten iterations move pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, above the reported Codex-CLI baseline of 71.9%. The frozen harness transfers to other model families with reported gains of 5.1 to 10.1 percentage points and uses 12% fewer tokens than the initial harness on SWE-bench Verified. Most gains came from tools, middleware, and long-term memory rather than the system prompt.

Predictions did not make the agent a reliable forecaster: regression forecasts reached 11.8% precision and 11.1% recall. Writing a prediction improves attribution; tests still determine whether the change survives.

The studies use different models, benchmarks, and budgets. Their common contribution is a method for turning operational failures into testable edits.

Search, Recursion, and Cost

Harness optimization belongs to a broader effort to improve the process that searches for solutions. Self-Taught Optimizer applies an optimizer to the function that improves programs. Average performance rose across iterations with GPT-4, while weaker models could degrade. Recursion amplifies diagnostic ability; it does not supply it.

Darwin Gödel Machine keeps an archive of agents, selects different parents, and evaluates code-editing descendants. With Claude 3.5 Sonnet, the paper reports 20.0% to 50.0% on a 200-task subset of SWE-bench Verified, after filtering on 10- and 60-task sets, and 14.2% to 30.7% on Polyglot. The sandboxed, human-supervised SWE-bench search cost roughly $22,000 at historical API prices.

AlphaEvolve uses evolutionary search where programs have strong automatic evaluators. Google DeepMind reports that it matched the state of the art on about 75% of more than fifty mathematical problems and improved it on about 20%. These are vendor-reported results.

All three depend on executable variation, repeatable evaluation, an evidence-preserving archive, and an explicit selection rule. Code suits this process because proposals can be run. The advantage weakens when correctness is hard to measure.

Weight updates are less reversible. SIA chooses between harness and weight changes, but its evidence is preliminary and its task agent is weaker than the directing models. Continual Harness studies online harness changes in long-horizon games and distillation from a stronger teacher. A faulty tool can be reverted; a weight update may alter behaviors outside the benchmark. Broken interfaces, context, and workflows should therefore be tested before changing weights.

Autonomy Depends on Verifiability

Self-improvement is most convincing where “better” is cheap and repeatable to measure: a test passes, a kernel is correct and faster, or a loss falls under a fixed budget. Software maintenance, research, strategy, and design have weaker evaluators. A diff can pass tests while increasing future migration costs; a technically valid experiment can pursue the wrong question.

The AI Scientist can coordinate ideas, code, experiments, analysis, and writing. That is substantial process automation, but scientific value still depends on questions, baselines, evidence, and domain judgment. A false positive is especially damaging when it is promoted into the machinery that generates later experiments.

The editable surface can include instructions, tool descriptions, context strategies, memory indexes, workflows, and bounded application code. Held-out tests, immutable traces, model and reasoning budgets, network policy, credentials, cost limits, and deployment rules should remain outside it.

Even with that separation, fixed benchmarks invite harness overfitting. Tests omit maintainability and user impact. Search costs can hide behind a final score, selection can collapse useful diversity, and simultaneous changes weaken causal attribution. Short evaluations may also reward a harness that completes today's task while making tomorrow's repository harder to understand.

Evidence Must Survive the Optimization Loop

A score alone cannot explain why a candidate improved. The record needs the harness version, declared model, reasoning and time budgets, tool calls, outputs, verifier result, cost, and terminal cause. Aggregate reports should link back to the raw trajectory. Otherwise a later investigator cannot distinguish a useful interface change from a lucky sample or an undeclared increase in compute.

The diagnosis also needs three levels. The verifier supplies a symptom such as “timeout.” The trajectory may show the behavior: the agent started a second server without checking the first. The harness analysis proposes the mechanism: the terminal returned output but did not preserve a process handle. Only the last level suggests a reusable system change. Grouping every timeout together would mix unrelated causes and encourage a generic prompt reminder.

Experimental roles should remain explicit. A held-in set can confirm that the intended failure is affected. A regression set protects behaviors that already work. Once its scores influence repeated promotion decisions, however, that regression set has joined the training process. Calling it “hidden” because the proposer never sees its trajectories does not make it an independent final evaluation. A third untouched set is needed for that claim.

Cost is part of the evidence. Increasing a timeout, adding retries, or calling a stronger model may lift pass rate while making the system slower or more expensive. Useful comparisons therefore keep the model and budget fixed or report a quality-cost curve. AHE's lower token use and Darwin Gödel Machine's estimated search bill are relevant for this reason: the final accuracy omits the resources used to discover it.

Finally, the intervention should be attributable. One narrow edit per hypothesis is easier to evaluate than a bundle that changes the prompt, tools, and memory together. Some repairs genuinely require a package; the experiment should label it as such rather than inventing a causal story afterward. Rejected candidates should also remain in the archive, both to prevent repetition and to preserve alternatives that may matter under a different model.

These controls do not require approval for every tool call. Human decisions belong at high-impact gates: changes to permissions or evaluators, substantial spending, publication, production access, and irreversible migrations. Routine searches and reversible file edits can remain inside the loop. This places oversight around consequences rather than keystrokes.

What the Results Establish

Harness engineering makes a narrow but important form of self-improvement possible. The agent can inspect a failure, propose a change to its operating system, and preserve that change when separate evidence supports it. The model weights need not move.

The reported gains show that tools, state, context, and workflow materially affect measured capability. They do not show an agent that can improve without boundaries, and several experiments reuse selection tasks or rely on specific benchmarks. The credible unit of progress is therefore not a fluent diagnosis or a higher score alone. It is a versioned change, a preserved trace, a fixed budget, and an evaluator the proposer cannot edit.

Sources and scope: Lilian Weng's “Harness Engineering for Self-Improvement”; OpenAI on harness engineering and the Codex agent loop; Anthropic on long-running agents; and the linked papers for ACE, MCE, Meta-Harness, Self-Harness, AHE, Darwin Gödel Machine, STOP, AlphaEvolve, SIA, Continual Harness, and The AI Scientist. Results use different protocols and should not be read as a ranking.