Harness Engineering: How AI Agents Learn to Improve the System Around Them

The first AI system capable of improving itself may not rewrite its own brain. It may read a failed run, change the machinery around the model, and keep the change only if an evaluator it cannot control says the system got better.

Imagine a coding agent starting a development server in the background. The command returns, the task moves on, and twenty minutes later the agent needs to know whether that process is still alive.

In one setup, the terminal tool returned a line of text and forgot the rest. The process has no durable identifier. Its logs have scrolled out of context. The agent starts another server, collides with the old port, misreads the error, and spends the next ten turns fixing a problem it created.

In another setup, background jobs are first-class objects. The harness records the process ID, command, working directory, status, and log path. The agent can poll the job, read new output, or stop it cleanly. The second system gives the same model a world it can inspect.

That difference is the subject of harness engineering.

A harness is the code that turns a model prediction into a working process. It decides what the model sees, which actions are available, how results come back, where state survives, when the agent must stop, and who gets to decide whether the work succeeded. The model chooses a next action; the harness makes that action part of a job that may last minutes, hours, or days.

For years, this layer looked like glue: a system prompt, a parser, a few retries. That description no longer fits. Changes around a fixed model can now move benchmark scores, costs, and reliability by large margins. Self-Harness, for example, reports a regression-split improvement from 40.5% to 61.9% for one tested model. Agentic Harness Engineering reports a rise from 69.7% to 77.0% on Terminal-Bench 2 after ten harness iterations.

The more interesting development is not the size of those gains. It is how the changes were found. Several recent systems let an agent inspect failed trajectories, propose edits to its own tools or workflow, and test those edits against separate evidence. In other words, the harness is becoming an object the agent can study.

This is a practical form of self-improvement, but only under one non-negotiable condition: the actor changing the system cannot also control the definition of success. Let the agent rewrite the evaluator, expand its permissions, or quietly increase its budget, and it is no longer learning from failure. It is learning to move the goalposts.

This article builds on Lilian Weng’s Harness Engineering for Self-Improvement and asks a narrower question: what would it take for an agent to turn its own mistakes into reliable improvements? The answer has less to do with science-fictional recursion than with logs, files, tests, clean interfaces, and disciplined experimental design.

The Model Is Only One Layer

A language model, by itself, receives a sequence and predicts a continuation. The continuation may contain a plan, a diagnosis, or a tool call, but the model does not keep a server alive, inspect a filesystem, or remember what happened yesterday.

An agent adds time.

It observes an environment, chooses an action, sees the consequence, updates its plan, and acts again:

goal → context → model → action → environment → observation
             ↑                                  │
             └────────── next turn ─────────────┘

The diagram is simple. Every arrow hides a design decision.

• Which files enter the context, and which stay on disk?
• Does a compiler error return in full or as a summary?
• How does the system identify and gate a destructive command?
• Does a background process become durable state or vanish into terminal output?
• When a test fails, does the agent retry, change strategy, or stop?
• Who decides that the final result is correct?

Those decisions can reveal capability or bury it. A model may understand perfectly well that it should poll a server before starting another one, yet fail because the tool exposes no process handle. Training the model harder would be an expensive way to compensate for a broken interface.

OpenAI calls the part of Codex that coordinates the user, model, and tools the agent loop. In an internal agent-first experiment, the company reports that Codex wrote roughly one million lines of code in about one-tenth the estimated human development time. The revealing part is where the human work went: engineers made the repository legible to agents, encoded architectural constraints, built feedback loops, and controlled the entropy of high throughput. (agent loop, harness engineering experiment)

Anthropic found the same principle at a smaller scale while studying work that spans several context windows. Its long-running agent harness uses an initializer to prepare the environment and durable artifacts. Later sessions read the plan, work in verifiable increments, and leave a progress file for the next session. No new neural memory is required. The continuity comes from files, tests, and a disciplined handoff. (Effective harnesses for long-running agents)

Observed capability therefore belongs to the whole stack:

base model
├── post-training policy
├── message and tool format
├── action selection and description
├── context construction and compression
├── persistent memory and artifacts
├── workflow, retries, and parallelism
├── evaluators, tests, and observability
└── permissions, sandbox, and oversight

An agent benchmark measures this system even when the leaderboard prints only a model name. A new harness can lift a score without making the transformer more capable in general. A new model can also perform worse inside a harness tuned around the habits of its predecessor. Model and harness are separate levers, but they are not independent ones.

Three Patterns Make the Difference

The most dependable agent patterns do not look futuristic. They look like good software engineering.

Make the Outcome Visible

An agent can improve only when its last action produces a useful signal:

plan → execute → measure → diagnose → modify → execute again

Andrej Karpathy’s autoresearch strips this loop to its essentials. An agent changes a small training setup, runs it for five minutes, checks the metric, keeps or discards the edit, and tries again. The researcher does not write every experiment. The researcher writes program.md, the operating contract for the automated research loop.

The autonomy is not what makes this credible. The evaluator is. Without a stable metric, “iterate until you improve” means “generate variations until one looks persuasive.” A useful loop exposes the current state, the permitted action, the outcome signal, and the rollback rule. Remove any one of them and attribution becomes murky: did the idea work, or did the run get more compute, a different test, or a measurement bug?

Our background-process failure becomes useful only when the system can record it precisely: the server was launched; no persistent handle was returned; a second launch caused a port collision; the task then failed. “The agent got confused” is not a diagnosis. The trace has to show where the interface stopped representing reality.

Let State Outlive the Prompt

Long tasks create more material than should fit in a context window: logs, diffs, screenshots, test reports, discarded attempts, and decisions. Replaying all of it on every turn is expensive and makes the relevant evidence harder to find.

The filesystem gives models an abstraction they already know how to navigate. The active context holds an index and the current state; detail remains in addressable files. The agent does not need to remember the entire server log. It needs to know that the job exists, where the log lives, and how to fetch the next unread chunk.

This is the difference between memory and context. Memory is what the system can retrieve. Context is what the model sees now. Good harnesses do not pour all memory into the prompt. They retrieve the smallest useful slice and keep the underlying artifact available for inspection.

Durable artifacts also make the work auditable. If a conclusion points to a trajectory, a test result, and a diff, another person can reconstruct it. A conclusion that survives only as fluent prose has no such receipt.

Give Parallel Work a Receipt

Subagents help when tasks are genuinely independent: searching several sources, testing distinct fixes, or reviewing non-overlapping modules. Starting more chats, however, is not coordination.

A parent agent needs the equivalent of a small process manager: start, inspect, interrupt, collect, and integrate. Each worker needs bounded ownership, visible status, and a durable output. The same rule applies to the development server. “I started it” is not enough. The caller needs an identifier, current state, logs, and a way to stop it.

The common principle is unglamorous: do not ask the harness to act intelligent. Ask it to make state explicit, evidence cheap to recover, and failure difficult to hide.

How a Harness Learns from a Failed Run

The object engineers optimize has expanded over time.

Prompt engineering changes the initial instruction. Context engineering decides which evidence the model sees and how it is arranged. Workflow engineering changes the order of operations: when to call a verifier, branch, retry, or delegate. Harness engineering reaches into the runtime itself — tools, middleware, memory, parsers, checkpoints, and recovery. A meta-harness then searches for a better way to propose and select those changes.

The steps are related, but code changes the character of the search. An instruction can ask the agent to remember background jobs. A harness edit can add a process_status tool, store job metadata, and make the behavior executable, versioned, testable, and reversible.

Agentic Context Engineering treats context as an evolving playbook built from successful and failed trajectories. A generator gathers experience, a reflector extracts reusable lessons, and a curator updates structured items rather than repeatedly rewriting one giant prompt. Meta Context Engineering moves up one level and evolves the function that retrieves, filters, and formats context.

Workflow research follows the same direction. Automated Design of Agentic Systems and AFlow let a meta-agent propose executable programs or workflow graphs, run them, score them, and retain the useful variants. The hand-designed workflow becomes a starting point rather than a permanent architecture.

Across these approaches, a credible improvement loop has five moves:

read the failed runs
→ identify a recurring mechanism
→ propose one bounded change
→ predict what it should fix
→ test it against separate evidence

Three recent systems make different parts of that loop concrete.

Meta-Harness: Let the Agent Read the Failed Runs

Meta-Harness begins with a simple complaint about text optimizers: they summarize too soon. A three-line report can say that a candidate failed while erasing the clue that explains why.

Instead of stuffing the full history into a huge prompt, Meta-Harness gives a coding agent a directory containing earlier implementations, scores, and trajectories. The agent explores that evidence with normal file and shell tools. Each candidate is a small repository with both code and experimental receipts.

The paper reports a 7.7-point gain over a strong context-management system in online text classification while using four times fewer context tokens. In retrieval-augmented mathematical reasoning, a harness found in one setup improves performance by an average of 4.7 points on 200 IMO-level problems across five held-out models. Discovered candidates also beat the hand-engineered baselines considered for TerminalBench-2.

Those results do not establish a universal harness. The TerminalBench search and final evaluation use the same 89 tasks. The authors check for task-string leakage, but this is not a clean held-out generalization test. Only the mathematical retrieval experiment measures transfer across five unseen models. Within that boundary, the finding is still useful: structured access to experimental history can beat an aggressively compressed summary of it.

Applied to our running example, the difference is concrete. A summary might say “server task timed out.” The raw trajectory reveals that the first launch succeeded, its state disappeared from view, and the second launch collided with it. The useful fact lived in the sequence, not the label.

Self-Harness: Turn the Trace into One Small Edit

Self-Harness asks a cleaner question: can the same model doing the work also improve its harness, without a stronger external agent designing the fix?

First, it groups failures by causal mechanism. Two tasks may both time out because of entirely different behaviors: one agent never polls a background job; another keeps exploring after it already has the answer. Calling both “timeout” produces a generic remedy. A useful record separates the verifier’s result, the behavior that caused it, the mechanism exposed by the trace, and the evidence connecting them.

Next, the proposer receives a clearly bounded editable surface, examples of behavior that must survive, and a history of earlier edits. It generates several distinct, narrowly scoped changes. Finally, every candidate runs on a held-in set meant to exercise the target weakness and on a regression split hidden from the proposer. Rejected ideas remain in the record so the system does not keep rediscovering the same failure.

The paper reports improvements for all three tested models on Terminal-Bench-2: 40.5% → 61.9% for MiniMax M2.5, 23.8% → 38.1% for Qwen3.5-35B-A3B, and 42.9% → 57.1% for GLM-5. The trajectories from that split are hidden from the proposer, but the scores are consulted at every promotion round. It is therefore a selection and regression set, not an independent final test.

The models, tasks, and initial harness are specific, so the percentages are not a general law. The experimental rule travels better: start from a recurring error, state what an edit is meant to fix, and reject it if separate regression evidence gets worse.

AHE: Write the Prediction Before Seeing the Score

Agentic Harness Engineering adds another discipline: every edit carries a prediction. The system records which failure the change should correct, which healthy behavior it should preserve, and what it might break. In the next round, the question is not merely whether the total score rose. It is whether the score rose for the reason the edit claimed.

To make that possible, editable components live in inspectable files; millions of trajectory tokens become per-task reports, aggregate patterns, and benchmark-level summaries; and every summary links back to raw evidence. A proposed background-job fix might predict fewer port collisions on asynchronous tasks while warning that extra polling could increase token use or delay completion elsewhere.

The paper reports that ten iterations move pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, above the reported Codex-CLI baseline of 71.9%. The resulting frozen harness transfers to other model families with reported gains of 5.1 to 10.1 percentage points and uses 12% fewer tokens than the initial harness on SWE-bench Verified.

Most of the gains came from tools, middleware, and long-term memory, not the system prompt. Operational structure transferred better than strategic prose. Yet writing predictions down did not make them accurate: regression forecasts reached only 11.8% precision and 11.1% recall. The prediction improves the audit trail; tests remain indispensable because the agent is a poor judge of what its own edit will break.

The table is not a ranking. Its numbers come from different models, benchmarks, and budgets. It shows how much territory the label self-improving now covers: memory, context construction, workflows, complete harnesses, and solution code.

When the Improver Becomes Part of the Search

Harness research belongs to a longer attempt to optimize not just an answer, but the process that searches for one.

Self-Taught Optimizer optimizes the function that improves programs. That new improver can then apply the same procedure to itself. Recursion did not guarantee progress in the experiments: the average improver became better across iterations with GPT-4, while weaker models could degrade. A recursive loop amplifies whatever diagnostic competence is already present; it does not create that competence for free.

Darwin Gödel Machine takes an empirical route. It keeps an archive of agents, selects different parents, lets them edit their own codebase, and evaluates their descendants. The search forms a tree rather than one winning lineage, preserving strategies that may look weak now but become useful stepping stones later.

Using Claude 3.5 Sonnet as the foundation model, the paper reports an increase from 20.0% to 50.0% on a 200-task subset of SWE-bench Verified, after progressively filtering candidates on 10- and 60-task sets, and from 14.2% to 30.7% on Polyglot. The discovered changes include better editing tools, long-context management, and peer review. The runs used sandboxing and human oversight. They were not cheap: the authors estimate the SWE-bench search at roughly $22,000 using historical API prices.

AlphaEvolve applies evolutionary search to programs with strong automatic evaluators. Fast models generate breadth, stronger models propose deeper changes, and measured programs seed the next generation. Google DeepMind reports that across more than fifty mathematical problems, AlphaEvolve rediscovered the state of the art in about 75% of cases and improved it in about 20%. Those are vendor-reported results, alongside applications in data centers, chip design, and AI training.

All three approaches depend on four engineering ingredients:

executable variation
+ cheap, repeatable evaluation
+ an archive that preserves evidence and diversity
+ an explicit selection rule

Code is unusually well suited to this kind of search because a proposal can be run rather than debated. That advantage shrinks as soon as correctness becomes difficult to measure.

Changing the model weights raises the stakes again. SIA — Self Improving AI with Harness & Weight Updates uses a meta-agent to propose a harness, a task agent to execute it, and a feedback agent to choose whether the next round should edit the harness or update the weights. The architecture avoids a one-size-fits-all response to failure, but the evidence is preliminary: the task agent is much weaker than the models directing it, and the baselines do not cleanly isolate every alternative. Continual Harness explores online harness updates in long-horizon games and distills labels from a stronger teacher into the policy model on low-reward trajectories.

The distinction that matters is reversibility. A bad process tool can be removed with a code revert. A weight update can spread its effects across thousands of behaviors the current benchmark never exercised. Before changing weights, a system should rule out broken interfaces, poor context, and weak workflows — then require a much broader, untouched evaluation before promotion.

Autonomy Follows Verifiability

Self-improvement works best where “better” is cheap to measure.

A kernel either produces the correct result or it does not; once correct, its speed can be measured. A test passes or fails. A loss moves under a fixed budget. An algorithm uses fewer operations. None of these evaluators is perfect, but each provides frequent, repeatable feedback that is hard to charm with eloquent prose.

Move toward long-term software maintenance, open-ended research, strategy, or design and the ground softens. A diff can pass every test while making the next migration painful. A paper can contain valid citations and a technically successful experiment while pursuing the wrong question. An interface can please a judge model and still frustrate every real user who touches it.

The faster, more causal, and more resistant to shortcuts the feedback is, the more autonomy the optimization loop can safely use.

The AI Scientist and related systems can coordinate idea generation, code, experiments, analysis, and writing. That is real process automation. It is not the same as making a discovery. Correct citations, faithful implementations, relevant questions, sensible baselines, and sound conclusions still depend on evidence chains and domain judgment.

The dangerous failure is not always an absurd answer. It can be a noisy experiment that looks successful, gets promoted, and then reshapes the process that generates every later experiment. A false positive can become infrastructure.

Draw a Hard Line Through the System

The agent may edit operating instructions, tool descriptions, output formats, context strategies, memory indexes, workflows, checkpoints, and bounded application code. It should not be able to rewrite held-out tests, immutable traces, the declared model and reasoning budget, network policy, credentials, authorization boundaries, cost limits, or deployment rules.

That separation blocks three obvious shortcuts. The agent cannot specialize the evaluator until a benchmark quirk looks like progress. It cannot raise the score by silently changing the model, timeout, or token budget. And it cannot solve a task by granting itself access outside the sandbox.

Separate evaluators and sandboxes do not eliminate risk. They make false progress harder to manufacture without leaving evidence. Several problems remain:

• Harness overfitting. Fixed weights do not prevent prompts, tools, or workflows from memorizing benchmark regularities.
• Incomplete measurement. Tests rarely capture maintainability, future migration cost, compatibility, or user impact.
• Search cost. A final score hides the thousands of rollouts spent finding it; useful comparisons need quality-cost curves.
• Collapsed diversity. Selection loops exploit what already worked and can discard promising approaches too early.
• Weak causal attribution. If prompt, tool, and memory all change together, the system has found a package correlated with success, not the mechanism.
• Short evaluation horizons. A harness that closes tasks quickly today may leave a repository that tomorrow’s agents understand less well.

These limits do not cancel the reported gains. They set the boundary of the claim: parts of system engineering can now be automated when the environment provides strong evidence. That is not the same as an intelligence that can improve without boundaries in any domain.

A Practical Playbook

You do not need a Darwin Gödel Machine to use the ideas that survive this research. Start with the failed background process and make each step operational.

1. Make Failure Queryable

Record the task, harness version, actions, tool outputs, duration, cost, verifier result, and terminal cause. Index the material so an investigator can drill down from a pattern to the raw trajectory.

2. Separate Symptom, Behavior, and Mechanism

“Timeout” is the symptom. “The agent started a second server without checking the first” is the behavior. “The terminal tool does not preserve background-process state” is the mechanism you can edit.

3. Version the Harness

Keep prompts, skills, tool descriptions, middleware, and memory schemas in reviewable files. Every run should identify the version that produced it. Opaque dashboard settings and copied prompt strings destroy reproducibility.

4. Limit Each Experiment

Prefer one change per hypothesis. Freeze the model, evaluator, budget, and permissions. If several components must change together, declare the package and schedule an ablation later.

5. Write the Prediction First

evidence: background jobs disappear from the observable state
cause: the terminal returns output but no persistent process handle
change: add a job registry and status command
prediction: fewer duplicate launches and port collisions
risk: extra polling may increase latency and token use

A prediction written after the score is a story. Written before the run, it is a test.

6. Protect Separate Evidence

Use a held-in set to check the targeted fix and a regression split hidden from the proposer to protect healthy behavior. If the regression split is consulted repeatedly for promotion, do not later call it an independent final test. Preserve a third, untouched set for that purpose.

7. Put Humans at High-Impact Gates

No human needs to approve every file search. Human judgment belongs where automated metrics are a poor proxy for value: changing permissions, spending significant money, publishing, contacting people, touching production, accepting an irreversible migration, or redefining the evaluator.

That is what moving up the stack actually means. Less supervision of keystrokes; more responsibility for objectives, boundaries, and evidence.

The Intelligence Outside the Weights

For years, the story of AI progress had three main characters: parameters, data, and compute. Agents make that cast incomplete.

Once a model acts in an environment, the quality of the loop matters too. An ambiguous tool can hide capability the model already has. A precise evaluator can turn mediocre attempts into iterative search. Durable memory lets work survive the context window. A permission boundary makes autonomy possible without turning every mistake into an incident.

The harness is the part of the system’s intelligence that lives outside the weights.

It does not think for the model. It decides which predictions can become actions, which consequences return as evidence, and which experience survives into the next attempt. Because those decisions live in code, they can be inspected, versioned, transferred, tested, and — within a fixed boundary — improved by the agent they support.

That is the lasting contribution of Meta-Harness, Self-Harness, AHE, and the Darwin Gödel Machine. They turn a craft practice into an experimental loop: observe the failure, name a cause, change one bounded surface, measure against separate evidence, keep or revert, and preserve what the experiment taught.

The model supplies the capability. The harness turns failure into an experiment. The evaluator decides whether the experiment counts. The human decides what is never up for negotiation.

The first AI that truly improves itself may not rewrite its own brain. It may learn to build a better place for its mistakes to become evidence.

Sources and scope. The orientation source is Lilian Weng’s “Harness Engineering for Self-Improvement”. The main industry sources are OpenAI Harness Engineering, Unrolling the Codex agent loop, Anthropic long-running agent harnesses, and autoresearch.

The research literature includes ACE, MCE, Meta-Harness, Self-Harness, Agentic Harness Engineering, Darwin Gödel Machine, STOP, and AlphaEvolve. The numerical results come from different protocols and should not be read as a ranking. The repository’s research dossier preserves the complete ledger of sources, claims, and caveats.