🪢 Harness Engineering for AI Agents

The Harness Gap

A frontier coding model running on a bare API endpoint solves roughly 20-25% of SWE-bench Verified. The same model inside SWE-agent or OpenHands solves 50-70%. The model weights are identical. What changes is the harness: the scaffolding that turns a chat completion into a reliable engineering teammate.

This gap has shifted the locus of progress in 2025-2026. Frontier labs are no longer just publishing models; they're publishing harnesses. OpenAI's Codex and Anthropic's Claude Code are essentially shipped harnesses around their respective models, and the value comes as much from the loop, the rules, and the verification as from the underlying weights. The folk wisdom ("the prompt is the program") has been replaced by a sharper one: the harness is the program.

What a Harness Actually Is

A harness is not a smarter model and not a better prompt. It is a closed-loop working system around the model with four pillars:

1. Environment design: what the agent sees and can touch. The repo, the sandbox, the tool surface, the OS. The agent acts inside the environment, not in conversation about it.
2. State management: context that persists across turns and sessions. AGENTS.md / CLAUDE.md, progress files, feature lists, scratchpads, the repo itself as the system of record.
3. Verification: the harness, not the model, decides whether work is done. Full-pipeline tests, type checks, lints, acceptance criteria, self-reflection prompts that compare actual state against declared goal.
4. Control & observability: explicit rules and boundaries (permission scoping, budgets, allowlisted tools) plus runtime visibility (trajectory logs, tool-call audit, replayability).

The diagram in your head should be: model in the middle, harness wrapping it, environment underneath. The harness reads from the environment, presents a curated view to the model, accepts a structured action, runs verification, updates state, and loops.

Repo as System of Record

The first principle of effective coding-agent harnesses is that the repository, not the chat transcript, is the source of truth. State that matters lives in committed files, not in conversation history.

Concretely:

• AGENTS.md / CLAUDE.md / .cursor/rules/: house rules, conventions, "always do X, never do Y". Versioned with the code so a different agent (or human) picks up the same constraints.
• feature_list.json (or similar): the spec for what's being built, broken into atomic units. The harness loads this at init; the agent updates it on completion.
• A progress file (e.g. progress.md, .claude-progress.md): what was attempted, what worked, what didn't, what's next. Bridges sessions when context windows roll over.
• Tests, fixtures, golden outputs: the verifiable contract for "done".

If a teammate (human or AI) can clone the repo and pick up where you stopped without reading the chat history, your harness is working. If they can't, the chat is doing too much load-bearing work.

State Across Sessions

Long-running tasks blow past any single context window. The naive answer, "stuff everything into the system prompt", fails for two reasons. First, prompt drift: one giant instruction file accumulates contradictions and special cases until the model can't reliably follow it. Second, signal-to-noise: relevant rules get buried under stale ones.

Effective harnesses use hierarchical state:

• Always-on rules (concise, repo-wide) → loaded into every session
• Task-scoped state (what we're currently doing) → loaded when work resumes
• Episodic memory (past attempts, observations) → retrieved on demand

The session-resume flow looks like: read AGENTS.md → read progress file → read the relevant feature/task entry → understand the current state of the working tree → decide the next step. None of that requires conversation history.

Initialization as a Distinct Phase

A common harness failure is treating session start as "just begin the loop". Effective harnesses make initialization a first-class phase that runs before the agent takes any action:

1. Load rules and conventions
2. Read the progress file and last-known task state
3. Check repo invariants (working tree clean? branch correct? dependencies installed?)
4. Run a fast smoke test (does the project build?) to detect drift since the last session
5. Decide a concrete next action, not "let me explore"

Skipping initialization is how agents end up re-editing files that already had the change, re-running build steps that already passed, or starting on the wrong branch.

Verification & Self-Reflection

The model cannot be trusted to grade its own work, both because of self-evaluation bias and because "done" is often a property of the system, not the patch. The harness owns verification.

Effective verification stacks layer:

• Static checks: type errors, lint, format. Cheap, fast, run on every patch.
• Unit tests: verify the specific change works.
• Full-pipeline / integration tests: verify nothing else broke. This is the gate; partial test runs let the agent ship regressions.
• Self-reflection: a prompt that asks the model to re-check the original acceptance criteria against the actual repo state. Not as ground truth, but as a final sanity pass that catches "I implemented X but forgot to wire it up".

The verification result is structured data the harness consumes (pass/fail per criterion), not free-text the model has to interpret.

Premature Victory

Models trained to be helpful are also trained to declare success. Left alone, a coding agent will reliably claim "Done!" on partial work: the function was written but never imported, the test was added but never run, the migration was created but never applied.

Harness-level mitigations:

• Acceptance criteria as data, not prose. If the harness can't programmatically check "criterion C passed", it cannot judge done.
• Blocking verification gates. The harness, not the model, emits "complete" only after the gate passes. The model's "I'm done" is a request to verify, not a verdict.
• Atomic feature units. Big tasks get decomposed into items the harness can independently verify, so partial completion is visible at item granularity.
• No silent skip. If a test was supposed to be added and isn't, the harness flags it; it does not accept "I decided that test wasn't needed" without an explicit acknowledgement.

Observability Inside the Loop

Production agents fail in long-tail ways: loops, ping-ponging tool calls, hallucinated file paths, ignored error signals. Debugging these requires the harness to be observable.

The minimum bar:

• Trajectory store: every tool call (name, arguments, result, timestamp, latency) persisted for replay. LangSmith, Langfuse, W&B Weave, or a homegrown JSONL log all work.
• Decision provenance: record why the agent picked a tool, not just what it picked. The model's reasoning text is part of the trace.
• Replayability: the trajectory plus the starting environment state should be enough to re-run a session deterministically (modulo model nondeterminism).
• SLOs over the trajectory: track loop rate, average steps to completion, tool-call entropy. Spikes are leading indicators of harness regressions.

The principle: when something goes wrong, your first move should be opening a trace, not adding print statements.

Clean State Discipline

Every session must leave the environment in a known-good state. If a session ends with a half-applied migration, a dirty working tree, a stale dependency lock, or a stopped background process, the next session inherits the mess, and either silently builds on broken state or wastes its initialization budget cleaning up.

Harness primitives for clean state:

• Session lifecycle: explicit init → run → cleanup phases, with cleanup running on both success and failure paths.
• Atomic commits: every meaningful change either commits or rolls back; no half-states left in the working tree.
• Sandboxed execution: the agent's experimental side effects (installed packages, scratch files, started services) live in an ephemeral environment that's torn down at session end.
• Idempotent operations: reruns of the same step are safe; the harness exposes "this has already been done" rather than re-applying.

Other Harness Layers (Brief)

The coding-agent framing above is where the discipline has crystallized, but the term applies more broadly:

• Eval harnesses wrap a model with a benchmark runner, a judge, a result store, and a scoreboard. The canonical example is lm-evaluation-harness (EleutherAI), which powers the HF Open LLM Leaderboard. Inspect AI (UK AISI) is the modern agent-aware successor. See Evaluation & Benchmarking for the surrounding mechanics.
• Training harnesses wrap a model with a data pipeline, optimizer, checkpointing, and logging. Examples: Axolotl, TRL, LLaMA-Factory, OpenRLHF, veRL. See Fine-tuning and RLHF / DPO.
• Serving harnesses wrap a model with a scheduler, KV-cache manager, tokenizer, and API. Examples: vLLM, SGLang, TensorRT-LLM. See Model Serving.

The common pattern: in every layer, the harness is what turns a model artifact into a usable system. The four-pillar framing (environment, state, verification, observability) maps onto each.