Harness

Before the parts, it helps to know why they are worth the trouble at all. Hold the model fixed and change only the scaffold around it, and the score moves hard: on SWE-bench Verified Mini the Holistic Agent Leaderboard has Claude Sonnet 4.5 at 68% with one scaffold and 34% with another, so the harness alone doubles the result, and LangChain got +13.7 points on Terminal-Bench 2.0 tweaking only the harness. While the industry refreshes leaderboards and debates which frontier model to standardize on, or whether SWE-Bench was fine with slopThe statistical middle (slop)Output that converges on the bland average of everything the model has ever read: plausible, smooth, and indistinguishable from anyone else’s.The onboarding text that reads like every onboarding text ever written. all along, the variable that dominates is the loop infrastructure around the model. Chollet supplies the theory on Why, that building is a search process and so the setup around the search matters, and clawbench, the agent benchmark from the OpenClaw ecosystem, takes the design stance to its conclusion: it scores the whole combination you ship, harness and configuration and model together. The lever sits here, in the harness, not in the model. Which makes it worth knowing exactly what goes into one.

I work with four clusters, and each one owns a different question about the same piece of agent output:

• Engineering + Architecture asks: is it correct?
• Product + Domain asks: is it right?
• Security + Compliance asks: is it allowed?
• Operations asks: does it hold up in production?

Correct and right are not the same question. A change can pass every test, satisfy every type checker, and still solve a problem no user has. Say it is right, too. It can still touch data it had no business touching, or clear all three and quietly fall over under real load three weeks later. Four clusters, then. Four ways for plausible output to be wrong, and the work has to survive all four before it counts.

One layer compiles, the other one doesn’t

The technical layer is the one you have probably seen versions of already, because it is where the public harness debate lives. Its building blocks:

• repo rules, AGENTS.md, skills and documentation (actually, this is context the agent uses as backpressure to hold on itself)
• language servers, compilers, linters
• tests in all their forms
• CI, trial runs in ephemeral containers, security checks
• PR diffs and reviewer subagents
• logs, traces, and UI checks
• hooks that fire on what the agent does, blocking a bad commit or write before it lands
• sandbox boundaries, approval gates, rollback paths

Each of these is something the agent’s work has to get past without a human watching every turn.

The product and domain layer is built from different stuff, and almost none of it compiles:

1. the actual user problem and the job to be done
2. explicit non-goals (the solution spaces you deliberately ruled out)
3. domain rules and edge cases
4. acceptance scenarios and counterexamplesCounterexampleA plausible-looking but wrong output you keep on file, so the system learns never to produce that kind of thing again.The bug you once shipped and then wrote a regression test for.
5. quality standards from the people who own product, UX, risk, and operations
6. variant comparisons with explicit trade-offs
7. and production signals like rework, support feedback, and usage patterns

Notice how much of this list is knowledge that lives in someone’s head until somebody does the work of writing it down in a form an agent can be held against.

Structural vs behavioral harness

Backpressure comes in two strengths, and what separates them is one question: can the agent get around it?

Some backpressure is a rule you write down and hope the worker follows. You put it in an AGENTS.md (a plain instructions file the agent reads when a run starts), or you hand it to a second agent you spin up to review the first one’s work. The agent reads these, and it can ignore them, or even rewrite them. One did exactly that, editing its own AGENTS.md to drop a rule it found inconvenient. This is the behavioral layer, and it’s advisory. It holds right up until the run where the agent decides it doesn’t.

The other kind is a wall the agent runs into and can’t move. A change blocked from being saved until it passes a check, a sandbox (a sealed-off copy of the system where a bad run can’t touch anything real), a permission gate a human has to open. The agent can’t author these or switch them off. The structural layer still holds when the agent is wrong and sure of itself at the same time.

The split runs through product rules too. A non-goal you wrote down, say “never email a customer twice in a day”, sitting in an instructions file is behavioral: the agent might honor it, might not. The same rule wired up as an automated check that fails the moment a second email goes out is structural. Same intent, two very different guarantees.

The field has been circling this same line. Idan Habler separates soft guardrails from hard boundaries. Reuben Brooks, building on Geoffrey Huntley’s backpressure work, calls it behavioral gates versus structural gates and argues the structural ones beat a smarter agent. The Evil Martians put it operationally: stop writing rules in AGENTS.md, move the ones that matter into checks that actually fire.

AGENTS.md is still harness. Addy Osmani and others count instruction files and review agents as parts of it, and they’re right to. The enforcement runs inside the harness itself: AGENTS.md is its soft and movable layer, and the question up top tells you which layer any rule sits in. If the agent can route around it, you’ve written a suggestion, however firmly you phrased it.

This is also where judge independence comes from, the rule from Grading that whatever grades the work can’t be the same thing that made it. A review agent the same run spun up shares its blind spots, and it can be talked out of an objection. Independence comes from the layer the agent can neither write nor bypass: a reviewer kept outside the run, a gate it doesn’t get a vote on. So before you trust a rule you’ve written down: what would actually stop the agent if it just ignored it?

Your test suite will not catch slop

Here is the job each layer actually does, which is what makes the two more than a tidy taxonomy. The technical layer mostly fights sprawl, the explosion: compilers, tests, CI, architecture fitness graders filter out what does not fit the system, and they push back against “too much, too wild”. Slop is the other layer’s problem. Scenarios, counterexamples, and domain rubricsRubricA written list of what “good” means for a kind of output, so the same standard can be applied to every result.Acceptance criteria, reused as a grading checklist instead of a one-off. (written grading criteria) pull the output away from the statistical middle and into the specific solution space of this product, this domain, this user. That is the back half of the move: context decides where in the possibility space the agent starts, and this layer pulls whatever comes out the rest of the way toward your product.

Technical backpressure keeps the work correct against the system. Whether the product ever needed it is the other layer’s call.

You need both, and they are not interchangeable. A perfect test suite will wave through onboarding copy that sounds like every onboarding copy ever written, because nothing in it knows what your product is for.

What the next model takes away

Both layers grow while you pay out leash, and neither grows forever. They also age differently, which is the part worth planning for. Anthropic’s own harness write-up argues for shrinking the scaffold as the model improves: every piece you bolt on encodes an assumption about what the model cannot do yet, and those assumptions expire. Between two model generations they tore out their task decomposition and folded a per-stage evaluator back into a single pass at the end, because the newer model held coherence for hours on its own.

The line that decides what you can remove cuts across both layers, not between them. Some of the scaffold only compensates for a capability gap: planning horizon, coherence over a long run, context-window juggling. A better model closes those gaps, so the crutches built for them are the first thing to strip out. The rest is backpressure that encodes a target the model had no way to know. That part stays, however capable the model gets.

The trap is reading the whole technical layer as the disposable side. Judgment lives inside it too. Whether a change fits your architecture, respects your bounded contexts, cuts the seam where you wanted it: a fitness function catches the mechanical part of that question, no cycles in the dependency graph, and stops there. The rest is a call graded by a rubric, an LLM judgeLLM judgeA second AI you ask to grade the first one’s output against your criteria. It scales review, but has blind spots of its own.A tireless reviewer who needs a clear checklist, or it will rubber-stamp anything that looks plausible., or a human, and a stronger model does not retire it, because it was never a capability gap to begin with. Architecture conformance lives in the technical layer and ages like a domain rule, not like a crutch. So “simplify as the model improves” holds with one correction: strip the scaffolding, keep the judgment, wherever in the harness it happens to sit.

Backpressure only counts if it sticks

There is a condition both layers share: backpressure is only effective once the signal changes a later decision. A failing test that gets retried into silence is noise. So is a rejected variant nobody learns anything from, however much compute it burned to produce. The signal has to land somewhere it changes the next run: a changed plan, a sharper rubric, a new counterexample. And a wall the agent hit once should leave a guard behind, a new test or a hook that fires before the bad write lands. Retained feedbackRetained feedbackFeedback only counts when it changes a later decision: an updated plan, a sharper test, a rejected option that stays rejected. The rest is noise, however expensive it was to produce.A review comment matters only if it changes the next commit. is what connects the two layers, because retention works the same way whether the signal came from a compiler or from a domain expert.

Agent struggle becomes harness backlog.

Example: Agent memory and skills

Agent memory is the most tangible retention site I know, and a skill is one form of it, procedural memory in executable form. A skill that does not get better after repeated use, or a memory that never gets corrected, is just static documentation wearing a different hat. The raw material for improving them sits right in the agent traces: retry loops and dead ends, wrong or outdated commands, missing setup assumptions, the places where the agent made a best-guess decision because the skill left a gap. Treat those signals as deltas, patch immediately when risk and evidence allow it, route the rest through a review gate, and maintenance becomes a closure mechanism inside the loop. You could even optimize the token efficiency of your skill by telling the agent to use the initial version as a baseline, and run experiments on it until token consumption is improved while the result stays the same (don’t let it rephrase the skill in caveman lingo).

The quiet layer

Why does the product and domain layer get so little attention? Because its failure mode is quiet. Sprawl shows up in CI logs and PR volume; you can see it and pull people back. Slop surfaces in the market, quarters later, or in some people’s stomachs, the longer they look at it. The engineering mainstream of the harness debate (OpenAI’s harness engineering write-up, Addy Osmani’s work on agent harnesses) covers the technical layer well. But the other layer needs different disciplines and different roles: product people, designers, domain experts, the people whose “this is off” never touches a compiler. Their judgment is the backpressure itself, the thing the output has to clear while it is being made, well before any review step at the end.

The wins and the costs land on different desks

There is a version of this that has nothing to do with tooling. Charity Majors wrote in June about the gap between the people experiencing AI wins and the people carrying the costs: “There is no natural feedback loop.” The wins land on one team’s dashboard while the cleanup, the reliability erosion, and the on-call pain land on someone else entirely, and the organization polarizes instead of learning. Seen through this framework’s lens, the skeptics in that fight are often the people carrying the backpressure the system failed to encode. They hold the realistic information about failure modes precisely because the consequences flow to them.

Her question for the enthusiasts is the one I would put on the wall:

“What would it take for you to feel comfortable shipping code to production without reading it?”

That single question turns a culture war into a loop-sizing requirement. Answer it honestly and you get a concrete list: the evals, the feature flags, the blast-radius limits, the rollback paths. Which is to say, you get a harness spec.

How many agents can one person actually check?

Context and backpressure both work on the content of the loop: context is the gate in front of it, deciding whether the agent can do the work at all, and backpressure is the continuous pressure inside it, holding that work against reality. Neither says anything about how expensive it is for the human to close the loop at the human gate. Even the best-calibrated loose loop ends in a human decision point, and when closing it means wading through diffs, logs, and pull requests, the ceiling on parallel loops is set by the verification cost per judgment rather than by containment. That cost is the third lever.

Three families of mechanisms, and they sit on different levers. Self-inspection by the agent (opening the build in a browser, reading the DOM snapshot or screenshot, running Lighthouse, axe, Core Web Vitals) is backpressure in the engineering cluster, already covered above, no new lever required. The numeric signals are the clean ones here: a performance budget that speaks when broken and stays silent when held is sounder than an LLM judge grading aesthetics. Self-demonstration sits on both levers at once. Simon Willison’s Showboat is a CLI through which the agent presents its own work; Peter Steinberger has agents record a video demo of what they built. The demo makes the gate cheaper, since the human grades an outcome instead of digging through the artifact, and the act of building the demo is forced self-confrontation: while assembling it, the agent stumbles over its own defects, feedback that would never have existed without the obligation to present. Legibility artifacts (screenshots, recordings, trace summaries) are pure gate relief and carry no backpressure at all.

The trap deserves its own paragraph. A polished demo can hide slop, and verification stress is partly productive: it is the friction where bad judgment gets caught. Self-demonstration makes the human gate cheaper to clear without making the judgment behind it any sharper. Demo mechanisms lower the cost of good judgment, while the judgment itself still has to come from outcome gradingOutcome gradingJudging whether a result is good, precisely enough that the judgment can be applied again and again, by a person or a machine.You already do the simplest version every time you write a test or a definition of done. against goldensGoldenAn output you have blessed as correct and keep around as the answer key, to compare new output against.The trusted fixture in an integration test: the known-good result everything else is measured against. and counterexamples.

Does any of this exist outside slide decks?

OpenClaw is the closest thing we have to a running reference in the wild: a 375k-star project with roughly 7,000 open issues and PRs, operated deep in the loose zone. There is a fascinating ring of metatools the team built around the agent. Almost none of it is the harness itself. It is what has to surround a harness before you can let an agent run loose for hours and trust what comes back, and it does two jobs at once: supply backpressure, and give the agent enough infrastructure to reach its own verdict.

• Context tooling like the crawl family that externalizes scattered knowledge into searchable stores, so the agent can ground itself instead of guessing
• Execution machinery like crabbox, fanning tests across operating systems and clouds in reproducible containers, so the agent can answer what no prompt can answer for it: does this bug actually reproduce?
• Backpressure like ClawPatch (a reviewer subagent with a fix loop) and clawbench, which puts pressure on the harness itself
• Verification through evidence ledgers and human gates that carry the final judgment back to a person

The price of that autonomy was an explosion of infrastructure around a comparatively small harness.

My favorite vignette is Vincent Koc running 24 tmux panes of agent sessions from a Mac Mini, reading failures, deciding what runs next. Rather than leaving the loop, the human steps one level up, and his judgment moves to the orchestration layer. Those 24 panes are also the verification cost ceiling from the section above. Containment is not what caps him. The cost per human judgment at the gate is, and every mechanism that lowers that cost lets the same attention orchestrate more panes.

An old discipline on a new substrate

None of this is unprecedented, and it helps to know that. Long before anyone wired a compiler into an agent loop, factories had already worked out how to run a process that produces variable output: how to catch defects without inspecting every part by hand, and how to feed what each run teaches back into the line. The discipline has names, and most of them trace to one place, the Toyota Production System and the lean tradition that grew from it. Lay the two side by side and the mapping is close enough to be useful rather than cute.

The parallel is fascinating. We’re looking at the industrialization of software product development. The reassurance: these are not hypotheses. Manufacturing has run on statistical process control and continuous improvement for decades, so the shape of the answer is known even where the agentic details still are not. And it is an orthogonal lineage. Most of the harness debate argues from the software tradition of Agile, DevOps, and CI/CD, while this one comes in from the factory floor, which is exactly why it tends to land with people who have run a real process.

The factory and the loop part ways on one thing, though. Dave Snowden’s CynefinCynefinA Welsh word (roughly 'kuh-NEV-in') for sorting problems by how knowable the answer is before you start: clear (one obvious right way), complicated (a right answer an expert can find), complex (no answer you can work out in advance, so you try something and learn from what happens), and chaotic (no stable cause and effect at all). Different kinds of problem call for different approaches.Tying your shoes is clear, fixing an engine is complicated, taking a product into a market is complex. You already switch approaches for each without naming the buckets. framework (HBR, 2007) sorts problems by how knowable the answer is before you start:

• Clear: cause and effect are obvious and there is one right way, like following a recipe.
• Complicated: a right answer exists, but it takes an expert to find it, like a mechanic diagnosing an engine.
• Complex: you cannot work the answer out in advance. You try something, watch what happens, and adjust, the way you learn what a market wants.

Classic manufacturing lives almost entirely in the first two. The ten-thousandth car door has a known target, and the whole Toyota line exists to hit it every time, every deviation a defect.

Most of what we point a loop at is the third kind, because building a software product is complex work, and so is most knowledge work. You don’t know the answer up front. It shows itself once you have tried a few. So we are building the same learning factory and the same hard-won discipline of running a process without checking every part by hand, and aiming it at complex work. That changes what we want from variance. A Toyota line fights to drive variance out. A loop is here to harvest it, generating several real variants and keeping the one that holds. The machinery carries straight over, and here the variance is the product. Backpressure is the wall that keeps it inside useful bounds. Some tasks really are ordered, well specified and fully tested, and there the loop behaves like the classic line and drives variance out. Complex is the common case, though, and complex is where the factory’s instinct flips.

The warning is the one the lean tradition keeps relearning. None of these mechanisms is a tool you install once. They are practices a team keeps alive, or they quietly rot, and a rotting harness is worse than none, because it still looks like it is working. Which is the same thing the roles page says from the other direction: you build and evolve the factory alongside the products it turns out, or you do not really have one.