Grading
When an agent hands you a result, what would convince you it’s good? Outcome gradingAssessing a result against explicit criteria that a person or machine can apply consistently.Checking a completed feature against its acceptance criteria is one familiar form. makes that assessment explicit enough to guide revisions. Some requirements have an exact answer a test can check. Others need a reasoned judgment, and some product questions need evidence from use. This chapter explains how to combine those forms of evaluation and assess the reviewers themselves.
You have been grading outcomes all along
Choose the method for the question you need to answer:
- Tests compare behavior with defined expectations. They can cover technical properties and precise domain rules.
- RubricA guide that names the qualities to assess and describes what meeting each criterion looks like.For a dashboard, it might require the reason for each case’s priority to be visible and supported by the case details. describe quality criteria and the evidence needed to assess them. They help reviewers explain their judgments.
- Scenarios describe situations the product should handle, including difficult cases supplied by people who know the domain.
- GoldenAn accepted result retained as a reference for assessing later results.An approved support reply shows the tone and level of detail expected in later replies. and CounterexampleA plausible but unsuitable result retained with the reason it fails the relevant criteria. It helps assess later results and the reviewer itself.A polished dashboard that hides a case the agreed rule marks as urgent shows what the reviewer should reject. provide trusted examples and plausible failures. Keep the reasons for those assessments alongside the examples.
- Review can be performed by people, agents, or both. Decide which judgments an agent may make, evaluate its performance on relevant examples, and define when a person must decide. Product goals and acceptance rules need accountable owners.
- Production signals show what happens after acceptance: whether people can use the result, what needs repair, and which assumptions fail.
Product people, designers, engineers, and domain experts contribute to these methods. Roles describes their work across the loop.
The spec changes jobs as the loop stretches
A specification records the intent, constraints, and known requirements. Keep unresolved questions visible too. In the dashboard example, the team may know exactly how access must be restricted while still exploring how to prioritize support cases.
| Loop zone | What the agent needs | Where decisions happen |
|---|---|---|
| Tight | Intent, examples, and enough context to try the next step | People inspect attempts and refine the requirement together. |
| Elastic | A bounded task, constraints, and agreed checkpoints | The agent uses checks while working and brings open questions to the checkpoints. |
| Loose | Discoverable context, dependable checks, and clear acceptance and escalation rules | The agent carries the task to its review gate; unresolved decisions return to the authorized person. |
Keep the specification current as the team learns. A new requirement can change the implementation and the criteria used to assess it. The Loops chapter distinguishes the work delegated between human decisions from the time it takes to learn whether a product change helps.
Correct the current result when it needs repair, and investigate whether the failure exposes a recurring gap in context, checks, or review. Preserve a useful lesson in the appropriate material and evaluate the change on later comparable tasks. Factory explains how the team makes that improvement part of its regular work.
Evaluate the grading criteria
A written rubric can omit a requirement, reward a superficial feature, or encode a mistaken assumption. A Judge/EvaluatorAn AI used to assess work against defined criteria. Check its judgments against examples evaluated by people who understand the task.For a dashboard, it can assess whether the explanation of each case’s priority is supported by the case details., a model asked to assess work against criteria, can make these errors too. Compare its judgments with independently assessed cases and investigate disagreements. The examples below show how.
Self-checks help the agent correct its work. For an independent review, give the reviewer access to the evidence and protect the acceptance criteria from changes by the agent producing the work. A separate session can reduce dependence on the producer’s account. Evaluate the reviewer’s judgments on cases people have already assessed.
Protect assessment cases too. Keep some HoldoutCases set aside while developing a solution or adjusting an evaluator, then used to assess it on unfamiliar cases.Keep some support cases separate while refining the dashboard, then check how it handles them. cases separate from the examples used to develop a solution, so you can examine how it handles unfamiliar cases. Watch for Reward hackingWhen an agent achieves a high score by exploiting the scoring rules while failing the intended task.Code might return a hardcoded answer that passes one test but fails when the input changes., where a result satisfies the scoring rule while missing the intended requirement.
Give each check a useful job
Use deterministic checks for criteria they can settle reliably. A code-style checker can report a rule violation; an architecture test can flag a forbidden dependency. Fast, inexpensive checks give the implementing agent BackpressureQuick, inexpensive feedback from checks built into the agent’s workflow. As the agent works, these checks report problems it can correct immediately.When the agent tries to save a code change as a commit, an automatic check rejects a variable name that breaks the project’s naming rules. The agent corrects the name. Its next commit attempt triggers the check again. during its work. Required checks can also be conditions for passing a gate.
Give the reviewer those results along with the intended behavior and the change itself. A reviewing agent should investigate questions the existing checks leave open. It may find a defect in a single line or question an architectural decision. Run these reviews at a gate, with time and budget appropriate to their cost.
Grade two dashboard variants
Suppose a support team wants to keep its reply commitments and help customers who cannot use the product. The team is exploring how its dashboard should prioritize cases. It hasn’t settled how to balance those goals when they conflict. The agent builds two variants:
| Variant | Proposed priority rule | Consequence in the example |
|---|---|---|
| A | Handle the case closest to its promised reply time first. | A routine question due in 20 minutes comes before a newly reported outage affecting several customers. |
| B | Handle the case with the greatest reported disruption first. | The outage comes first; the routine question may miss its promised reply time. |
Both variants pass the code-style and architecture checks. Tests confirm that each implements its proposed rule correctly, using the same case data. Choosing between them requires a judgment about the consequences of those rules. The team has yet to agree an expected answer for this conflict. Writing “the outage must come first” into a test would encode a product decision that still needs to be made.
A reviewer, human or agent, can examine the proposals against the team’s goals and the available evidence:
“A keeps reply deadlines visible, but several reports of one outage remain separate cases and could keep waiting behind routine questions. B brings that disruption forward, but a stream of new incidents could leave older requests waiting indefinitely. Try grouping related reports and making overdue commitments visible, then review how support staff choose their next case.”
That review draws out consequences, questions the design, and proposes something the team can investigate. Its reasoning needs scrutiny too: reports that look related may have different causes, and reported disruption may be uncertain. An agent can make a reasoned recommendation within its authority; unresolved tradeoffs go to the people responsible for the product.
Try bounded variants with support staff and observe their decisions. Their experience may change the requirement. Once the team settles a rule, deterministic tests can enforce it and provide quick backpressure during implementation. Evaluating whether that rule continues to serve the product’s goals remains part of review and product learning.
Check the reviewer too
Evaluate a reviewing agent on separate scenarios assessed by people who understand the support work. Examine whether its findings follow from the evidence, whether it identifies material consequences, and whether it recognizes decisions that need more information. Include cases where disruption is uncertain or a reply commitment has consequences the initial report doesn’t mention. A reviewer that always favors outages would miss the judgment this task requires. Allow different recommendations when their reasoning is sound and they respect the agreed constraints.
Use some examples to refine the criteria, then assess the reviewer on separate cases. Where a clear acceptance decision exists, count incorrect approvals and incorrect rejections separately. For unresolved product choices, assess the reasoning and evidence without treating one preferred design as the only valid answer. One good judgment on these two variants gives little evidence about unfamiliar cases. Recheck the reviewer when its model, instructions, or evaluation criteria change.
Keep the observations tied to the task and the versions used. Record which criterion failed, whether the review found the problem, and what correction was required. Later evaluations can then examine whether better context or a new mechanism reduces the same failures on comparable tasks. Harness explains those mechanisms; Factory explains how the team organizes and funds their improvement.
Use variants to make a decision
When the product question is still open, generating several variants can make the alternatives easier to assess. Best-of-nGenerate several independent attempts at a task, then select the best according to agreed criteria.Trying 3 solutions and comparing their quality and cost before choosing one. means generating multiple attempts and selecting among them. Use it when the comparison is worth the implementation and review cost.
Build variants close enough to the intended system that the team can learn from them. For the dashboard, use representative cases and the actual components, then inspect the consequences of each priority rule with support staff. Preserve why an alternative was rejected and what might justify revisiting it.
If an agent handed you 5 solutions tomorrow, could you explain which you would choose, what evidence supports that choice, and what still needs to be tried?