The Diff Is Enough

The reflex, when you wire an agent into code review, is to give it everything. Index the repository. Embed the services. Let it wander the call graph until it understands the system.

I built the opposite: an agent that sees the diff and nothing else. No repository scan, no retrieval, no exploration. It runs on every pull request, on a cheap model, and it produces the same verdict the expensive setup produces — because the question being asked was never “do you understand this system.” It was “is this change safe to merge,” and the answer to that is sitting in the diff.

What this post covers:

• Why the average diff — not the model, not the framework — is the measurement that decides your architecture
• What a hundred lines can honestly tell you, and what it cannot
• Keeping prompts in their own repository, selected per technology at build time
• The output contract: a 1–5 score, an A–E grade, a recommendation — and why structure beats prose
• What a review costs, and why the unit is the push, not the pull request

Measure your diffs before you design anything

Before writing a line of pipeline code, I measured our pull requests.

The average diff is about a hundred lines.

That number resolved most of the architecture, and almost nobody takes the measurement before choosing a stack. A hundred lines is self-contained. Reading it does not require a call graph or fifteen files in context. It requires the diff and a set of rules — which means the review task is small, and small tasks have completely different economics from the ones the tooling discourse assumes.

So the agent sees the diff, the rules for that technology, and the PR metadata. That’s the entire input. Around five thousand tokens, every time, whether the service is six months old or six years old.

The honest limit of that decision looks like this:

The left column is a large class of real defects, and it is exactly the class a reviewer between two meetings skims past. The right column is why humans still approve the merge. Two different jobs — the agent takes the one that rewards consistency over insight, and says so out loud in its own comment.

Where it runs

Inside the pipeline that already fires on pull request. No bot service, no webhook receiver, no infrastructure to justify to anyone.

The part worth copying: the review prompts live in their own repository. The pipeline clones it, resolves the service’s stack, and loads the matching prompt.

Standards change more often than the services they govern, so a rule update is one pull request in one place rather than fifty. Prompts get diffs, approvers and history, like the code they judge. And once selection happens at build time, “which rules apply here” becomes a routing decision you can make as precise as you want — technology today, kind of service next.

The verdict

One comment, three fields: a score from 1 to 5 for how much we trust the change, a grade from A to E for the quality of the code, and a recommendation derived from both.

The two axes are not redundant. Collapsing them into one number destroys the information a reviewer actually wants, and the mapping between them is a policy decision you own, tune, and can defend to a developer who disagrees.

## AI Review

**Score:** 4/5
**Code quality:** B
**Recommendation:** Approve with comments

**Findings**
- [major] New endpoint does not validate `amount` before persisting.
  Negative values reach the DB. → src/handlers/charge.ts:42
- [minor] Catch block logs the message but drops the stack trace.
- [nit]   Magic number 30000 for the timeout; move to config.

**Not covered by this review**
Cross-service contract impact — this review only sees the diff.

Three properties matter more than the review text. A grade is hard to ignore in a way that prose is not. A grade is data, so scores accumulate into a distribution you can watch per team and per service. And the recommendation stays a recommendation — a model that can block a release will eventually block one it shouldn’t, at the worst hour, and that single incident costs more than the agent earned all quarter.

The model is the last decision, not the first

Once the input is five thousand tokens and the output is a short structured verdict, model capability stops being the bottleneck. There is no long-horizon reasoning to do and no plan to form. We run Claude Haiku 4.5 on top of Claude Code, and it clears the bar comfortably.

Sonnet is better, in the way you would expect — sharper on the judgment calls at the margins. The gap is real; it is just narrow at this task size. If the average pull request were a thousand lines across fifteen files, I would be writing a different paragraph. Model choice is a function of your workload, which you cannot know until you measure it.

The genuinely instructive result was GPT-5.6 Luna. The analysis quality was competitive. What disqualified it was that it would sometimes not respond in the expected format.

That sounds like a footnote. It isn’t. In a chat window a malformed response costs one retry. In a pipeline it is a broken step, a missing comment, and a developer who has now seen the gate fail twice and has stopped reading its output — including the times it was right. For unattended workflows, format reliability outranks reasoning quality. You are not shopping for the smartest model; you are shopping for the most boringly obedient one that clears your bar. No benchmark measures this. Your pipeline measures it on day one.

What it costs

One review of one average diff: ~5,000 tokens in, ~800 out. At published September 2026 rates:

Nothing here is expensive, which is the point: at these absolute numbers a two-cent difference should never outweigh a broken build step. The premium tiers are not unaffordable — they are capability this task never exercises.

The number that actually moves the bill is not in the table. The unit is the push, not the pull request. Our agent re-reviews the whole diff on every push, so a PR that takes four rounds of feedback costs four reviews — it charges you most precisely when it is helping most. Multiply the column by your average pushes before you budget anything, and build the incremental path earlier than I did.

What I would fix

Review incrementally. Carry the previous verdict forward and re-score only what moved. This is the most expensive defect in our implementation and the most obvious in hindsight.

Route rules by kind of service, not only by technology. For a long time every service got every rule, so Java services were graded against Node rules. Noise teaches developers to skim, and a skimmed gate is a decoration.

Design the comment format before the prompt. Most of my early effort went into review instructions. Most of the value came from forcing a structured, comparable verdict.

The reframe

Every decision in this system — the cheap model, the flat cost, the fact that it runs on every pull request instead of the important ones — follows from one measurement taken before any of them.

A hundred lines. Self-contained. Already in the payload your CI has in hand.

The context you didn’t send is the context you don’t pay for, don’t wait for, and don’t have to keep in sync. Most review questions never needed it.