I didn't get it wrong. I got it right about the wrong copy.
- A review agent, reading last month's code with total confidence
This series started with a number. I asked a set of AI agents to triage old tickets, and forty came back marked obsolete. A second set of agents, told to prove each verdict wrong, overturned thirteen of them. That's the first post.
Since then I've leaned on refutation everywhere. Tell a fresh agent "prove this wrong." Default to not confirmed. Make every "nothing found" earn its zero. I wrote that up as a field guide, and then wrote about where my own orchestration fails.
Which leaves the obvious question. If refutation is the check, what checks the refutation?
What I Expected to Find
I expected refuters to be wrong the way the original agents were wrong. Reasoning that stops short. A conclusion bigger than its evidence.
So I went back through my incident notes and a client's merge log looking for that. It does show up, in one specific shape: a checker that argued about the code instead of going and looking at it.
But the failure I found most often was different: the refuter reasoned well about bad evidence.
The Stale Copy
Some background first. When you clone a repository, you get a copy of the code as it stood that day. The shared server copy keeps moving. Git calls that shared version origin/main, and your view of it is only as current as your last fetch. A local copy that hasn't been updated in a while is "stale." It still looks like the code. It just isn't the code anymore.
On a client's platform, I keep several of these copies side by side. On one day this September, they were 41, 308 and 17 commits behind.
That day, a blast-radius pass (an agent whose job is to find what a change might break) checked my work against one of the stale copies. It reported that a ticket I'd cited had zero hits anywhere in the repository, and that the guard I'd described had no such check at all. Its conclusion: I had invented the mechanism.
I hadn't. On origin/main, the ticket appeared in five files, and the check sat on line 89 of that guard. Both had landed in a commit the stale copy didn't have.
The refuter's logic was fine. If the code really had no such check, "you made this up" was the right call. It was reading the wrong code.
It wasn't a one-off. On the same platform, one local checkout ran 37 commits behind at one point and produced confident wrong review findings until I told reviewers to read from origin/main instead. A mapping agent read an old branch and reported that a review workflow didn't exist, when it had already merged on main.
The Probe That Pointed at Nothing
The second pattern is the one the field guide already warned about: a search that finds nothing looks exactly like a search pointed at nothing.
It applies to checkers too. A query I ran to rule out a data problem came back with zero mismatches. The zero was worthless. Security rules on the table hid every row from the account I queried with, so I had compared zero rows against zero rows. Once I could see the rows, the real answer was still zero. The first zero proved nothing.
Mutation checks have the same trap. You break the code on purpose and confirm a test fails. But one broken version failed under the test suite's database settings and passed under production's. The test looked like coverage. It only covered a setup production doesn't run.
When the Checkers Disagree
Sometimes two reviewers look at the same change and reach opposite answers. My logs have a pattern here too.
Once, a code reviewer said a data fix reached too far. The blast-radius agent disagreed, and it had listed every place in the code that wrote to that field. I checked independently, and the one that enumerated was right. Another time, two reviewers gave contradictory readings of the same setting. Both were partly right. A third pass found it changed three things. The note I left myself: read the function, don't trust any one summary of it.
The rule I took from both: when checkers conflict, trust the one that looked, not the one that argued. The agent that ran the query, listed the callers or read the function beats the one that only reasoned about it.
A Refutation Is a Claim
Here's the part I had to learn the hard way. When a refuter says "that's wrong," that's not a verdict. It's another claim, and it gets the same standard as the first one.
So before I weaken a comment or retract a finding because a refuter objected, I re-derive the answer from origin/main. Same rule I apply to any agent.
I'll be honest about the edge of this. The same pass also reported a second defect built on a field name the code doesn't use. My notes don't say whether that came from the stale copy or from a plain misread. So my claim is narrow. The refuter misses I can trace came from one of two places: a bad instrument, or reasoning in place of looking. I haven't found one where a checker looked at the current code and still got it wrong.
When to Trust, When to Push Back
Here's where the series lands for me.
- Point every checker at the current source of truth. Not a local copy, not a branch from last week. Put it in the prompt, because the agent will reach for whatever is on disk.
- Make every zero earn its keep. Before you believe a check found nothing, show that it finds something when something is there.
- Prefer the agent that looked. Queries, callers, a failing test, the function itself. Evidence someone gathered beats evidence someone inferred.
- Treat "that's wrong" as a claim. It needs the same proof as whatever it's refuting.
Trust agents that look. Push back on what they're reading, and on any checker that argues without looking. And push back on your own checkers too, because the check you built to catch mistakes will make them with the same confidence as everything else.