What AI code review catches, and what it doesn't

When agents write code, it's natural to have other agents review it. ARC does: every piece of work an agent builds is reviewed before it counts. Running that review on all of our own builds has shown us where it earns its cost and where it doesn't.

More judges isn't more judgement

The research is clear that adding AI judges doesn't add independent opinions at the same rate. A 2026 study found that a panel of nine models gave only about 2.2 effectively independent votes, with a ceiling near 2.6 regardless of panel size, because models trained on similar data make similar mistakes (Nine Judges, Two Effective Votes). The best single judge often matched the whole panel.

The one lever that cuts that correlation is asking each judge a different question. A reviewer focused on security and one focused on correctness disagree in useful ways. Two reviewers asked the same thing mostly agree with each other.

What AI judges get wrong

• They agree with experts about as often as experts agree with each other, which is good but far from certain (Zheng et al., NeurIPS 2023).
• They favour their own kind of output. The same study measured models rating their own answers higher, which matters when the same model family writes and reviews.
• They cave under pushback. Models changed their answer to match the user in 58% of tested cases, and in 14.7% switched from a correct answer to a wrong one (SycEval, 2025).
• Critique without evidence can make things worse. Asked to review their own reasoning with nothing external to check against, models often degrade a correct answer (Huang et al., 2023). Critique grounded in actual test results works much better (Reflexion).

What we saw ourselves

On one large build in July 2026, two bugs in our own gate cost us about five extra iterations.

False failures. One reviewer returned "fail" verdicts whose written findings contained no defects, only confirmations that things were correct. Each one sent passing work back for another round. The last piece of the build only just passed before the retry limit. One more spurious verdict would have stopped the build for nothing.

Lost findings. When work did need fixing, the next attempt got the verdict but not the explanation. The agent retried knowing that something was wrong but not what. It usually found the real problem anyway, but at the cost of a full extra round.

Neither problem was the model's judgement. Both were plumbing around it. Both were invisible until we read the verdicts themselves rather than counting passes and fails.

Where AI review earns its cost

• Consistency with written standards. Reviewers are good at noticing that a change ignores a documented convention, an error format or a logging rule.
• Security smells. Hard-coded secrets, missing authorisation checks and unsafe input handling are well suited to a reviewer that looks for nothing else.
• Scope drift. A reviewer reading the request and the change together notices work nobody asked for.

What it can't do is prove the code works. That job belongs to the build and the tests, which check the code against something outside any model. For us, a failing test always outranks a reviewer's approval.

Questions to ask about any AI review step

• Does each reviewer have a distinct job, or is it the same check repeated?
• What happens when a reviewer rejects work without naming a defect?
• Does the agent see the full finding on retry, or just a verdict?
• What outranks a reviewer: tests, or nothing?

Sources

All accessed 2026-10-30.

• "Nine Judges, Two Effective Votes", 2026. https://arxiv.org/html/2605.29800
• Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", NeurIPS 2023. https://arxiv.org/abs/2306.05685
• "SycEval: Evaluating LLM Sycophancy", 2025. https://arxiv.org/html/2502.08177v4
• Huang et al., "Large Language Models Cannot Self-Correct Reasoning Yet", 2023. https://arxiv.org/abs/2310.01798
• Shinn et al., "Reflexion: Language Agents with Verbal Reinforcement Learning", 2023. https://arxiv.org/abs/2303.11366