Our own tests said the rebuild was done. Every dialog was unclickable.

In July 2026 we finished an autonomous rebuild of a line-of-business web application. The legacy system ran on Oracle stored procedures; the new one is a .NET API with an Angular front end. Agents wrote it story by story, and each story passed a review gate before it merged.

When the build finished, every number we tracked was green.

Then the end-to-end browser tests ran. Every modal dialog in the application sat underneath its own dark backdrop and couldn't be clicked: two versions of a UI library disagreed about stacking order. Separately, every modal and one main editing tab rendered at zero opacity, because a CSS class had been renamed between those versions. No unit test checks whether a person can see or click anything.

The client and system aren't named here. The figures come from our build logs.

Two ways the metrics misled us

The coverage number was inflated by the tool measuring it. Before we reached 88.95%, the coverage tool reported a higher figure. A rule meant to skip compiler-generated code also removed the bodies of asynchronous methods from the count, and those methods were most of the service layer. The tool was measuring the easy parts and ignoring the hard ones. Fixing its configuration brought the number down to the honest one.

Tests written beside the code shared its blind spots. Unit and contract tests confirm that each function does what its author intended. When the same agent writes the function and its test, they share assumptions. The tests that found the real defects were black-box tests: they drove the finished application from outside, the way a user does, with no knowledge of how it was built.

Why this matters beyond one project

Public coding benchmarks fail in the same ways. Their grading tests are sometimes wrong, weak tests let incorrect fixes pass, and scores depend on the program wrapped around the model. We cover the details in What a SWE-bench score actually measures.

The research that comes closest to modernization work grades the way our browser tests did. RepoMod-Bench asks agents to port whole repositories to another language, then runs the same hidden black-box tests against the original and the port (Li et al., 2026).

The result is a steep drop with size. Agents passed 91.3% of tests on projects under 10,000 lines and 15.3% on projects over 50,000. Enterprise systems are usually far larger than that, so this is the range where measurement matters most.

What we measure now

• Behaviour through the real interface. Browser and API tests written against the legacy system's behaviour, then run against the new one.
• Every rule accounted for. Each extracted business rule is owned by a specification, kept in place on purpose, or excluded with a reason. Nothing is silently dropped. (More in Where 1,658 business rules ended up.)
• Checks on the checks. A coverage figure, a review verdict or a pass rate is itself a measurement, and it gets audited like one. (More in What AI code review catches.)

Six questions to ask about any AI coding claim

1. Which test set? Benchmarks have subsets with very different trust levels.
2. Who ran it? An independent leaderboard, a paid review and a self-report are different kinds of evidence.
3. One answer per task? Picking the best of several attempts inflates scores if the picker can see the grading tests.
4. How big was the margin? On a few hundred tasks, a 2-point lead is noise.
5. Do the tasks look like your work? A one-hour bug fix in a famous Python library says little about a 400,000-line VB.NET system.
6. Was the result checked from outside? Black-box tests of the finished system are the strongest evidence there is.

Sources

All accessed 2026-10-30.

• Li et al., "RepoMod-Bench: A Benchmark for Code Repository Modernization via Implementation-Agnostic Testing", 2026-02-26. https://arxiv.org/html/2602.22518