We gave the model everything. 92 of 100 outputs failed.

Context windows keep growing, and it's tempting to treat that as the answer to large legacy systems: put the whole thing in the prompt and let the model sort it out. We tried a version of that in May 2026, on the first full end-to-end run of ARC's planning stage, and it failed badly.

The test system was phpMyAdmin, the open-source database tool. It's a useful stand-in for a mid-sized legacy application: years of accumulated features and plenty of business logic. Discovery extracted 3,117 business rules from it, about 8.5 MB of text.

What went wrong

The planner splits a system into pieces and writes a plan for each one. Every one of those planning steps was given the full rule set, on the theory that more context can't hurt.

For the most detailed pieces, the assembled prompt came to about 2.02 million tokens. The model's limit was 1 million, so the API rejected the request. 92 of the 100 planned artifacts came back blocked, and the run ended with none of the 3,117 rules covered.

One part worked. A coverage check at the end of the pipeline refused to certify the output, so the failure was loud rather than a plan full of placeholders. That gate is why the run was a cheap lesson rather than an expensive one.

Bigger windows don't fix this

The obvious response is to wait for a 2-million-token window. The research suggests that wouldn't help much.

• Models use the middle of a long context worse than its beginning and end. In the original study, accuracy dropped when the relevant fact sat in the middle of the input (Liu et al., "Lost in the Middle").
• Across 18 models, performance degrades as input length grows, even on simple tasks the model handles easily at short lengths (Chroma, "Context Rot", 2025).

An agent writing the plan for one feature needs the dozens of rules that feature owns, not 3,117. The rest is noise it has to read past.

What fixed it

We stopped handing everything to every step. Each piece of the plan now works from the rules that belong to it, and nothing else. A final check, which doesn't depend on a model, makes sure no rule is left without a home before the plan can pass.

The re-run had zero oversized prompts. It produced 406 complete plan documents, and the coverage report showed 0 of 3,117 rules missing.

One detail matters for honesty: 2,450 of those rules were placed by the final check, not chosen by a model. That's the design working as intended. It also means "every rule is covered" and "a model reasoned about every rule" are different claims, and we report them separately.

What this means if you're evaluating AI on a large system

• Ask how context is chosen. "We use a 1M-token model" isn't an answer for a system with millions of lines. How the right material reaches each step, and what happens to what's left over, is.
• Ask what happens when it doesn't fit. A tool that fails quietly is worse than one that fails loudly.
• Separate coverage from reasoning. A plan can account for every rule mechanically while a model reasoned about only some of them. Both numbers are worth seeing.

Sources

All accessed 2026-10-30.

• Liu et al., "Lost in the Middle: How Language Models Use Long Contexts", TACL 2024. https://arxiv.org/abs/2307.03172
• Chroma, "Context Rot: How Increasing Input Tokens Impacts LLM Performance", 2025-07. https://research.trychroma.com/context-rot