The Window Got Bigger. The Memory Did Not.
Co-Founder & CEO of CLRT
The upgrade looked like the end of a familiar complaint. The assistant that lost the thread halfway through a long contract, that forgot the decision taken three exchanges earlier, that had to be told the same context every morning, was now offered a window of a million tokens, and in one case ten million. Whole data rooms fit inside it. So the purchase was made on the number, the way storage or bandwidth is bought, and the same number now justifies the next purchase, a memory layer that remembers you across sessions. The problem is that the number was never the capacity that mattered. The benchmarks that measure how much of a window a model can actually use find a figure smaller than the advertised one by a factor of tens to thousands, and the memory products inherit the gap.
Start with the measurement the marketing omits. A model's advertised window is the number of tokens it accepts without an error. Its effective window is the number it can accept while still doing the job it did on a short prompt, and no buyer can predict one from the other. The NoLiMa benchmark from Adobe Research, accepted at ICML 2025, defines effective length as the longest context at which a model keeps at least 85 percent of its short-context score, and its leaderboard puts the two side by side. GPT-4.1 advertises a million tokens and holds 16 thousand. GPT-4o advertises 128 thousand and holds 8 thousand. Gemini 1.5 Pro advertises two million and holds 2 thousand. Llama 4 Scout advertises ten million and holds 1 thousand. The gap runs from roughly sixteen times to ten thousand times, widest on exactly the models whose windows are sold hardest.
The shape of the decline matters more than the endpoint, because it begins long before the window is full. NoLiMa sees what a needle test misses because its answer does not share words with the question, so the model has to infer rather than match. Under that condition, 11 of the 13 models tested, all claiming at least 128 thousand tokens, fell below half their short-context score by 32 thousand tokens, a quarter of the smallest window advertised. GPT-4o, one of the two exceptions, still slid from 99.3 percent to 69.7. Reasoning did not rescue the harder subset: o4-mini fell from 99.6 to 11.7. Chroma, a vector database company whose July 2025 report is vendor research, tested 18 models on a memory task and found the same pattern from the other side: a focused prompt of about 300 tokens beat the full history of about 113 thousand tokens on every model tested.
It would be convenient to believe the 2026 models have solved this. LongBench Pro, a January 2026 benchmark of 46 long-context models on 1,500 natural documents rather than synthetic needles, concludes that effective context length is typically shorter than claimed. An August 2026 preprint adds the other half of the diagnosis, on the newest models: Claude Haiku 4.5, Sonnet 4.6, Opus 4.7 and GPT-5.5. It shortened a long prompt two ways. Cut blindly from the middle, the score collapsed as the cut deepened, because the cut deleted the answer along with everything else. Cut with knowledge of which passages were distractors, performance was preserved or improved, and the two smaller Claude models gained significantly. What is in the window decides the result. The failure was never that the window was too small. It was that the window was full of the wrong things, and a bigger window makes that easier. Capacity is a purchase. Curation, deciding what the model should look at for this task at this step, is engineering judgment, and no price list carries it.
Which brings us to the next purchase. Having bought the window as capacity, the same buyer is offered memory as capacity, a layer that stores what you said across sessions. Measured, it fails in the same place. LongMemEval, published at ICLR 2025 by researchers at UCLA and Tencent, put 500 questions to commercial chat assistants and long-context models across sustained conversations and recorded a 30 percent accuracy drop on remembering information across them, on the products executives actually use. Worse, memory adds a failure the plain window did not have. A June 2026 study of three memory systems across five model families, revised in August, found that memory amplifies sycophancy in every condition tested, by up to 40 percent over the same model without it. The mechanism is exact: the memory layer stores compressed snippets of what the user said, keeping the misconception and dropping the correction that followed. The assistant that remembers you remembers your errors as facts.
The clearest confirmation that curation beats capacity comes from a company that sells capacity. Anthropic's own engineering guidance, published in September 2025, defines context rot as the decline in recall as the token count rises, describes a finite attention budget that every token spends, and prescribes three remedies: compacting the history, keeping structured notes outside the window, and handing focused sub-tasks to separate agents that return summaries of a thousand or two thousand tokens. Every remedy keeps the window small. The vendor whose models accept hundreds of thousands of tokens tells its builders to feed it far fewer, chosen well. A larger window is a bigger room. Memory is a bigger cupboard. Neither decides what goes in, what comes out, what is stale, and what the model should attend to now. That decision is the product, it is made in engineering for each workflow, and it cannot be bought by the token.
Capacity is a purchase. Curation is engineering judgment, and no price list carries it.
A deeper dive
The mechanism is worth holding precisely, because it explains why the obvious in-house fixes backfire. A model does not read a long context the way a person reads a file. It spreads a fixed quantity of attention across every token present, so each additional token dilutes the attention available to the ones that matter, and the dilution is uneven: Chroma's tests found that a single irrelevant passage resembling the answer lowers accuracy on its own, and that more of them lower it further. Stanford researchers named the pattern in 2023, finding performance highest when the relevant passage sat at the start or end of the context and lowest in the middle; windows have grown many times over since and the pattern has not gone. This is why the reflex of putting everything in the prompt, the whole contract, the whole ticket history, is the opposite of a safeguard: it surrounds the passage the task depends on with the passages most likely to distract from it. It also explains why the vendor's needle test passes while the real task fails: a needle test asks for a sentence that shares words with the question, and business questions almost never share words with the evidence that answers them.
The second-order trap arrives when the organisation, having noticed the problem, reaches for a summary. Compaction, condensing an old conversation into a shorter one, is a genuine remedy, but it is a lossy decision about what mattered, made before anyone knows what will be asked next, and what it discards cannot be recovered by any later prompt. A memory layer is the same decision made permanent. The memory sycophancy result shows what that looks like: the extraction step stores discrete snippets of what the user said, so the correction, which arrived elsewhere in the conversation, is not what gets written. Over months the store fills with a flattering version of what the organisation believes, and the assistant that consults it becomes more agreeable and less accurate at once, exactly where an executive is most likely to trust it. None of this is fixed by a larger store. It is fixed by deciding, per workflow, what is retrieved, what is compacted, what is written to memory and what never is, and by measuring the task at the length it actually runs rather than the length the demo ran. That is a design discipline, and no vendor sells it.
Work with CLRT
If your assistant still loses the thread after the window upgrade, the problem was never capacity, and the memory upgrade will not solve it either. It is a question of what the model should be looking at, for which workflow, at which step, and that is diagnostic work before it is engineering work. CLRT Ascent at ascent.clrtstudio.com is where we map which of your workflows can carry a long context and which need curation built around them. And when a workflow is worth running, CLRT builds the retrieval, compaction and memory discipline into the system, with the evaluations that prove it holds at the length your business actually runs.
Vishal Sachar is the Co-Founder and CEO of CLRT, where he helps UAE businesses make sense of applied agentic AI and put it to work. He writes on agentic systems, AI governance, and the economics of automation. Reach him at vishal@clrtstudio.com or on LinkedIn.
