Half the Time Is Not the Same as Every Time
Co-Founder & CEO of CLRT
Somewhere in your planning this quarter there is a sentence that reads, roughly, the agent can now do a day's work. It comes from a chart most executives have seen by now: a line of frontier models climbing steeply, each point labelled in hours, the newest at twelve or seventeen. The workflow being designed around that sentence is a real one, and the plan assumes the agent will be handed it in the morning and produce it by evening, most days, without a person in the loop. The chart does not say that. The chart says something narrower and less useful, and the organisation that published it has spent this year saying so in plain language. What it measures is the length of task at which the model succeeds half the time. The column you need is the other one.
Start with what the number is. METR, the evaluation lab that publishes the chart, defines the 50 percent time horizon as the length of task, measured by how long a human expert takes, at which the model is predicted to succeed half the time. Half. The same data file also carries an 80 percent column, the task length at which the model succeeds four times in five, and nobody quotes it. For Claude Opus 4.6 the headline figure is 718.8 minutes, near enough twelve hours. The 80 percent figure is 69.9 minutes. For Claude Mythos Preview, the headline is 1,044.8 minutes, a little over seventeen hours, and the four-in-five figure is 185.9 minutes, about three hours. Across the 2026 models in the file the gap between the two columns runs from four times to ten times. A twelve-hour agent, read at the reliability a business would actually accept, is a seventy-minute agent.
The lab has been unusually direct about this. In January it published a note clarifying the metric's limitations, which states plainly that time horizon is not the length of time AIs can work independently. It adds that the error bars have historically been a factor of about two in each direction, and that horizons are 40 to 100 times lower for visual computer-use tasks than for the software tasks the headline measures. An earlier METR analysis across nine benchmarks put the frontier computer-use horizon at roughly 50 times shorter than other domains. Then, in May, the page carrying the chart gained a notice: measurements above 16 hours are unreliable with the current task suite, and the data file's doubling-time fit excludes any model above that line. The newest model in the file sits above it. The instrument has been outrun by the thing it measures.
The reason the columns diverge is now measured rather than argued, in two preprints posted two days apart at the end of August, neither yet peer reviewed. One, a study of nine models across 10,664 agentic tool-use trajectories, finds that task success follows a geometric law governed by a single per-step reliability parameter, which rises with model scale but saturates well below one, and that every model tested falls from near-perfect success to near zero within sixteen steps. Its projection puts reliability at 0.42 at the length of a standard agent benchmark and 0.24 at hundred-step production horizons. The other, a benchmark of long dependent API workflows across 19 models and 44,362 transcripts, reports success falling from 93 percent on individual subtasks to 74 percent on clean twenty-subtask chains, and 61 percent once the trials no model passed are counted. The two papers disagree on the exact shape of the curve. They agree on its direction, and on the number that describes production.
The second finding in that benchmark is the one to write down. Across the 19 models, best-of-five performance spans seven points; all-five-of-five reliability spans 44 points. Pick a model on its best attempt and the field looks interchangeable. Pick on whether it gets the same result every time and the field spreads by a factor of six. This is the same lesson Sierra's tau-bench recorded in 2024, when the best function-calling agents passed under half its tasks once and under a quarter of the retail tasks eight times running. Every demo you have watched was a best-of-n event. Every workflow you will run is an all-of-n event. Read METR's chart on the four-in-five line and the calendar moves as well: the reliable horizon of the newest model in the file is where the headline horizon stood eight to twelve months earlier. The frontier you are planning around arrived, at the reliability you need, about a year ago.
None of this says the work cannot be handed over. It says the question was never how long the agent can work. It is which of your processes can tolerate a one-in-five miss at seventy minutes, or a one-in-two miss at twelve hours, and what surrounds the agent when it misses. A workflow that produces a draft a person will read anyway tolerates the coin flip comfortably. A workflow that posts a journal entry, sends a client the number, or closes a ticket does not, and the difference is not a property of the model. It is a property of the step count, the cost of a wrong output, and whether wrongness is detectable before it leaves the building. Those are judgments about your business, made process by process, before any engineering starts. The model is the same for everyone. Knowing where the four-in-five line falls in your operation is not.
The number the market quotes is the coin flip. The number a workflow needs is the four in five, and it is a fraction of it.
A deeper dive
The mechanism is worth holding precisely, because it decides which fixes work. An agent run is a chain of steps, and Mittal's study finds the chain's success is close to the product of one per-step reliability figure. The hope is that a better model pushes that figure to one. The finding is that it does not: the parameter rises with model scale and then saturates below one, so the curve flattens at a ceiling rather than reaching it, and the ceiling guarantees collapse at some horizon for every model, including the next one. The same study tested the obvious remedy, bounding the context so the model is not reading its own accumulated history, and found it steepened the decay rather than easing it. Degradation tracked the step count, not the context length. The API-workflow paper adds a detail that changes where the risk sits: on its clean chains, three-quarters of the failing runs had done the work correctly and failed at the last mile, handing the result over. That is not a reasoning failure, and it will not be fixed by a smarter model. It is a systems failure, and it is invisible to any evaluation that reads the transcript rather than the delivered artifact.
The second-order trap is that the intuitive responses each make the problem harder to see. Retrying a failed run until it passes turns an all-of-n process into a best-of-n one, which is exactly the measurement the benchmark just showed to be uninformative, and it hides the reliability gap inside a rising bill. Putting a person at the end of a twelve-hour run asks a reviewer to verify twelve hours of intermediate decisions in a fraction of that time, which is where reviewers approve rather than check. Evaluating the workflow at the short chain lengths the benchmarks use produces a confident go decision for a hundred-step process the same paper projects at one in four. And the messy, high-context, cross-application work an operator most wants to hand over is the domain METR measures at one-fortieth to one-hundredth of the headline. The organisations that will get this right are not waiting for the line to reach a month. They are deciding, process by process, which step counts and which miss rates they can live with, and building the checks at the points where the chain is known to break rather than at the end.
Work with CLRT
Deciding where the four-in-five line falls in your operation, which processes tolerate the coin flip and which need the chain engineered and checked at its known breaking points, is the work CLRT does, and it is judgment before it is engineering. CLRT Ascent is the diagnostic built for exactly this question: it maps where agentic AI actually pays in your business at the reliability you would accept, not the reliability a chart implies, at ascent.clrtstudio.com. When a process clears that bar, CLRT builds the verification and delivery layer that turns a twelve-hour headline into a workflow you can stop watching.
Vishal Sachar is the Co-Founder and CEO of CLRT, where he helps UAE businesses make sense of applied agentic AI and put it to work. He writes on agentic systems, AI governance, and the economics of automation. Reach him at vishal@clrtstudio.com or on LinkedIn.
