Skip to content
LIVE
The Executives BriefThe Executives Brief

METR's new test: AI beats almost every human at code, flunks judgment

The nonprofit's updated time horizon plot adds open-ended tasks for the first time, and not one AI-designed evaluation was good enough for its own researchers to use.

ByLama Al-RashidTechnology Correspondent, The Executives Brief
·5 min read
METR's new test: AI beats almost every human at code, flunks judgment
Executive summary

METR, the AI evaluation nonprofit, updated its widely cited time horizon plot with open-ended, judgment-heavy tasks and found that frontier models still trail humans on them even as they outperform almost every human at software engineering. For executives betting on AI to automate research and development, the report says the bottleneck is not raw capability but the ability to measure, and therefore trust, progress on the work that matters most.

METR's updated time horizon plot lands with a split verdict that should unsettle anyone planning to hand research and development to a model. The AI evaluation nonprofit asked frontier models to design new AI evaluations that METR could actually use in its own work, then had METR employees score the results. Not one AI-designed evaluation was good enough for those employees to consider it useful for their research. The same report notes that the most advanced models are now better than almost every human at software engineering. Read those two findings together and you get the central tension in AI right now: systems that can out-code nearly everyone still cannot be trusted to judge what is worth doing.

The report is an update to METR's time horizon plot, a chart that became one of the most cited artifacts in AI because of a single clean number: every four months, the length of the tasks frontier models could successfully complete doubled. It was also, by METR's own framing, widely misinterpreted, owing to the complexity of the measures behind it. The method is straightforward in principle. METR has models attempt a large suite of tasks, each of which humans have also completed. Each task's difficulty is scored by how long it took the human testers to finish it, and models are then rated by the length of the tasks they can complete. The new version adds tasks estimated to take humans far longer than anything in the original suite, plus others that are more open-ended.

The question METR is chasing is whether progress on "verifiable" tasks, where success or failure can be checked automatically, carries over to work where it cannot. We know how to teach neural networks verifiable tasks: researchers have the system attempt one many times, strengthen the neural pathways that contribute to success, and weaken the pathways that contribute to failure. That is reinforcement learning, and it is why AI systems have become so good at coding and math, two domains where a solution can be graded automatically. Applying the same technique to tasks where success is less well defined, or takes longer to ascertain, such as choosing a new research direction, is much harder. It is also much harder to evaluate performance on those tasks, which is why METR's original plot left them out entirely.

That omission mattered, because these are precisely the tasks that would have to fall before AI systems can automate research and development, and AI R&D is a live nerve right now. Parts of the AI community are still reeling from a different, heavily redacted METR report from earlier this month, which claimed AI might be speeding up the pace of AI progress at Anthropic by a factor of two. That report sparked fears the company, which just had a blockbuster IPO, could soon reach full recursive self-improvement, a regime in which AI systems autonomously design their own successors with no human involvement.

So METR built fuzzier, judgment-laden tasks into the new suite. In one, models design new AI evaluations for METR's researchers, who then score them, producing the zero-usable-results outcome above. In another, models score philosophical arguments, a task meant to capture whether a model can exercise good judgment. METR writes in the report that its employees disagree about how relevant that task is to automating AI R&D, and on it, too, models fell short of human performance. The measurement itself is shaky. For the philosophy task, just one person rated most of the arguments, and agents are judged by how well they agree with that individual. Completion times in the new report are mostly estimates rather than measured, because paying people for tasks that might take hundreds of hours, or that only a few experts are qualified to complete, is prohibitively expensive and time-consuming.

Outside researchers see the direction as right even if the signal is noisy. "We're at the point where AI evaluations need to get more creative," says Sayash Kapoor, a computer scientist and incoming professor at UC Berkeley. "Trying to push the envelope of the space of evaluations is a welcome change." Kapoor says he was not surprised that models fall short on open-ended work; in an August preprint, he and collaborators reported that AI models are not yet able to reproduce human computer science research on their own. Najoung Kim, a linguist and computer scientist at Boston University, put the caution plainly: "The improvement we observe, I think that does tell us something for the current moment. I'm not sure just yet how to think about how it would extrapolate." Matt Putz, a member of the technical staff at METR, said in an email that hard-to-verify or judgment-laden tasks are "much harder to study, and especially to quantify," adding: "Ultimately, I don't think we'll be able to establish trends as clean and robust as for the easy-to-specify [tasks]."

There is a counter-signal worth holding alongside the skepticism. Progress on verifiable tasks does not appear to be slowing, and the newest models, such as Opus 5.5, tend to hit what METR calls "borderline above the bar" performance on the evaluation design tasks, not yet usable for METR's research but promising. A year ago, every model scored in the "clearly below the bar" category. OpenAI added its own data point in a Wednesday blog post, reporting that its models do not do as well on the real-world tasks its employees assign them as they do on the easier, more verifiable tasks in METR's original suite. Kapoor notes that people still disagree about how to read the gap: he thinks research taste, the ability to choose promising questions, hypotheses, and experiments, will prove a major bottleneck in automating AI R&D, while others contend models trained on easy-to-verify tasks will eventually handle open-ended work too.

For executives, the practical read is that the benchmark justifying an AI budget may be measuring the wrong thing. Coding agents are genuinely strong and getting stronger; the judgment-heavy work upstream of them, deciding what to build, which experiment to run, which research direction to fund, is where models still trail the humans they are meant to assist. METR's own caveat is that it includes few of these tasks, which makes extrapolation hard, and the experts agree we are far from definitive answers on whether, or how fast, automated AI R&D arrives. Kapoor's summary is the one to keep: "This move toward open-ended tasks is promising," he says, but "there is just such a lot more to be done to actually get a fuller understanding."

Executive ActionsLocked

This story's Key Insights and Take-aways are locked.

Create a free account to unlock Executive Actions for one credit.

Register to Unlock

Always free for Executives Club members. Join the Club

More in Technology