Enterprises are paying for multi-model orchestration, confident it creates a safety net. But the study’s biggest punchline is simple and brutal: the models can be wrong at the same time, and that “all-wrong together” tail sets a ceiling no router can escape.

On the open-ended MATH-500 benchmark, the researchers tested a 67-model pool that included GPT-5.5, Claude Opus 4.8, and Gemini 3.1 Pro. Based on standard pairwise correlation math, they predicted the entire pool would fail simultaneously on only 2.3% of questions. In reality, the co-failure rate was 5.2%. That is an underestimate by roughly 2.25x, and it is the kind of gap that turns “cost-saving optimization” into “expensive placebo.”

To understand why, you have to understand what most orchestration strategies assume. Teams route or ensemble models with the belief that different models have different blind spots. So if one model stumbles, another might catch it. The study says the real limit is not how often models disagree, and not whether each individual model is “good” on average. The limit is the percentage of prompts where every model in the pool gives the wrong answer at once. That is the named metric: the co-failure ceiling.

Most developers orchestrate multiple language models using three common architectures. Model routers act like traffic cops, sending complex prompts to expensive models and simpler prompts to cheaper ones. Cascades send every prompt to a cheap model first, escalating only when confidence drops. Mixture-of-Agents (MoA) asks multiple models the same question and fuses their outputs into a synthesized answer. All of these add operational overhead, a literal “shadow price” to inference costs: extra latency, more complex infrastructure maintenance, and increased governance risk across multiple API providers.

Boards and CFOs usually accept that overhead only if the performance gain is real. The problem is how engineers justify the model pool. They often lean on “pairwise error correlation,” picking models that fail on different types of prompts. The intuition is intuitive: if Model A fails on SQL but is great at Python, and Model B fails on Python but is great at SQL, then together they should cover each other. Pairwise correlation looks low, so the system should be safer.

The study says that logic can fall apart in two ways. First, if you vote across diverse but unequal models, the weaker ones can outvote the stronger one. The paper cites Josef Chen, the paper’s author, saying that in their experiments, “Naive majority voting across unequal models had negative mean gain (minus 10 points on our hard mix): diverse-but-weaker members outvote the strong one.” Second, and more importantly for the “why is this inaccurate math so common” story, pairwise correlation cannot see the all-model failure tail. Chen describes the driver as a “common-mode atom,” a slice of queries on which the entire market fails together, which no pairwise statistic can see.

The study also shows that task format can make co-failure worse. When researchers took graduate-level science questions from the GPQA benchmark and changed them from multiple-choice to free-response formats, the all-wrong tail expanded to 12.7%. That is a warning for teams trying to “just route harder prompts” into better answers. Open-ended generation is exactly where teams tend to deploy multi-model systems. The math says they buy the least exactly where they want the most.

The researchers lay out a sharper framework for where the co-failure ceiling bites and where it changes shape. In ceiling-bound environments like open-ended math, co-failure is high because the task is too hard and models fail simultaneously. No amount of routing bypasses the lack of underlying capability. In realizability-bound environments like graduate-level science, co-failure may be near zero because at least one model often knows the answer, but the models can disagree subtly. Routing cannot reliably select the right answer without something approaching an oracle.

This is where governance, capital allocation, and compliance risk meet. Many enterprises now build orchestration to chase marginal accuracy gains, then rely on the same correlation metrics to predict production performance. The study says that assumption can be wrong specifically when it matters most, when frontier models “disagree.” Teams pay orchestration overhead up front with latency, complexity, and multi-provider operations, on the assumption that a “diversity dividend” arrives later. Chen’s quote gets at the failure mode: “So teams pay the orchestration overhead up front (latency, complexity, multi-provider operations) on the assumption that a diversity dividend arrives later. Usually it doesn't, because today's best models agree, and, worse, they fail on the same queries … the prompt simply carries little signal about which model will be the one that's right when the frontier disagrees.”

The good news is that the paper also offers a practical pre-deployment check that does not require new model contracts or expensive experiments. The researchers propose calculating the absolute co-failure ceiling for free using a mathematical formula called a Clopper-Pearson bound. It is a worst-case scenario calculator. Like flipping a coin ten times and getting eight heads, you cannot guarantee the true head rate stays that high forever. The bound takes a small sample of test questions and outputs a mathematically guaranteed ceiling.

In practice, enterprises would hold out a dataset and count the number of prompts where all models fail together. The paper’s example: if a team tests five agents on 50 sample queries and finds they all fail together on just two questions, a developer might assume 96% accuracy in production. The Clopper-Pearson calculation corrects that optimism, showing the true co-failure rate could actually be as high as 12% given sample size uncertainty. The study notes that integration can be “trivial,” described as a counting job over eval logs already produced, running in the same CI stage as the eval suite and re-triggering when tests are updated. For executives, that means you can de-risk multi-model plans before you ship routing logic into a production system with real customers, real SLAs, and real liability.

If you are a founder, investor, or operator overseeing AI systems, the strategic stakes are straightforward. Multi-model orchestration can be useful, but only within the limits of what models can collectively cover. The co-failure ceiling tells you where that limit lives, and it tells you not to trust pairwise confidence when the tail is shared. Tomorrow’s orchestration stacks will win or lose based on whether leaders demand ceiling-aware evaluation instead of correlation-based reassurance.