Arbor from Renmin and Microsoft Research beats Claude Code and Codex by 2.5x

If you’ve watched an AI coding agent work beautifully in development and then go sideways in production, you’ve seen the core problem: it can’t reliably attribute what changed, so it can’t accumulate learning. Arbor, introduced by researchers at Renmin University of China and Microsoft Research, targets that exact failure mode. In practical tests, Arbor delivered more than 2.5 times the verifiable performance gains of standard AI coding agents across real-world engineering tasks, while operating under the same resource budget, using a new structure for long-horizon optimization.

The trick is not “give the agent more time.” The paper’s co-author Jiajie Jin makes the point bluntly: “Automation can keep an AI working for a very long time - but a loop is not the same as progress.” Arbor is designed for the part that typically breaks: autonomous optimization (AO) is a loop, but without a data structure to preserve state and evidence, repeated attempts turn into faster repetitions of the same mistake.

So what is autonomous optimization in plain English? An AI agent starts with an initial mutable artifact, like a machine learning codebase or a data pipeline, and a specific objective. It iteratively improves that artifact using experimental feedback, without step-by-step human supervision. Teams often assume that the fix is simple: add more compute or more time. But the research argues that complex tasks take many attempts, and the standard architectures are missing the capacity to accumulate and compare evidence across attempts.

In current agent systems, the “memory” is often a conversation transcript. That works up to a point. AO tasks span hundreds of turns and quickly exceed context window limits, which means the agent starts losing factual evidence over long histories. When early experiments fail, the agent can stall. When evaluation noise swings, it can chase the wrong signal. In short: it does experiments, but it doesn’t keep the durable structure of what happened and why.

Arbor addresses this by separating research strategy from implementation details. It uses two collaborating components. The coordinator is a long-lived AI agent that acts like a principal investigator. It never directly edits the target codebase. Instead, it owns general state for the optimization research, observes accumulated evidence, generates new hypotheses and directions, and decides what to explore next. The executors are short-lived, focused agents. When the coordinator wants to test an idea, it spins up an executor in an isolated environment, essentially a fresh git worktree. Each executor gets one hypothesis, implements the idea, runs evaluations, debugs errors, and reports results and created artifacts back to the coordinator.

The key data structure is what the researchers call “Hypothesis Tree Refinement” (HTR). HTR represents the entire research process as a persistent, branching tree. Each node binds four things: a hypothesis, the executable artifact, the factual evidence produced, and a distilled insight. This matters because Arbor can explore multiple competing directions without losing its place. Broad ideas sit near the root, and concrete refinements branch toward the leaves. If an executor’s experiment fails, the tree records why it failed as a negative constraint, so future exploration doesn’t repeatedly stumble into the same ditch.

The framework also tackles a second issue: reward hacking and overfitting to development metrics. Agents can sometimes “improve” a score without producing improvements that transfer. To prevent that, HTR enforces a strict “merge gate.” Even if an executor reports a fantastic development score, the coordinator tests the candidate against a held-out test evaluator in an isolated worktree. The artifact is only merged into the current best trunk if it demonstrably improves the test score. In enterprise terms, this is the difference between optimizing for a scoreboard and optimizing for real performance.

Finally, Arbor solves a practical isolation problem that shows up in real engineering workflows: general coding agents often chain tool calls on a single shared working tree. That architecture makes it hard to test parallel hypotheses safely. Arbor treats each lever as a separate hypothesis, including changes that enterprise teams typically tangle together, like chunking strategies, retrieval methods, and prompts.

The paper illustrates the point with a common enterprise scenario: optimizing a Retrieval-Augmented Generation (RAG) pipeline for an internal AI assistant. Jin explains that when a single agent like Claude Code or Codex is asked to “improve accuracy,” it typically changes multiple things in one pass, such as chunking, the prompt, and the retrieval method. That entanglement makes it impossible to attribute which change actually helped. It also directly mutates the repository without isolation. Arbor avoids this by implementing and evaluating each lever in its own isolated git worktree, producing clean attribution, for example: “constraint decomposition on the retrieval side gave +X; breadth-first search actually hurt.”

Where this gets strategically interesting is how it reframes how boards and operators think about autonomous systems. AO isn’t just a model capability problem. It is a systems design and evaluation integrity problem: how you store evidence, how you branch experiments, how you prevent overfitting, and how you ensure progress is measurable in held-out tests. Arbor generally falls under “loop engineering,” popularized by industry figures like OpenClaw creator Peter Steinberger and Claude Code lead Boris Cherny, and it pushes the industry toward designing iterative cycles that observe, reason, and then act with accumulated, verifiable insight.

For enterprises rolling out AI agents into production workflows, the second-order implication is big: the bottleneck shifts from “Can the agent try things?” to “Can it learn what works without corrupting the product and without mistaking dev metrics for reality?” Arbor’s 2.5x verifiable gains under the same compute budget suggest that the path to better outcomes may be less about bigger models and more about better experiment structure. If your agent team is spending months doing trial-and-error across chunking, retrieval, and prompts, Arbor is a blueprint for turning that chaos into cumulative optimization.