Same model, 3 coding-agent tools. Similar scores, up to 3x the bill.
A study ran 5 models through 3 popular coding-agent tools on hundreds of real programming tasks. Swapping the tool changed results no more than rerunning the same one. It changed cost a lot.
Companies buying AI coding agents usually compare the tools by benchmark score. A paper posted to arXiv on October 3 by Yangze Liu and Zhongyi Han of Shandong University suggests that score tells a buyer far less than it seems, and that the bill tells more.
A coding agent is a language model plus the software around it: the prompts and tools wrapped around the model, plus its rules for trimming a long conversation. Vendors ship changes to that layer constantly and advertise the gains. The authors held the model fixed and swapped only that layer.
What they tested
They ran 5 models, among them DeepSeek-V4-Flash and GLM-5.3-Flash, through 3 agent tools as shipped: Claude Code, OpenCode and the minimal mini-SWE-agent. Claude Opus 5 ran inside Claude Code as a reference point. The tasks came from SWE-bench Verified, a standard set of real bug fixes from open-source projects, split into 447 ordinary tasks and the 45 hardest ones. Each run got the same budget of 300 model calls, with no web access, so no agent could look up the published fix.
The important design choice was rerunning identical setups. Earlier comparisons mostly ran each combination once, while other research has found that reruns of a single setup on this benchmark can differ by 2.2 to 6.0 points on their own.
Swapping the tool moved scores about as much as luck
On the 45 hard tasks, changing the tool flipped the outcome on 13% of tasks. Rerunning the same tool also flipped 13%. The tasks a tool won in one run were often lost in the next. On the larger pool, the heaviest tool (Claude Code) and the lightest (mini-SWE-agent) came out equivalent within 5 points on both models tested there.
The one difference that cleared the noise was a loss. OpenCode trailed by up to 9 points on the large pool. On one model, about half of that gap came from runs where the model's output hit a length cap and OpenCode did not ask it to continue, so the attempt ended with nothing written.
The bill is where the tools differ
With the same model and tasks priced on one list, cost per task differed by up to 3x. Most of that difference comes from text the tool resends with every step: its system prompt and tool descriptions. The authors measured 16,581 tokens of it in Claude Code against 829 in mini-SWE-agent, with OpenCode in between at 7,025. On GLM-5.3-Flash, a task cost 1.81 yuan under Claude Code and 0.97 under mini-SWE-agent. OpenCode was cheapest at 0.56, partly because the model stopped sooner, which is also where it lost tasks.
The ranking also depends on the model. On DeepSeek-V4-Flash, mini-SWE-agent took far more steps, so the lightest tool ended up moving the most tokens. Caching softens the cost, since 96 to 99% of input tokens were cache hits, but it did not change which tool was cheaper.
What it means for companies in Indonesia
Teams in Indonesia that pay for coding agents per token feel this directly. Before you pay for a tool because of a few benchmark points, ask whether those points survive a rerun. On 45 tasks, the study's data say a 13-point gap is caught only half the time.
The cost side is easier to check yourself. Run the same set of your own tasks through each candidate tool, on the model you actually plan to use, at least twice. Record how many tasks each tool completes and what each run costs in tokens. The authors suggest 3 numbers to compare between tools: the size of the first call, how fast the input grows per step, and the step count.
Also watch how a tool fails. A tool that gives up when the model's output is cut off, or that overflows the model's context window on long tasks, loses work you are still paying for.
What to watch
The study covers one benchmark, and its frontier reference model ran in only one tool, so it cannot say whether closed frontier models behave the same way. The authors ask that anyone publishing a comparison of agent tools should also publish a rerun of the same setup. Until vendors do that, treat small advertised gains as unproven, and judge tools on cost and failure behavior with your own tasks.