Even the best AI coding tool fully finished only 1 in 4 real requests
Researchers from Alibaba and two Chinese universities tested 10 AI models on 273 real app changes people had asked for. The top model met most requirements but finished the whole job only 26.4% of the time.
The short version
- Researchers from Alibaba and two universities in China built a test from requests real people made to AI coding tools.
- The best of 10 AI models met 69.9% of the requirements on an average task. It fully finished only 26.4% of tasks.
- Every model did worse when one request touched many parts of an app. No single model came first on everything.
How the test works
The researchers come from Alibaba Group, Fudan University and Zhejiang University. They call their test Co-Design Bench.
The team started from 26,314 real coding sessions and 37,463 distinct user requests. From those, they picked 273 tasks where people asked an AI to change a working app. Many requests came with a screenshot, video, audio clip or document attached.
Experts wrote 2,331 hidden checks for those tasks. A check might ask whether a button now does what the user wanted. A second AI then double-checks every pass, looking for proof in the code and the running app.
The tasks come in two kinds. Some are small edits to one part of an app. Others need coordinated changes across a whole software project.
All 10 models ran inside the same coding tool, Claude Code, with the same instructions. That keeps the comparison fair.
The best model still missed something in most tasks
The top model, OpenAI's GPT-6 Astra, satisfied 69.9% of checks on an average task. Yet it left at least one requirement unmet in 73.6% of its tasks. Its full-pass rate was 41.0% on small edits and 17.9% on project-wide changes.
Changes that touch more parts of an app were harder for every model. Tasks needing three or four kinds of change scored lower than tasks needing one or two.
The paper also groups tasks by what the user wanted most. When the main goal was a new working feature, models fully finished only 10.3% of attempts. That was the lowest rate of any group.
More effort did not buy better results. On small edits, OpenAI's model typically took 32 rounds of work. Another model took 69 rounds and still scored lower.
Rankings also moved between the two kinds of task. Alibaba's smaller Qwen model came fifth on small edits but second on project-wide changes. Overall it ranked fourth. OpenAI's model led overall. Anthropic's Opus 5 came second and a larger Qwen model came third.
What it means for your company
This test puts numbers on a common experience with AI coding tools. The AI gets most of the way, then misses a detail. On real requests, that happened in most tasks, even for the best model.
The gap matters for planning. A change that looks mostly done still needs a developer to find what is missing. If your team counts AI output as finished work, your estimates will be too optimistic.
The results also argue for smaller requests. Broad changes that touch many parts of an app were harder for every model tested.
Model choice depends on the job. The ranking shifted between small edits and project-wide work. A model that looks best on one kind of task may not be best for yours.
What to do next
- Ask your developers to check every AI-made change against the original request before it goes live.
- Write clear acceptance checks for each request, the way this test did. They help people and AI alike.
- Break big changes into smaller requests that each touch one part of the app.
- Test two or three models on your own past requests before choosing one.
- Track how often AI work is fully done on the first try, and use that figure in your plans.