Checking only an AI agent's reply overstates its CRM success by up to 11 percentage points
Salesforce researchers tested 16 AI agents on 555 customer-record tasks. Replies passed their check far more often than the actions behind them, and the best agent finished 83.6% of tasks.
Companies now let AI agents change customer records. A paper posted to arXiv on October 23 by 3 researchers at Salesforce grades agents on the changes they made to those records.
What they tested
The benchmark, CRM-Agent-Bench, has 555 multi-turn tasks across 9 business areas, from retail and travel to payments, IT service management and customer support. Every record is synthetic, so it runs without a live CRM. A simulated customer talks to the agent, and its next message depends on what the agent just said, so no two conversations need to play out the same way.
A task passes only if every check it declares passes at once. Those checks cover the right actions, the right details in each action, the right order, the reply, the final state of the records, and that records the customer never asked about stay untouched. A friendly reply cannot make up for a refund that never went through.
The authors ran 16 AI systems from the major model families, including 3 open models run on their own servers rather than a paid service. Strict success ranged from 64.32% to 83.60%. Gemini 3.8 Flash led. The top 8 sat within 2.88 points of each other, so the headline rate alone barely separates the leaders.
Replies looked better than the work
Replies passed their check on 92.3% of tasks for the strongest system, and above 77% for every system. The check on whether the agent actually took the right actions passed only 70.3% to 89.8% of the time. Grading on the reply alone overstated success by 10.5 to 11.0 points in the 4 systems the authors broke down this way. In all 16 runs, the most common failure was an action that did not match what the task required.
One example from the paper: an agent ran a $15.50 checkout twice. Every check on what it said and did passed. Only the check on the records caught it, because the customer's loyalty balance had doubled to 310 points, when 155 was correct.
Rules were less of a problem than finishing the job. On the 165 tasks with something the agent must not do, the agents avoided the forbidden step in 91.2% of trials, but completed the whole task in only 66.8%, against 82.1% on tasks without such limits.
Customer support is the hardest place to start
Customer support was by far the weakest area. Across all 16 systems it averaged 56.35% strict success, more than 20 points below travel, the next-hardest area. Every model scored lower there than its own overall rate. Onboarding was the easiest, at 90.24%.
Cost did not follow accuracy. Across the 13 runs with metered prices, the estimated bill for the full test ran from $2.14 to $191.00. The leader cost $16.12. The runner-up, Claude Opus 5, scored 82.70% and cost $191.00. Results also moved between reruns. Over 3 rounds, the leader averaged 85.53%, yet only 434 of 555 tasks passed in all 3.
What it means for companies in Indonesia
If your first AI agent will sit in customer support chats, this study says that is where agents most often get the work wrong.
When you test an agent before rollout, check the records themselves. Did the refund post once to the right account? Did anything else change? Build your test cases from your own processes, including the cases where the agent must refuse or ask for more information. Run each case more than once, because the same agent can pass a task in one run and fail it in the next. Then price the agent per completed task, not per conversation.
What to watch
The data are synthetic and the simulated customer is itself an AI model, so real customers may be harder. The main leaderboard ran each task once. When vendors quote a score, ask for it broken down by business area, not as a single number.