AI assistants fail most often by acting too soon or doing too much

The short version

• Researchers had 20 experts grade 8,000 responses from 10 AI models that had to decide on their own whether to step in.
• When experts rejected a response, 61.20% of the time the AI had acted too early or gone too far.
• If you plan to let AI act without being asked, set what it may do before you give it access.

What the test found

Most AI tests hand the model a clear request and score the result. This one checks whether an AI assistant can tell when to help if nobody has asked.

A team from the University of California, Davis, the University of Southern California and the research group 2077AI built a test called HALO. They published it on October 29. Each AI model got access to a simulated person's apps, such as email and calendar, plus files it could search. Events then played out over time, and the model chose what to do.

It had 4 options. It could watch quietly, alert the person, prepare something, or carry out the action itself. There were 400 scenarios across 10 areas of life, from work to health. Experts then read each response. They judged whether it was useful and well timed, and whether the evidence supported it.

Google's Gemini 3.8 Flash scored best, with 72.88% of its responses accepted. The best model free to download, DeepSeek V4 Pro, reached 56.63%. The lowest score was 30.38%. So even the leader had more than 25% of its responses turned down.

The study's sharpest finding is about why. Across all rejected responses, 61.20% were marked either premature or excessive. The researchers' own example makes this easy to picture. A calendar entry for an upcoming trip may justify an alert about supplies. It doesn't give the AI permission to buy them.

What is new here

Earlier tests of this kind mostly used automated scores or a simple pass or fail. Those can tell you a model failed. They rarely tell you how. Here, people recorded the reason for each rejection, so you can see the pattern.

For anyone buying these tools, 3 patterns in those reasons matter:

• Holding back counts. Some models stepped in more often and got more of those actions accepted. They then lost that gain by acting in cases where doing nothing was right.
• Reading first pays off. Alerts were accepted more often when the AI had opened the relevant local files. Models free to download looked things up about 53% less often than paid ones, and they scored lower.
• Good timing isn't enough. Among well-timed responses that were still rejected, alerts tended to say too little, and actions tended to go too far.

Alerts were by far the most common choice, at 69.20% of responses. Direct action made up 16.09%. An alert that arrives at the wrong moment is still a failure, so the timing of these messages shaped much of the result.

What it means for your company

Assistants of this kind promise to spot a risk in your inbox and act before anyone asks. This study gives you a reason to slow down on the last part of that promise.

A wrong alert costs a moment of attention. A wrong action can commit your company to a purchase or send a message you can't recall. The paper's own diagram shows an AI asking a service desk to replace a machine and approving the full cost. The researchers mark that action as both premature and excessive.

The limits of the study matter too. The scenarios and apps are simulated, and most of the settings are personal life. The authors say the results may not transfer directly to long-term real use. The paper also reports nothing in Indonesian. Read these numbers as a warning about a type of failure, and run your own tests in your own language and systems.

What to do next

• Start with alerts only. Let the AI notify people for a few weeks and record which alerts were useful.
• Write down what the AI may do without approval, including spending limits. Purchases and messages to outsiders should need a person at first.
• Judge any assistant by its rejected actions as well as its accepted ones. Ask vendors how often their product stayed quiet when it should have.
• Give it access to the right records before you judge it. An assistant that can't read your policies or past files is more likely to guess.
• Ask staff to say whether each alert came at the right time and said enough. Those answers will tell you what to fix.