44% of famous findings about AI's flaws no longer hold, Stanford finds

The short version

• Stanford researchers retested 50 well-known findings about how AI models behave, using 20 models released between July 2024 and September 2026.
• 44% of the findings no longer hold, and another 34% have weakened. On average a finding that faded lasted 15.7 months.
• The weaknesses that remain are mostly about judgment. Flattering the user is one. Check those first, then recheck the rest.

What the researchers did

Company AI policies often rest on findings from research papers. "AI makes up citations." "AI can't count letters." "AI gives in when you push back." Many of those papers tested models that are now two or three years old.

Owen Queen and James Zou at Stanford took 50 widely cited findings and ran each original test again. They used 20 models from eight developers, from small free ones to large paid ones. Then they tracked how each finding changed as newer models came out.

Each finding got one of four labels. Expired means the effect has mostly gone. Weakened means it is still there but smaller. The other two are persists and strengthened.

What they found

Of the 50 findings, 22 have expired. Another 17 have weakened. Only 8 still hold, and 3 have grown stronger.

The faded ones are mostly simple mistakes that newer models no longer make. These include counting letters, missing facts buried in long documents, and inventing sources.

The citation test shows how far things moved. In 2023, a study found ChatGPT invented 55% of the sources it cited. In the Stanford rerun, OpenAI's latest model invented none. Anthropic's Claude invented about 1%.

That doesn't mean every model is safe. A model from Alibaba's Qwen family still invented 69% of its references in the same test. A finding can fade across the market while one model still has the problem.

Timing matters too. The researchers estimate 16% of findings had expired a year after publication, and 39% after two years.

What hasn't gone away

The findings that persist are mostly about judgment. Models still tend to flatter users and agree with them. When AI is used to grade answers, it still favors its own work. Models still answer when they should say they don't know.

Two other problems have weakened but not gone. A basic "prompt injection" now has about a quarter of its original effect. That is an attack where hidden instructions in a document or web page take over the AI. Errors on legal questions are still close to three-quarters of their original level.

What it means for companies in Indonesia

Many Indonesian firms wrote their AI rules in 2023 or 2024. Some banned uses because of problems that newer models have largely fixed. Others trust AI in areas where the old warnings still apply.

The Stanford work suggests a better habit. Treat what you know about AI as having a shelf life of about a year. Test the specific model you use, on your own tasks, before you relax a rule.

It also points to where human review still pays off. If AI scores sales calls or checks other AI's work, its taste for flattery and for its own answers can skew results. Keep a person in those loops.

What to do next

• List the AI risks your policy is built on. Note when each was found, and on which model.
• Retest the ones older than a year on the model you actually use today.
• Test each model separately, since one can still fail where most have improved.
• Keep people reviewing any AI that grades or approves things, where flattery and self-preference still show up.
• Keep defenses against hidden instructions in documents and web pages, which have weakened but not gone.
• Put a yearly review of your AI rules in the calendar.