AI can now fix 2 in 3 security flaws in real software on a single try

The short version

• Collinear AI, the company behind a public security test for AI, released 100 tasks that ask AI to find and fix security flaws in real open-source software.
• Google's leading model fixed 67.8% of the flaws on a single try, on average. Anthropic's was just behind at 67.3%.
• A much cheaper model fixed 57.0% at $0.04 an attempt, so check the price as well as the rank.

What the test asks

Each task gives the AI a copy of a real open-source software project and a short instruction. The AI must find a security weakness and fix it. The fix must not break anything that already worked.

Software marks every answer automatically, so anyone who reruns the test gets the same score. The 100 tasks cover 91 different kinds of software weakness. The makers also keep a private set of tasks. That shows how models do on tasks they cannot have seen before.

People reviewed every task by hand. The makers also made sure a model can't pass by copying a published fix.

What the results show

Eleven models took the test, and the top two are very close. Google's leading model fixed 67.8% of tasks on a single try. Given four tries, it fixed 81.0%. Anthropic's leading model scored 67.3% on a single try and 78.0% with four.

Some flaws are still hard. Between them, the models solved 95 of the 100 tasks at least once. Even so, the top model failed 19% of the tasks every time it tried.

The models differ most on cost. A model from DeepSeek, another AI company, fixed 57.0% on a single try at $0.04 an attempt. Anthropic's model cost $1.83 an attempt for about ten points more. Google's cost $3.92 an attempt for a small extra gain.

That spread is large. The most expensive option costs nearly 100 times the cheapest for each attempt, for a gain of about ten points.

What it means for your company

On real code, the best models now fix about two-thirds of the problems on a single try.

The other third still matters, because a defender has to get it right every single time, as the test's makers point out. Treat AI as a fast first pass. Have a person check each fix before it goes live.

The price gap matters for Indonesian companies with large amounts of older code. Running thousands of fixes at $0.04 an attempt is a very different budget from $3.92 an attempt. A sensible mix may be a cheap model for the easy majority and a stronger one for the hard cases.

The test uses open-source projects, which may look quite different from your own systems. Results on your code could be better or worse. The only way to know is to try it on a system you understand well.

A cheap model may also send your code to servers abroad. Check where it runs before you use it on sensitive work. For a bank or a telco, that answer may decide the choice before price does.

What to do next

• Ask your security team or software supplier how they find and fix flaws today, and how long a fix takes.
• Run a small trial on one system that isn't critical. Let the AI propose fixes and have engineers approve each one.
• Compare a cheap model and a top model on the same flaws before you commit to either.
• Check where each model runs and who can see your code.
• Your technical team can run the public test on any model you are considering, once they accept its terms of use.