Training AI on harmless data can still make it easier to misuse
A new study traced the problem to examples that look fine to a human reader, then removed them. If you customize AI models with your own data, test their safety again afterward.
The short version
- Customizing an AI model with new examples can weaken its safety, even when nothing in those examples is harmful.
- DatologyAI found the risky examples by measuring how they change the model, then removed 20% of a popular collection.
- People trying to trick the models into unsafe answers succeeded up to 22% less often. They still got through about a third of the time.
What the researchers found
Many companies customize a ready-made AI model by feeding it their own examples. It is how a general model learns your products, your tone or your language.
Earlier research showed the catch. Customizing can make a model less safe even when the examples are ordinary. A person reading them would see nothing wrong, and an AI reviewer often misses them too.
DatologyAI, a company that sells data services for AI, tried a different test. Instead of judging what each example says, it measured how each one changes the model's behavior. It then looked for examples with the same effect as clearly unsafe answers.
The team checked all 939,343 examples in a popular collection called Tulu 3. It removed the 20% that most resembled the unsafe answers, then used the rest to customize 3 small models.
How much it helped
The team measured how often a standard automated attack could trick each model into an unsafe answer. Lower is better.
- Qwen: from 47.7% to 37.0%.
- Llama: from 38.0% to 30.3%.
- Olmo: from 48.3% to 41.3%.
The usual approach did far worse. An AI judge reviewed the content and removed its worst 20%. The attack rate for Qwen barely moved, at 47.3%. For Llama it got slightly worse, at 39.0%. That fits the main idea: the harm isn't visible in the words.
Ability mostly held up across 6 tests covering areas such as math, coding and medicine. Qwen matched or beat its earlier scores on all of them. Llama and Olmo showed some minor drops.
What is new
Tracing a model's behavior back to its examples used to be a small lab exercise, because the computing cost was so high. A faster method makes it possible to screen a large collection before you use it.
Simple coding tasks were removed at 1.74 times the normal rate. Examples showing software errors were also more likely to go. One possible explanation is that quick, obedient answers make a model more likely to obey harmful requests. The authors stress that this is only a suggestion and they haven't proven the cause.
What it means for your company
Say you customize a model for customer service in Bahasa Indonesia, using your own chat logs and manuals. Your review finds nothing offensive in the data, and the new model answers well. This research says that may tell you little about whether it still refuses dangerous or abusive requests.
Keep the size of the result in perspective. The tests used models far smaller than leading commercial ones. Attacks also still worked 30.3% to 41.3% of the time after filtering. Filtering lowers the risk, and you still need other protections around the model.
The method also comes from a company with services to sell, though it published its results for others to check.
What to do next
- Treat every customized model as a new product. Run your safety tests again before it goes live, even after a small update.
- Ask any vendor who customizes a model for you to show safety test results from before and after.
- Don't accept "we reviewed the data" as proof of safety. Ask how they tested the finished model.
- Keep a record of which data went into each version, so you can trace a problem back later.
- Screen new batches of examples before adding them to a large collection, and test the model again afterward.
- Limit what a customized chatbot can do by itself. A model with fewer permissions does less damage when an attack works.