Notion's AI supplier kept chats it promised to delete, then got hacked

The short version

• Notion says a security incident at Fireworks AI, one of its AI suppliers, affected fewer than 1% of Notion workspaces.
• The exposed data was Notion AI chat transcripts. Fireworks had promised Notion it would keep none of it. Notion found it had.
• Notion cut its systems off from Fireworks on 3 November. It tells customers to change any password or key pasted into Notion AI.

What happened

Fireworks AI runs AI models for other companies. On 9 October it reported that an outsider had got into some customer accounts. A later update showed the damage was wider. The attacker reached account and billing details. For some customers, it also reached stored keys and training data, along with unfinished AI models. Fireworks says it does not store payment card numbers, so those were safe.

Fireworks links the attack to a criminal group that steals access to AI models and resells it. On 24 October the group got in again through a cloud provider that hosted Fireworks' research machines. Fireworks says it spotted that the same day.

Notion used Fireworks in two ways. It sent stored AI chats there for the analysis behind Insights, a feature that shows admins how staff use AI. It also used Fireworks to run some free-to-download AI models inside Notion. In the past, Notion also used chats that customers had volunteered for testing to train small models there.

The promise that didn't hold

Notion offers enterprise customers a commitment called zero data retention. Under it, model suppliers should not keep the requests and answers that pass through them. Fireworks gave Notion that commitment in its contract. During the investigation, Notion learned that data had been kept anyway.

The exposed chats hold whatever people typed into Notion AI, and what it answered. Meeting notes were not affected unless someone pasted them into a chat. For most affected workspaces, Notion says the transcripts carried nothing that tied them to a person or company. The text itself may still contain sensitive details. Notion recommends treating the data as exposed.

Notion has switched off Insights and stopped using Fireworks. It has deleted its data there and halted all training runs. It is now auditing its critical suppliers, such as AI model hosts and data processors, to check they keep their data promises.

Owners of affected workspaces can download the exposed text and a list of affected members. Members can see their own exposed chats from the same help menu.

What it means for you

Every AI tool your staff use may pass their words to other companies behind the scenes. A promise in a contract only protects you if the supplier keeps it. Here, a no-keep promise failed quietly, and a hack later turned it into a leak.

Staff may paste passwords, draft deals or customer details into AI chats. Those words can travel further than the person typing expects.

The lesson reaches beyond Notion. Most business software now adds AI features, and many of them rely on outside model hosts. Each one is another company holding your staff's words.

What to do

• If you use Notion AI, ask your workspace owner to open the security incident review page and change any passwords or keys found there.
• Ask each AI vendor which other companies process your data, and how it checks they keep their promises.
• Tell staff never to paste passwords or access keys into any AI chat.
• Add your AI suppliers' own suppliers to your vendor risk reviews.