Databricks' AI got 91% of questions right by reading staff notes
In a test Databricks built, its data assistant beat two leading AI tools because it could read the notes and reports staff had already written. Your team's working files may matter as much as which AI you pick.
The short version
- Databricks tested its data assistant, Genie, against 2 leading AI tools from other labs. The questions were about a made-up energy company's business.
- Genie answered 91% correctly. The other two scored 45% and 58%. They also did more than twice the computing work and took 3 to 4 times longer.
- The difference was context. Genie could read the notes and dashboards staff had already written, not only the raw tables.
What Databricks tested
Databricks built the test with an outside partner. It invented a UK electricity and gas supplier called Calderwick Energy. The company has hundreds of tables, and the largest holds about 120 million rows of meter readings.
Like a real company, it is messy. There are copied notebooks (the files where analysts write up their work). There are abandoned analyses and old documents too. Some dashboards haven't been touched in months.
Each AI got the same questions. Each ran 5 times per question in the same live workspace. Databricks did not name the two rivals. It calls them Agent A and Agent B, from leading AI labs.
Most earlier tests of this kind give the AI only tables. On those tests, Databricks says, the tools score about the same. This one also hides the answers in the places real companies keep them: in saved queries and written notes.
Why the right answer was hidden in a notebook
One example shows the problem. A question asked for the planning rate the commercial team set for FY24/25.
The obvious table has a column for that rate. For FY24/25, it says 0.31 pence per kWh. An AI that reads the table answers 0.31, and that is wrong.
The notebook that builds the table explains why. The table copies the latest year's assumptions back onto earlier years. Further down, the same notebook keeps the real history, and it says 0.52.
Genie found that note and answered 0.52 in all 5 runs. It also showed which notebook the answer came from, so a person could check it. Neither rival got it right. One answered 0.31 every time.
Most company data works this way. Tables hold the numbers. The notes and old queries your analysts wrote explain what the numbers mean.
What is new this week
Databricks made Genie Ontology available to all its customers. It reads a company's notebooks and dashboards, and organizes what they say about the business.
Databricks also released add-ons that let ChatGPT and Claude use Genie. They can ask Genie for the company's agreed definitions. Staff can keep using the assistant they like, and the numbers still match the official ones.
Read the result with care. Databricks designed the test and entered its own product. It also says the gap may be smaller in companies that have little written down yet, or when the tables alone hold the answer.
What it means for you
Many companies judge AI tools by which model is smartest. This test suggests a bigger factor: whether the tool can find what your team already knows.
In most companies, that knowledge is scattered. The revenue definition sits in one analyst's notebook. Reporting rules sit inside a dashboard. Exceptions sit in a saved query someone wrote years ago.
If an AI can't see those, it gives a confident answer from the table. That answer can be wrong in exactly the way the 0.31 answer was.
What to do next
- Before you buy an AI data assistant, test it on your own questions with known answers. Include questions where the table alone gives the wrong answer.
- Write down your key definitions, like revenue and active customer, in one agreed place.
- Clear out copied and abandoned reports. They confuse people and AI alike.
- Ask every vendor how its tool finds the source of each answer, and whether it shows you that source.