An AI model that tested well cost Airbnb several times more in daily use
Airbnb's engineers judge every new AI coding model by what a finished task costs. One model passed those tests on price, then cost several times more once staff changed how they worked with it.
The short version
- Airbnb tests each new AI coding model on its own engineering tasks and records two numbers: a quality score and the cost to finish one task.
- One model scored better than the previous best at about the same cost per task. In real use, weekly spend per engineer rose several times over.
- Airbnb moved model A's heaviest users to a newer, cheaper model. In their first full week, spend on their main model fell to about a seventh.

What Airbnb did
Thousands of Airbnb engineers, and people who work closely with them, use AirChat, the company's internal tool for AI coding assistants, every week. Those assistants call AI models through Amazon Bedrock, Amazon's service for running models from several providers. Airbnb's engineers described the setup in a post on Amazon's blog on 18 November.
New models arrive fast. Airbnb counted 8 new models in about 5 weeks. By mid-October, about two thirds of its engineers' AI coding requests went to those new models.
So Airbnb built a fast test. It keeps seven sets of real tasks taken from its own engineering work, each with a grader that scores the result. On the afternoon a model comes out, AI assistants run it through every set. By the next morning, Airbnb has a score and a cost per task, enough to make a decision for the whole company.
Cost per task means the full price of finishing one job. An assistant may call the model many times to finish one task, and it re-reads the work so far each time. Airbnb prices all of it.
What is actually new
The useful part is what happened after a model passed. Airbnb calls it model A.
On Airbnb's coding tasks, model A scored more than three points above the previous best. Its cost per task was about the same. Effort is a setting for how long the model thinks before it answers. Airbnb rolled model A out and asked people to use the lower effort settings for most work.
A few weeks later, the bill told a different story. Average weekly spend per user was several times higher than on the old model. Each request cost about three times as much, and the average user sent about three times as many.
The cause was behavior. Model A could work alone for a long time, so engineers started long, self-directed sessions. Most of the cost went on the model re-reading that long history. Its written output was only about a tenth of each request's cost.
Airbnb's own summary: the tests measured the model on a task, while real use measured how people chose to work with it.
The fix was simple because the tests existed. A newer model scored higher than model A at about a sixth of its cost per task. Airbnb switched model A's users with one setting change. Their requests stayed about the same, and their spend on their main model fell to about a seventh.
Settings matter too. On one model, moving from low to high effort raised the score by 0.8 points. The cost per task was 2.5 times as high. Airbnb now caps effort where the extra cost buys little.
What it means for companies in Indonesia
Many teams pick an AI model from the maker's launch charts and a price per word. Airbnb's experience shows both can mislead. A model that is cheap per word can still cost more per finished job, and real use can differ from any test.
The fix takes discipline more than money. Keep a set of real tasks from your own work. Price the whole task. Then watch the bill after rollout, because people change how they work when a tool gets better.
This applies well beyond coding. Customer service assistants and report writers can run up costs the same way.
What to do
- Build a small test set from your own past work, with a clear way to score each result.
- Record the cost of finishing each task, including failed attempts, next to the quality score.
- After any rollout, compare spend per user and requests per user with what the tests predicted.
- Set a default model and cap the most expensive settings, then let staff opt up when they need to.
- Keep at least one alternative model ready, so you can switch when a cheaper one matches the score.
