A smaller vocabulary cut an AI grader's running cost 77%, and its grading improved

A team at Dharma-AI has shown that changing nothing but a model's vocabulary can cut its running cost by 77%, and make its answers better. They published early results on Hugging Face on October 21 from a paper called VocabX, which is still under review, so the numbers may change.

It matters to any company that runs a small model trained for one job.

What they changed

Every language model reads and writes text in tokens, and its tokenizer decides how text is split into them. Open models such as Qwen and Gemma come with tokenizers built for many languages and many kinds of text. That makes sense for a general model. It is wasteful for one that does a single job.

The team runs an automated grader for Portuguese essays, built on Qwen3-4B. Its tokenizer held 151,669 entries, most of which the grader never uses, and it split the text the grader writes most often into many small pieces. The JSON wrapper that starts every response, for example, took 12 tokens in the original tokenizer.

They rebuilt the vocabulary for the task, keeping the same base model, data and training recipe. Repeated spans like that JSON wrapper became a single token. The vocabulary shrank from about 152,000 entries to about 65,000.

Why output tokens are the expensive ones

The paper's key point is that not every token costs the same. Reading input is fast, because the model processes it in one pass. Writing output is slow, because the model generates one token at a time and waits on memory for each one. In the grader, a generated token cost 7.4 times as much as a token read.

So the team chose new vocabulary entries by the cost they save, not by how often they appear. That shifted the savings toward the model's responses.

The results

On the same machine, with the same 500 test requests, the average response fell from 350 tokens to 101. Measured inference cost fell 77.0%. Quality, on a score that combines grading accuracy and screening of off-topic essays, rose from 0.703 to 0.810.

An early version merged each competency's grade into the surrounding JSON as one token. It was the most compact option, but quality fell to 0.689. Giving the grade its own token cost 2 extra tokens per response and brought quality back up. The tokens that carry the model's actual decision should stay separate.

Across 5 public domains, including contracts and Python code, and 3 model families, the method compressed text more than the strongest existing method in all 15 combinations, by 1.2 to 9.3 points.

What it means for companies in Indonesia

Many Indonesian AI projects have this shape: a single language and a fixed output format. A model that extracts invoice fields or classifies support tickets in Bahasa Indonesia uses a small slice of a vocabulary built for the whole world.

The study does not measure Indonesian, so don't assume the same 77%. But the logic carries over. If your model writes the same structure on every request, and runs on a general-purpose tokenizer, you are likely paying for tokens you don't need. For a model serving millions of requests a month, that difference shows up directly in the GPU bill.

What to do next

For any small model you run in production, measure how many tokens a typical response takes, and look at what those tokens are. Repeated wrappers, field names and fixed phrases are the first place to look. Ask your team or vendor whether the tokenizer has been adapted to the task, or simply inherited from the base model. And when you compare model costs, weight output tokens more heavily than input tokens, because a generated token can cost several times as much as one read: 7.4 times in the grader.