On Oct. 21, 2026, Dharma-AI said that a specialised tokenizer called VocabX cut the measured inference cost of its Portuguese essay grader by 77.0%. The grader runs on Qwen3-4B, and the base model was not changed. The grader's bench score also went up, from 0.703 to 0.810. The figure covers one narrow job: one language, one output format, 500 requests on one RTX 5090 GPU running vLLM. It does not show that a general-purpose model would get a similar cut. Every number comes from Dharma-AI's own blog post, and the paper behind it is still under review. No independent replication has been published.
What Dharma-AI reported (Oct. 21, 2026)
- Model and task: an automated grader for Portuguese essays, built on Qwen3-4B. It returns a JSON assessment with a grade and a comment for each competency.
- Test: seven versions of the grader were trained on the same base model, and the tokenizer was the only difference between them. Each version ran the same 500 requests on an RTX 5090 GPU with the vLLM inference server.
- Cost measure: cost was "calculated from measured runtime". It is not a token count or an API price.
- Result: cost fell 77.0% and the bench score rose to 0.810, from 0.703 for the production grader. The average response fell from 350 to 101 tokens.
- Status: the numbers come from a paper, "VocabX: Cost-Weighted Vocabulary Adaptation for Domain-Specific Language Models". Dharma-AI says it is "still under review, and may change before publication." The post does not announce a code or model release.
Why a smaller vocabulary cuts cost
A general-purpose tokenizer is built to cover many languages and domains. Qwen3-4B's tokenizer has 151,669 entries. A grader that only reads and writes Portuguese essay assessments in JSON uses a small part of that. VocabX's vocabulary has 64,767 entries. In the grader, the output layer went from about 152,000 rows to about 65,000. Dharma-AI notes that this layer, the LM head, is almost 10% of the parameters in a model of Qwen3-4B's size.
Two effects follow:
- Fewer tokens per response. VocabX adds "macro tokens" for repeated multi-word spans, from short expressions up to whole lines and JSON structures. That is how the average response could fall from 350 to 101 tokens.
- A cheaper step per generated token. At every generation step, the model scores every row of the output layer, so fewer rows means less work on each step.
The post does not divide the 77.0% between these two effects. It gives one overall figure.
Weighting tokens by what they cost
The method weights each token it removes by the cost of the field it appears in: system prompt, user message or response. In the grader case, Dharma-AI says "a generated token costs 7.4 times a token read." It adds that this ratio varies with hardware, the problem, batch size and context length, and "can be larger than that." In practice, that weighting steers the vocabulary toward shortening what the model writes, not what it reads.
To find repeated spans, five "miners" search for long words, whole lines, expressions, full sentences, and punctuated structures such as JSON and tags. Some candidates do not appear in the sample, so the method gives them an estimated gain instead of zero. According to Dharma-AI, only 16.6% of candidates in a contracts corpus appeared in a 400-document sample.
The grade token: where compression went too far
The first VocabX version merged each competency's grade with the JSON around it into one token. It cut cost by 73.3%, but its bench score fell to 0.689, below the production grader's 0.703. Dharma-AI's explanation: "Compression has a cost when the compressed span carries the very decision the model exists to make."
The final version gives the grade its own token. That adds 2 tokens per response, and it produced both the larger cost cut and the higher score.
| Grader version | Bench score | Measured cost reduction |
|---|---|---|
| Production grader | 0.703 | baseline |
| Grade fused with JSON | 0.689 | 73.3% |
| Grade as its own token (final) | 0.810 | 77.0% |
The bench score "combines grading accuracy and the accuracy of screening off-topic or rule-breaking essays." The post gives no weights or formula for it.
Grader versions: bench score and measured cost
Sort the versions by bench score to see which grade-token change moved quality.
| Grader version | Bench score | Measured cost reduction | Source |
|---|---|---|---|
| Production grader | 0.703 | no figure given | Source: How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real… (huggingface.co), Oct 21, 2026 |
| Fused grade and JSON token | 0.689 | 73.3% | Source: How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real… (huggingface.co), Oct 21, 2026 |
| Separate grade token | 0.810 | 77.0% | Source: How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real… (huggingface.co), Oct 21, 2026 |
One 500-request run on one test machine, with every figure Dharma-AI's own and no cost figure given for the production grader.
Results beyond the grader
Dharma-AI also compared token compression across 15 combinations: 5 public domains (contracts, medical papers, scientific papers, news, Python code) and 3 model families (Qwen3, Llama-3.1, Gemma-3). It says VocabX beat the strongest reference method in all 15, by 1.2 to 9.3 points. The reference methods were fast vocabulary transfer, multi-word tokenization, and in-place BPE tokenizer expansion. On contracts, VocabX reduced tokens by 62.6% with 113,426 vocabulary entries. The best in-place BPE setup reached 61.8% with 207,803 entries. These are compression results reported by Dharma-AI, not runtime cost results.
Independent context: what a bigger output layer costs
A separate arXiv preprint shows the same effect from the opposite direction. "In-Place Tokenizer Expansion for Pre-trained LLMs" (Smith et al., submitted July 16, 2026) is one of the reference methods above. It kept the model body and inputs the same and doubled the embedding and LM-head matrix from 65K to 128K rows. Decode throughput fell:
- 8.6% on the M4 Max CPU
- 7.4% on the M4 Max GPU
- 9.4% on the Snapdragon 8 Elite Gen 5 phone class
Prefill throughput stayed within run-to-run variance. The authors explain that decode "is memory-bandwidth-bound and pays the larger LM-head read on every step." The preprint does not test VocabX. What it does confirm is that output-layer size has a measurable cost on every generated token, and that is one of the two effects VocabX relies on.
When this approach fits
From what the post describes, vocabulary specialisation makes the most sense when these conditions hold:
- Narrow, stable workload: one language and one output format, with repeated structure such as JSON fields.
- Output-heavy cost: generated tokens cost several times more than input tokens on your hardware.
- Retraining is possible: Dharma-AI trained each grader version. A tokenizer swap is not a drop-in change.
- The decision token stays separate: keep the token that carries the model's actual answer out of any merged span, and evaluate quality next to cost, not cost alone.
A general-purpose or multilingual model needs the broad vocabulary that VocabX removes. That is why the post says the general-purpose choice "makes sense" for those models.
What to watch next
The next step is the full VocabX paper, with its tables and the interventions Dharma-AI discarded. The 77.0% figure should be treated as provisional until the paper is published and someone reproduces the result on other hardware. For a related serving topic, see vLLM steering without disabling graph replay.
Limitations: all grader figures are self-reported by Dharma-AI from one 500-request run on one GPU, and the underlying paper is unpublished and under review.
