When a language has too little training text, mixing in English did more for Arabic language models than hyperparameter tuning did. That is the finding of "Mix, Don't Tune," a paper Apple Machine Learning Research listed in October 2026 (first posted to arXiv on 13 May 2026). Across about 1,000 pre-training runs at four sizes from 150M to 1.43B parameters, the authors found that mixing beat tuning on Arabic validation loss and on downstream accuracy, and the lead grew with model size. At 1.43B parameters, mixing cut Arabic validation loss by 12.9% against a monolingual baseline. Tuning alone cut it by 3.1%. Everything below comes from one paper and has not been independently replicated.
The study at a glance
- Paper: "Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings," arXiv:2605.13225 (v1, 13 May 2026, cs.LG, CC BY 4.0).
- Authors: Paul Jeha (DTU; work done while at Apple), Anastasiia Sedova, Louis Béthune, Skyler Seto, Jes Frellsen (DTU), Pierre Ablin and Natalie Schluter. Frellsen, Ablin and Schluter are listed as equal senior authors.
- Target and helper languages: Arabic is the low-resource target. English is the high-resource auxiliary.
- Models: LLaMA-style decoder-only transformers at 150M, 380M, 600M and 1.43B parameters. Sequence length 2048, XGLM tokenizer with a 256K vocabulary.
- Scale of testing: about 1,000 models trained in total.
The problem being solved
Most languages lack enough text for modern pre-training budgets. Models end up repeating the same data many times, and the authors note that this repetition hurts generalization. The paper compares two remedies:
- Aggressive hyperparameter tuning, for example high weight decay, to make repeated data go further.
- Data mixing, where text from a high-resource language fills the rest of the budget.
How the experiment was set up
Data
- Arabic: a subset of FineWeb2 (about 55B Arabic tokens), cut down to 200M unique tokens, the scarce resource in the experiment.
- English: FineWeb, about 350B tokens, which the authors treat as "effectively unlimited" at these scales.
- Total token budget: 100 times the non-embedding parameter count. That works out to 2.3B tokens at 150M, 12.2B at 380M, 28.5B at 600M and 97.3B at 1.43B.
- Monolingual baselines: Arabic-only corpora of 25M, 50M, 100M, 200M, 500M, 1B and 2B tokens.
How mixing worked
Each training sample is Arabic with probability α and English otherwise. The ratio is set by how many times the 200M Arabic tokens may be repeated (R_max ∈ {1, 2, 5, 10, 20, 50, 100}). Monolingual runs repeat Arabic up to 100 times.
How tuning worked
- AdamW (β1 = 0.9, β2 = 0.95), gradient clipping at 1.0, bf16, constant learning rate with no warmup or decay.
- Grid: weight decay λ ∈ {0.01, 0.1, 0.3, 1.0, 3.0} × learning rate η ∈ {1e-4, 3e-4, 1e-3, 3e-3, 1e-2}, run at every scale.
- "Basic" hyperparameters use μP transfer from the 150M model (base pair λ = 0.1, η = 0.01). For hidden layers the learning rate is divided by the width multiplier and weight decay is multiplied by it. At 1.43B this gives λ = 0.4 and η = 0.0025.
Evaluation
- Main metric: Arabic validation loss on a held-out FineWeb2 split.
- Downstream: ARC Easy, 5-shot, in English and in Arabic. The Arabic version was machine-translated with NLLB-200-3.3B. MMLU was left out because the models are too small for it.
Should you mix or tune for your low-resource model?
Answer three questions about your pre-training run to see what this one paper's results suggest for you.
What size of model are you pre-training in a language with little text?
- About 150M parameters or smaller → Mixing is unlikely to help at this size
- Between 380M and 1.43B parameters → next question
- Larger than 1.43B parameters → Outside what the paper tested
Can you fill the rest of your token budget with a high-resource language, the way the authors used English for Arabic?
- Yes, I have a high-resource language to mix in → next question
- No, I only have target-language text → The paper's main remedy does not apply
How will you judge whether mixing worked?
- Target-language validation loss only → Loss alone may understate mixing
- Downstream tasks as well as loss → Set the mixing ratio first, then transfer settings
At 150M parameters every bilingual run picked α = 1, meaning Arabic only. In the authors' words, mixing "does not help at this scale."
The study covers models from 150M to 1.43B parameters only. The lead of mixing grew with size up to 1.43B, but the paper has no results for larger models.
The study compares tuning with mixing in a high-resource language. Without one, only tuning is left, and tuning alone cut Arabic validation loss by 3.1% at 1.43B, against 12.9% for mixing.
The authors say validation loss captures mainly the regularizing effect of mixing, not the added knowledge. On Arabic ARC Easy at 1.43B the multiplier was 8.1x, against 3.2x on validation loss. They advise judging on downstream tasks too.
The authors recommend mixing in a high-resource language, sweeping the mixing ratio first and transferring hyperparameters from a small proxy with μP. The best Arabic share fell from 0.33 at 380M to 0.10 at 1.43B, and μP came within 0.5% of the per-scale best loss.
This follows one unreplicated study of a single language pair (Arabic and English), single-seed runs and one benchmark; it cannot tell you the right mixing ratio for your own languages or model.
What the results show
Mixing vs tuning on validation loss
| Model size | Gain from mixing | Gain from tuning alone |
|---|---|---|
| 380M | 5.7% | 1.2% |
| 600M | 8.7% | 0.9% |
| 1.43B | 12.9% | 3.1% |
Improvement in Arabic validation loss over the monolingual baseline, as reported in the paper.
Two more results support the headline:
- Tuning hyperparameters on top of mixing improved Arabic validation loss by less than 1% at every scale.
- A bilingual 600M model with plain μP hyperparameters (validation loss 2.870) beat a monolingual 1.43B model with per-scale tuned hyperparameters (2.995). The bilingual model is about 2.4 times smaller.
Smaller is different
At 150M parameters, every bilingual run picked α = 1, meaning Arabic only. In the authors' words, mixing "does not help at this scale." The best Arabic share fell as models grew: 0.33 at 380M, 0.14 at 600M and 0.10 at 1.43B with tuned hyperparameters. Larger models did best with mostly English in the mix.
The "2–3x" and "2–13x" figures
The authors express the benefit of mixing as an equivalent amount of extra Arabic data. They fit a log-linear curve to the monolingual baselines, then ask how much unique Arabic text a monolingual model would need to match the bilingual one.
- Validation loss, same hyperparameters: 2.1x (380M), 2.6x (600M), 3.2x (1.43B).
- Arabic ARC Easy, same hyperparameters: 2.0x, 3.1x, 8.1x.
- With hyperparameter tuning on top of mixing: the benchmark multiplier reaches 13.4x at 1.43B, equal to about 2.7B unique Arabic tokens.
The abstract's ranges cover both conditions. The top of the 2–13x range needs tuning on top of mixing. The paper also says that under a different checkpoint-selection method the 1.43B figure rises to 47.4x, and it treats 13.4x as the lower bound.
Why validation loss understates mixing
The authors argue that mixing does two things. It regularizes the model by diversifying the training signal, and it adds knowledge the repeated Arabic corpus cannot supply. Arabic validation loss captures mainly the first effect. That is why the accuracy multipliers run higher than the loss multipliers, and why the authors advise against comparing monolingual and bilingual strategies on validation loss alone.
What the authors recommend
- Mix in a high-resource language.
- Set the mixing ratio first. The ratio dominated the variance in Arabic validation loss in most of the authors' views of the results.
- Transfer hyperparameters from a small proxy model with μP rather than running a full search at every scale. The μP settings came within 0.1% (380M) and 0.5% (600M and 1.43B) of the per-scale best bilingual validation loss.
The title says "Don't Tune," but the advice is really about order of priority. Hyperparameters still matter. At 1.43B the paper finds the loss landscape around the μP point is sharper, and hyperparameter factors explain roughly half the variance in the smallest neighbourhood tested.
For teams training in low-resource languages
- Below a few hundred million parameters, these results give no reason to expect mixing to help. At 150M it did not.
- At larger scales, a sweep over the mixing ratio is likely to pay off more than a wide hyperparameter grid, at least for a pair like Arabic and English.
- Judge mixing on downstream tasks, not only on target-language loss.
Limitations
The paper lists its own: one benchmark (ARC Easy), a machine-translated Arabic test set, one language pair, static mixing ratios only, single-seed runs, and a fixed hyperparameter grid whose optimum often lands on its edges. All of it is a single study of models up to 1.43B parameters.
