> ## Content Index
> Fetch the complete content index at: https://zipmex.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Mix or tune? Apple researchers say adding English beat hyperparameter tuning for Arabic pre-training
- URL: https://zipmex.com/blog/mix-or-tune-apple-researchers-say-adding-english-beat-hyperparameter-tuning-for-arabic-pre-training/
- Published: 2026-10-13T17:25:30.000Z
- Updated: 2026-10-14T13:09:07.000Z
- Description: Apple researchers report that mixing English data into Arabic pre-training beat aggressive hyperparameter tuning. What they found, and the limits.
- Author: ZIPMEX
- Tags: AI, #zipmex-newsroom

Apple Machine Learning Research on Oct. 13 [published a study](https://machinelearning.apple.com/research/mix-dont-tune?ref=zipmex.com) on language models short on training text. It tests two fixes: aggressive hyperparameter tuning, or mixing in data from a high-resource language. The paper, titled "Mix, Don't Tune," uses Arabic as the target language and English as the helper. Its authors report that mixing in English beat tuning on validation loss. Mixing also beat tuning on downstream task accuracy. The gap grew with model size.

**Key facts from the study**

- **Target and helper:** Arabic as the low-resource target, English as the high-resource auxiliary.
- **Model sizes:** Four scales, from 150M to 1.43B parameters.
- **Scale of testing:** About 1,000 pre-training runs.
- **Main result:** Mixing gave larger improvements than tuning on validation loss and downstream task accuracy.
- **Gain on validation loss:** Equivalent to 2 to 3 times the unique target data.
- **Gain on task accuracy:** Equivalent to 2 to 13 times the unique target data.

Arabic pre-training: mixing vs tuning/ KEY FACTSArabic pre-training: mixing vs tuningSTUDY SCOPEAuthors compared the levers at four model scales, 150M to1.43B parameters, over approximately 1000 runs.REPORTED GAINAuthors say mixing's gain equals 2–3× the unique targetdata on validation loss, 2–13× on task accuracy.MIXING VS TUNINGAuthors report mixing beats hyperparameter tuning, withthe gap growing with model size.Source: Apple Machine Learning Research, Oct 13, 2026

![Diagram titled Arabic pre-training: mixing vs tuning, with three facts. Study scope: four model scales from 150M to 1.43B parameters, about 1000 runs. Reported gain: mixing equals 2–3× the unique target data on validation loss and 2–13× on task accuracy.](https://storage.ghost.io/c/d4/f3/d4f34881-0300-4bbe-b8df-78a26d28be4d/content/images/2026/10/zipmex-240e5e2edfae362d.png)

Original ZIPMEX key-facts diagram: three facts reported by Apple Machine Learning Research on mixing versus hyperparameter tuning for Arabic pre-training. Figures are the authors' reported findings. data: Apple Machine Learning Research (Jeha et al.), 2026-10-13 · ZIPMEX key-facts diagram; data: machinelearning.apple.com

## Why most languages need a fix

For most of the world's languages, the authors write, pre-training has too little text. Models must repeat their training data many times. That repetition degrades generalization. One remedy is aggressive hyperparameter tuning, such as high weight decay. The other remedy mixes in data from a high-resource language to help the low-resource target.

![Screenshot of the machinelearning.apple.com page with the cited passage highlighted](https://storage.ghost.io/c/d4/f3/d4f34881-0300-4bbe-b8df-78a26d28be4d/content/images/2026/10/zipmex-shot-m-163e6391b796700c.jpg)

Screenshot of [machinelearning.apple.com](https://machinelearning.apple.com/research/mix-dont-tune?ref=zipmex.com), captured Oct 13, 2026\. Highlighted: the passage this section cites.

## Mixing beat tuning, and the lead grew with size

The authors report that the gap widened as models got bigger. The gain from mixing scaled steeply with model size.

## How big the authors say the gain is

The authors put the benefit of mixing in terms of extra data. On validation loss, the boost is equivalent to 2 to 3 times the unique target data. On downstream task accuracy, the equivalent range is 2 to 13 times. Each figure belongs to its own measure. The narrower range describes validation loss only. The wider range describes task accuracy only.

## Mixing vs tuning for Arabic pre-training, as the authors report

How did mixing English data and aggressive hyperparameter tuning compare on each measure the authors report?

| Measure                              | Mixing in English data                                         | Aggressive hyperparameter tuning        | Source                                                                                                                           |
| ------------------------------------ | -------------------------------------------------------------- | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| Validation loss                      | Larger improvement; equivalent to 2–3× the unique target data  | Smaller improvement than mixing         | Source: [Apple Machine Learning Research, Oct 13, 2026](https://machinelearning.apple.com/research/mix-dont-tune?ref=zipmex.com) |
| Downstream task accuracy             | Larger improvement; equivalent to 2–13× the unique target data | Smaller improvement than mixing         | Source: [Apple Machine Learning Research, Oct 13, 2026](https://machinelearning.apple.com/research/mix-dont-tune?ref=zipmex.com) |
| Model size, 150M to 1.43B parameters | Gain scales steeply with model size                            | Gap behind mixing grows with model size | Source: [Apple Machine Learning Research, Oct 13, 2026](https://machinelearning.apple.com/research/mix-dont-tune?ref=zipmex.com) |

Cells restate the authors' summary findings; per-size and per-task scores are not given in that summary.

## Analysis: why the gain is counted in extra data

A data-constrained language lacks unique target data. Stating the gain from mixing as an amount of that data uses the units a team is short of. That makes the 2 to 3 and 2 to 13 times figures easy to weigh. A change in loss alone would be harder to read.

## Analysis: why the size trend matters

Within the tested range, bigger models showed a bigger lead for mixing over tuning. That suggests the choice of remedy matters more as models grow, at least up to 1.43B parameters. Teams planning bigger runs in a language with little text face exactly this choice. For them, the mixing ratio looks like the first setting to get right.

## Why validation loss may understate mixing

The authors argue that target-language validation loss systematically underestimates the value of mixing. Mixing does two things. It regularizes the model by diversifying the training signal. It also contributes knowledge the repeated Arabic corpus cannot supply. Validation loss captures only the first effect.

## Analysis: a possible reason the accuracy range is wider

The authors argue that validation loss captures only the regularizing effect of mixing. Extending that point to task accuracy goes beyond what they state. If validation loss misses the knowledge that mixing adds, another measure may pick up more of it. Downstream task accuracy could be such a measure. That would fit the wider 2 to 13 times range reported for accuracy.

## What the authors recommend

**The authors' recommendations**

- **First:** Mix in a high-resource language.
- **Second:** Prioritize the mixing ratio over hyperparameter tuning.
- **Third:** Transfer hyperparameters from a small proxy model via μP.

## Analysis: the title is narrower than the advice

The title says "Mix, Don't Tune," but tuning stays in the recipe. The advice is an order of priorities, not a ban. The mixing ratio comes first. Hyperparameters still matter, and the recommendation is to transfer them from a small proxy model. The main finding, that mixing gave larger improvements than tuning, supports putting the ratio first.

## What the study leaves open

The published summary of the study gives no methods or result tables. It does not list the mixing ratios tested. It does not explain how the 2 to 3 and 2 to 13 times figures were calculated. The results cover one language pair, Arabic with English, and models up to 1.43B parameters. The full paper, which this summary does not include, is where to check how the 2 to 3 and 2 to 13 times figures were calculated.

Sources: [Apple Machine Learning Research, Oct 13, 2026](https://machinelearning.apple.com/research/mix-dont-tune?ref=zipmex.com)

Promotion · ZIPMEX

### Trade **Apple** perpetuals on ZIPMEX

135 marketsCrypto · stocks · gold · oil · FXPTS on every tradePTS → ZIPMEX tokens

[Trade AAPL-PERP](https://trade.zipmex.com/perps/PERP%5FAAPL%5FUSDC?utm%5Fsource=zipmex-news&utm%5Fcampaign=perps&utm%5Fmedium=article&utm%5Fcontent=story-189817)

long or short · 24/7

[See all markets](https://trade.zipmex.com/perps/markets?utm%5Fsource=zipmex-news&utm%5Fcampaign=perps&utm%5Fmedium=article&utm%5Fcontent=story-189817)

135 markets on trade.zipmex.com

Perpetual futures are leveraged derivatives. Prices can move fast and you can lose all of your margin. Not available in every jurisdiction. Not investment advice.

ZIPMEX promotes trading on trade.zipmex.com. Promotions are labelled and kept separate from news coverage, which is selected and checked without regard to them. Perpetual futures are leveraged derivatives. Prices can move fast and you can lose all of your margin. Not available in every jurisdiction. Not investment advice.