Background: The Kaplan Era
In 2020, Kaplan et al. at OpenAI published Scaling Laws for Neural Language Models (arXiv:2001.08361), establishing that model performance scales predictably with parameters, data, and compute. Their key finding: parameters should scale faster than data. This led to a generation of very large, relatively data-hungry models — GPT-3 (175B parameters, trained on ~300B tokens) being the canonical example.
The field accepted this framework. Labs competed on parameter counts.
The Chinchilla Paper
In March 2022, Hoffmann et al. at DeepMind published Training Compute-Optimal Large Language Models (arXiv:2203.15556) — commonly called the Chinchilla paper after the 70B-parameter model trained to test the new hypothesis.
Their finding: Kaplan's laws were derived from models that were all significantly undertrained.
Kaplan et al. had fixed dataset size and varied parameters. Hoffmann et al. fixed compute budget and varied both parameters and tokens jointly. The result was a fundamentally different optimal ratio:
For a compute-optimal model, parameters and training tokens should scale in roughly equal proportion.
The rule of thumb: ~20 tokens per parameter for compute-optimal training.
Under this framework, GPT-3 (175B parameters, 300B tokens) should have been trained on ~3.5 trillion tokens to be compute-optimal. It was undertrained by roughly an order of magnitude.
What Chinchilla Demonstrated
The 70B Chinchilla model — trained on 1.4 trillion tokens — outperformed Gopher (280B parameters), GPT-3, and Megatron-Turing NLG (530B) on the majority of downstream benchmarks, despite being 4× smaller than some of them.
This was not a marginal improvement. Chinchilla beat Gopher on 10 of 16 BIG-bench tasks while using the same total compute budget.
Implications That Are Still Underappreciated
1. Most pre-2023 benchmarks are confounded
Benchmark comparisons between models trained under different data regimes are not measuring capability at the same compute point. A 175B model and a 70B model at the same FLOP budget are not comparable unless training data volume is also comparable.
Recommendation: When evaluating research claims, check the tokens-to-parameters ratio. A model claiming superiority while being undertrained relative to Chinchilla norms is not a fair comparison.
2. The emergence narrative is partially an artefact
Much of the discourse around "emergent capabilities" at scale (Wei et al., 2022) documents capabilities that appear suddenly at certain parameter thresholds. But if those models were all undertrained, the apparent threshold may reflect training data starvation, not a genuine parameter-dependent phase transition.
Arvind Narayanan and Sayash Kapoor have argued (2023) that several celebrated emergence results don't replicate under Chinchilla-optimal training conditions. This is an active area of dispute with significant implications for AI forecasting.
Recommendation: Be skeptical of emergence claims that haven't been reproduced in compute-optimal training settings.
3. The LLaMA models validated the thesis
Meta's LLaMA (Touvron et al., 2023) and LLaMA 2 were explicitly trained under Chinchilla-informed data regimes — smaller models, more tokens. LLaMA 7B outperformed GPT-3 on most benchmarks. This was not a surprise to anyone who had internalised the Chinchilla results.
Recommendation: For practitioners choosing a base model, a smaller Chinchilla-optimal model will generally outperform a larger undertrained model of similar provenance — and will be significantly cheaper to run.
4. We may be entering a data-constrained regime
Chinchilla scaling implies that compute-optimal training requires data that scales linearly with parameters. At frontier scale (hundreds of billions to trillions of parameters), this quickly exceeds the available high-quality internet text.
Hoffmann et al. acknowledged this: at some point, the limiting factor shifts from compute to data. Labs are now increasingly turning to synthetic data, code, and multi-modal training to extend the available token budget — which introduces its own quality and distribution questions.
Recommendation: The Chinchilla laws describe an optimum under data abundance assumptions. Watch for papers examining performance under data-constrained regimes — this is where the next set of scaling surprises is likely to emerge.
A Useful Mental Model
Think of parameters as capacity and tokens as curriculum. Kaplan suggested building a large classroom and filling it somewhat. Chinchilla showed that a smaller classroom with a richer curriculum produces better-educated graduates at the same cost — and that most classrooms in 2020–2022 were dramatically undercurriculum'd.
Recommended Reading
- Hoffmann et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556
- Kaplan et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361
- Touvron et al. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971
- Wei et al. (2022). Emergent Abilities of Large Language Models. arXiv:2206.07682
- Narayanan & Kapoor (2023). Revisiting the Mirage of Emergent Abilities in Large Language Models. AI Snake Oil Newsletter.
