Neurons indepth

Chinchilla

Chinchilla “law” (usually called Chinchilla scaling laws) says that for large transformer LLMs, you should not just keep increasing parameters; for a fixed training compute budget, you get the best performance when model size and number of training tokens are balanced, with roughly ~20 tokens per parameter for dense models

What Chinchilla law is

• It comes from DeepMind’s 2022 paper “Training Compute-Optimal Large Language Models” (the Chinchilla paper).deepmind+1

• They empirically studied how loss scales with: model parameters NNN, training tokens DDD, and compute budget CCC.emergentmind+1

• They found earlier practice (GPT‑3 style) used models that were too big and trained on too little data, leaving performance on the table for the same compute.geeksforgeeks+2

Core message it tries to convey

• For compute‑optimal training, you should scale parameters and data together, not just parameters.

• There is an approximately constant optimal ratio of training tokens to parameters; Chinchilla found about 20 training tokens per parameter (e.g., 70B params → ~1.4T tokens)

• Given a fixed FLOP budget CCC, both optimal NNN and DDD grow as power laws of CCC, and you minimize loss when neither the model is undertrained (too few tokens) nor too small (too few params).

Interpretation: if you have compute for a “GPT‑3‑sized” run, you’re usually better off training a smaller model for more tokens rather than a huge one for fewer tokens.

How to use Chinchilla law in practice

For a practitioner, the usual workflow is:

1. Decide your training compute budget
   • Estimate total training FLOPs from your hardware, time, and cost constraints.emergentmind+1
2. Pick model size vs data using the ~20:1 rule
   • For a dense decoder-only LLM, target:
     • D≈20×ND \approx 20 \times ND≈20×N (tokens ≈ 20 × parameters).lifearchitect+2
   • Example: If you can afford ~1.0T tokens:
     • Choose N≈50BN \approx 50BN≈50B parameters and train on D≈1.0TD \approx 1.0TD≈1.0T tokens.mbrenndoerfer+1
3. Check feasibility against your dataset
   • If you don’t have enough unique data for 20×N tokens, either:
     • Reduce model size, or
     • Accept that you’ll be off Chinchilla‑optimal and might need more augmentation/regularization.victorleungtw+2
4. Adjust for your regime and quality
   • For very small models or very high‑quality data, the exact optimal ratio can shift, but 20:1 is a solid baseline rather than a strict law.epoch+2
   • Newer works show some projects going higher (30:1–190:1) with different data qualities and objectives, but Chinchilla is still the classic starting point.geeksforgeeks+1
5. Avoid common anti-patterns
   • Don’t train a huge model (e.g., 70B) on only a few hundred billion tokens if you can afford more compute; you’ll undertrain the parameters.deepmind+2
   • Don’t fix a small dataset and keep scaling parameters; at some point you hit strong diminishing returns.emergentmind+1

A practical mental rule you can keep as an engineer: _“For a dense transformer LLM, aim for roughly 20 training tokens per parameter given your compute, and size the model and dataset together instead of just chasing parameter count.”_lifearchitect+2

Are you thinking about applying this for your own pretraining run (e.g., a custom 1–10B model), or more for interview/system-design discussions?

• Understanding intermediate layers using linear classifier probes https://arxiv.org/pdf/1610.01644
• Representation Learning: A Review and New Perspectives https://arxiv.org/pdf/1206.5538
• https://distill.pub/2018/building-blocks/
• On the Number of Linear Regions of Deep Neural Networks https://arxiv.org/pdf/1402.1869