OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / LLM Architecture and Optimization Methods / Pretraining: Best Practices
Lesson 47 of 415 6 min read
1.7.3

Scaling Laws

What is it?

Scaling laws describe how model performance and training requirements relate to model size, dataset size and compute. Compute-optimal training requires balancing the number of model parameters with the number of training tokens.

Key points

1. Model size and training data must be considered together. 2. Chinchilla scaling laws describe compute-optimal relationships between model size and data size. 3. Increasing parameters without sufficient data can waste compute. 4. Overtraining smaller models can reduce inference costs because inference cost depends strongly on model size.

Balancing model size and training tokens

The book gives an example of a 70-billion-parameter model being compute-optimal at approximately 1.4 trillion tokens under the Chinchilla approach. In practice, some models are trained on more tokens than this because smaller models can be cheaper to serve.

Real-world application

Estimate the available compute budget before selecting model size and training-token targets. Evaluate both training cost and the expected cost of deploying the model.
Key takeaway

The best training configuration is a balance between model size, data size and compute.