1.7.3
Scaling Laws
What is it?
Scaling laws describe how model performance and training requirements relate to model size, dataset size and compute. Compute-optimal training requires balancing the number of model parameters with the number of training tokens.
Key points
1. Model size and training data must be considered together.
2. Chinchilla scaling laws describe compute-optimal relationships between model size and data size.
3. Increasing parameters without sufficient data can waste compute.
4. Overtraining smaller models can reduce inference costs because inference cost depends strongly on model size.
Balancing model size and training tokens
The book gives an example of a 70-billion-parameter model being compute-optimal at approximately 1.4 trillion tokens under the Chinchilla approach.
In practice, some models are trained on more tokens than this because smaller models can be cheaper to serve.
Real-world application
Estimate the available compute budget before selecting model size and training-token targets. Evaluate both training cost and the expected cost of deploying the model.
Key takeaway
The best training configuration is a balance between model size, data size and compute.