1.7.4
Key Hyperparameters
What is it?
Pretraining hyperparameters control how the model learns, how much data it processes in each update and how its training progresses. These settings must be coordinated rather than selected independently.
Key points
1. Learning rate controls the size of parameter updates.
2. Batch size determines how many tokens contribute to an update.
3. Learning-rate schedules control how the learning rate changes during training.
4. Weight decay helps regulate parameter growth.
5. Context length determines how many tokens the model can process in a sequence.
6. Precision affects memory use and numerical performance.
Published model configurations
Examples from the book:
Llama 3 405B: 15T tokens, 16M-token batch, peak learning rate 8e-5, WSD schedule, context length 8192.
Llama 3 8B: 15T tokens, 4M-token batch, peak learning rate 3e-4, WSD schedule, context length 8192.
These are reported model configurations, not universal defaults.
Real-world application
Use published configurations as reference points. Tune learning rate, batch size, schedule, weight decay, context length and precision for the model and available hardware.
Key takeaway
Hyperparameters work as a coordinated training configuration, not as isolated settings.