OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / LLM Architecture and Optimization Methods / Pretraining: Best Practices
Lesson 48 of 415 7 min read
1.7.4

Key Hyperparameters

What is it?

Pretraining hyperparameters control how the model learns, how much data it processes in each update and how its training progresses. These settings must be coordinated rather than selected independently.

Key points

1. Learning rate controls the size of parameter updates. 2. Batch size determines how many tokens contribute to an update. 3. Learning-rate schedules control how the learning rate changes during training. 4. Weight decay helps regulate parameter growth. 5. Context length determines how many tokens the model can process in a sequence. 6. Precision affects memory use and numerical performance.

Published model configurations

Examples from the book: Llama 3 405B: 15T tokens, 16M-token batch, peak learning rate 8e-5, WSD schedule, context length 8192. Llama 3 8B: 15T tokens, 4M-token batch, peak learning rate 3e-4, WSD schedule, context length 8192. These are reported model configurations, not universal defaults.

Real-world application

Use published configurations as reference points. Tune learning rate, batch size, schedule, weight decay, context length and precision for the model and available hardware.
Key takeaway

Hyperparameters work as a coordinated training configuration, not as isolated settings.