1.7.5
Common Failure Modes
What is it?
Pretraining can fail because of unstable optimization, problematic training data or a mismatch between training conditions and deployment requirements. Monitoring and recovery procedures help identify these problems.
Key points
1. Loss spikes: Sudden increases can result from bad data batches or numerical instability.
2. Memorization: The model may reproduce training data verbatim.
3. Context-length mismatch: A model trained on short sequences may perform poorly with much longer inputs.
4. Recovery can involve returning to a checkpoint and skipping problematic batches.
5. Deduplication and extraction monitoring help address memorization.
6. Continued training on long documents and RoPE scaling can help prepare for longer contexts.
Handling a pretraining loss spike
Suppose training loss suddenly increases after a particular batch.
Possible response:
1. Inspect the batch for problematic data.
2. Check for numerical instability.
3. Roll back to a stable checkpoint if necessary.
4. Skip the offending batch and resume training.
The book describes checkpoint rollback and batch skipping in its discussion of Llama 3.
Real-world application
Monitor training loss, checkpoints, data quality and context-length behaviour. Establish recovery procedures before starting expensive training runs.
Key takeaway
Monitor failures early. Data quality, numerical stability and context-length compatibility all matter.