OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / LLM Architecture and Optimization Methods / Pretraining: Best Practices
Lesson 49 of 415 6 min read
1.7.5

Common Failure Modes

What is it?

Pretraining can fail because of unstable optimization, problematic training data or a mismatch between training conditions and deployment requirements. Monitoring and recovery procedures help identify these problems.

Key points

1. Loss spikes: Sudden increases can result from bad data batches or numerical instability. 2. Memorization: The model may reproduce training data verbatim. 3. Context-length mismatch: A model trained on short sequences may perform poorly with much longer inputs. 4. Recovery can involve returning to a checkpoint and skipping problematic batches. 5. Deduplication and extraction monitoring help address memorization. 6. Continued training on long documents and RoPE scaling can help prepare for longer contexts.

Handling a pretraining loss spike

Suppose training loss suddenly increases after a particular batch. Possible response: 1. Inspect the batch for problematic data. 2. Check for numerical instability. 3. Roll back to a stable checkpoint if necessary. 4. Skip the offending batch and resume training. The book describes checkpoint rollback and batch skipping in its discussion of Llama 3.

Real-world application

Monitor training loss, checkpoints, data quality and context-length behaviour. Establish recovery procedures before starting expensive training runs.
Key takeaway

Monitor failures early. Data quality, numerical stability and context-length compatibility all matter.