OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / SFT Best Practices and Techniques / When SFT Hurts – Catastrophic Forgetting and Alignment Tax
Lesson 396 of 415 7 min read
10.5.1

Catastrophic Forgetting: Structural Erasure

Understand how training on a new distribution can overwrite parameters that encode previously learned capabilities.

What is it?

Catastrophic forgetting occurs when a model trained on one data distribution is subsequently trained on a different distribution, and the new weight updates overwrite parameters that encoded earlier capabilities. The source describes this as an unintentional optimisation failure: knowledge is destroyed and cannot be recovered without retraining from a suitable checkpoint.

Key points

Training on distribution DB can overwrite parameters learned from distribution DA. The loss on Task A can increase after optimisation on Task B. Symptoms include breakdown on tasks absent from fine-tuning data, reduced language diversity, lower factual accuracy and degraded multilingual ability. The Fisher Information Matrix identifies parameters important for the original task. Unconstrained updates on Task B can damage high-Fisher-information directions for Task A.

Forgetting Mathematics After Chat Fine-Tuning

A base model can solve mathematics and generate multilingual text. It is then fine-tuned only on a narrow chat dataset. After fine-tuning, it may perform well on the new chat style but fail at mathematics, generate less diverse language and lose some multilingual ability. This is catastrophic forgetting when the original capabilities have been overwritten in the model parameters.

Real-world application

Evaluate the model on tasks that were important before fine-tuning. Preserve a baseline checkpoint and compare performance after each training stage, especially when the new dataset differs substantially from the original training distribution.
Key takeaway

Catastrophic forgetting is structural knowledge loss caused by parameter updates that damage previously learned capabilities.