OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / SFT Best Practices and Techniques / When SFT Hurts – Catastrophic Forgetting and Alignment Tax
Lesson 398 of 415 8 min read
10.5.3

Catastrophic Forgetting vs Alignment Tax

Compare the causes, symptoms, training phases, reversibility and detection methods of the two forms of capability regression.

What is it?

Catastrophic forgetting and alignment tax can both reduce benchmark performance, but their causes differ. Forgetting is an unintentional parameter overwrite during sequential training. Alignment tax is a deliberate constraint introduced during preference optimisation. Their symptoms, reversibility and detection methods are different.

Key points

Catastrophic forgetting: unintentional optimisation artifact. Alignment tax: expected trade-off for safety and helpfulness. Forgetting physically overwrites parameters; alignment tax suppresses or constrains output distributions. Forgetting is associated with sequential SFT and domain continued pre-training. Alignment tax is associated with PPO, DPO, KTO and RLHF. Forgetting is generally irreversible without retraining from a checkpoint. Alignment tax can be partially reversible by adjusting beta, system prompts or fine-tuning. A pre-training evaluation perplexity spike suggests forgetting; stable perplexity with lower capability benchmark win rates suggests alignment tax.

Comparing Two Types of Capability Regression

Case A – Catastrophic forgetting: The model loses its ability to solve mathematics after fine-tuning on unrelated chat data. The original knowledge has been damaged. Case B – Alignment tax: The model still has mathematical knowledge but responds with excessive caution or generic answers after preference optimisation. The knowledge may be recoverable through prompting or fine-tuning.

Real-world application

Compare the base and fine-tuned checkpoints. Monitor pre-training evaluation perplexity and raw capability benchmarks. Also monitor preference benchmarks to identify whether alignment improvements are accompanied by capability regressions.
Key takeaway

Forgetting destroys capabilities; alignment tax constrains their expression. Diagnose the cause before selecting a mitigation.