10.5.3
Catastrophic Forgetting vs Alignment Tax
Compare the causes, symptoms, training phases, reversibility and detection methods of the two forms of capability regression.
What is it?
Catastrophic forgetting and alignment tax can both reduce benchmark performance, but their causes differ. Forgetting is an unintentional parameter overwrite during sequential training. Alignment tax is a deliberate constraint introduced during preference optimisation. Their symptoms, reversibility and detection methods are different.
Key points
Catastrophic forgetting: unintentional optimisation artifact.
Alignment tax: expected trade-off for safety and helpfulness.
Forgetting physically overwrites parameters; alignment tax suppresses or constrains output distributions.
Forgetting is associated with sequential SFT and domain continued pre-training.
Alignment tax is associated with PPO, DPO, KTO and RLHF.
Forgetting is generally irreversible without retraining from a checkpoint.
Alignment tax can be partially reversible by adjusting beta, system prompts or fine-tuning.
A pre-training evaluation perplexity spike suggests forgetting; stable perplexity with lower capability benchmark win rates suggests alignment tax.
Comparing Two Types of Capability Regression
Case A – Catastrophic forgetting:
The model loses its ability to solve mathematics after fine-tuning on unrelated chat data. The original knowledge has been damaged.
Case B – Alignment tax:
The model still has mathematical knowledge but responds with excessive caution or generic answers after preference optimisation. The knowledge may be recoverable through prompting or fine-tuning.
Real-world application
Compare the base and fine-tuned checkpoints. Monitor pre-training evaluation perplexity and raw capability benchmarks. Also monitor preference benchmarks to identify whether alignment improvements are accompanied by capability regressions.
Key takeaway
Forgetting destroys capabilities; alignment tax constrains their expression. Diagnose the cause before selecting a mitigation.