OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / SFT Best Practices and Techniques / When SFT Hurts – Catastrophic Forgetting and Alignment Tax
Lesson 399 of 415 9 min read
10.5.4

Mitigation Strategies for Forgetting and Alignment Tax

Explore data replay, EWC, LoRA, conservative learning rates, beta tuning, diverse data and targeted RL budgets.

What is it?

The source recommends different mitigation strategies for catastrophic forgetting and alignment tax. Forgetting mitigation protects previously learned knowledge during SFT. Alignment-tax mitigation balances capability preservation against safety and preference constraints.

Key points

Catastrophic forgetting mitigation: Data replay: mix 5–10% of pre-training data into SFT. EWC: penalise changes to parameters with high Fisher information. LoRA: train low-rank adapters while keeping base weights frozen; the active adapter can still change behaviour. Conservative learning rate: 1–5 × 10^-6 with 1–3 epochs. Progressive training: gradually increase the SFT data proportion. Alignment-tax mitigation: Tune beta carefully; the source gives 0.05–0.3 as a typical range. Use high-quality, diverse SFT data. Conditional alignment is described as a research-only technique. Use nuanced feedback through Constitutional AI or RLAIF. Monitor capability benchmarks and stop RL when regression exceeds acceptable thresholds, typically 2–5% MMLU regression.

Choosing the Right Mitigation

If fine-tuning causes loss of earlier knowledge: - Replay a portion of pre-training data. - Apply EWC or LoRA. - Use a conservative learning rate and fewer epochs. - Introduce new training distributions progressively. If preference optimisation causes excessive behavioural constraints: - Revisit beta. - Improve SFT data diversity. - Use nuanced preference feedback. - Monitor capability regression and limit the RL training budget.

Real-world application

Select the mitigation according to the diagnosed problem. For forgetting, protect important parameters and preserve data coverage. For alignment tax, balance policy constraints against capability benchmarks.
Key takeaway

Use knowledge-preservation techniques for forgetting and alignment-strength controls for alignment tax.