10.5.4
Mitigation Strategies for Forgetting and Alignment Tax
Explore data replay, EWC, LoRA, conservative learning rates, beta tuning, diverse data and targeted RL budgets.
What is it?
The source recommends different mitigation strategies for catastrophic forgetting and alignment tax. Forgetting mitigation protects previously learned knowledge during SFT. Alignment-tax mitigation balances capability preservation against safety and preference constraints.
Key points
Catastrophic forgetting mitigation:
Data replay: mix 5–10% of pre-training data into SFT.
EWC: penalise changes to parameters with high Fisher information.
LoRA: train low-rank adapters while keeping base weights frozen; the active adapter can still change behaviour.
Conservative learning rate: 1–5 × 10^-6 with 1–3 epochs.
Progressive training: gradually increase the SFT data proportion.
Alignment-tax mitigation:
Tune beta carefully; the source gives 0.05–0.3 as a typical range.
Use high-quality, diverse SFT data.
Conditional alignment is described as a research-only technique.
Use nuanced feedback through Constitutional AI or RLAIF.
Monitor capability benchmarks and stop RL when regression exceeds acceptable thresholds, typically 2–5% MMLU regression.
Choosing the Right Mitigation
If fine-tuning causes loss of earlier knowledge:
- Replay a portion of pre-training data.
- Apply EWC or LoRA.
- Use a conservative learning rate and fewer epochs.
- Introduce new training distributions progressively.
If preference optimisation causes excessive behavioural constraints:
- Revisit beta.
- Improve SFT data diversity.
- Use nuanced preference feedback.
- Monitor capability regression and limit the RL training budget.
Real-world application
Select the mitigation according to the diagnosed problem. For forgetting, protect important parameters and preserve data coverage. For alignment tax, balance policy constraints against capability benchmarks.
Key takeaway
Use knowledge-preservation techniques for forgetting and alignment-strength controls for alignment tax.