OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / SFT Best Practices and Techniques / When SFT Hurts – Catastrophic Forgetting and Alignment Tax
Lesson 397 of 415 7 min read
10.5.2

Alignment Tax: Behavioural Constraint

Understand how preference optimisation and KL regularisation can suppress model capabilities without erasing the underlying knowledge.

What is it?

Alignment tax is a deliberate trade-off in which raw model capability can decrease because the policy is constrained to produce safer, better-formatted and preference-aligned outputs. The source explains that KL regularisation during DPO or PPO limits deviation from the reference policy. Knowledge is suppressed rather than erased.

Key points

Alignment tax is an expected behavioural constraint. DPO and PPO can penalise deviation from a reference policy through KL divergence. The model may lose access to high-variance reasoning paths. Symptoms include over-refusal, stylistic stiffness and excessive caveats. Raw capability benchmark scores may decline while preference benchmark scores improve. The model may still contain the knowledge but produce safer, more generic responses.

Knowledge Suppressed by Alignment

A model can solve a complex reasoning problem before preference optimisation. After alignment training, it may produce cautious, generic answers or refuse a benign request. The underlying knowledge may still exist. The aligned policy has been constrained so that some outputs are less likely to be generated. This is behavioural suppression rather than necessarily permanent knowledge loss.

Real-world application

Track raw capability benchmarks alongside preference and safety evaluations during alignment training. Investigate whether performance loss is caused by behavioural constraints before changing the model architecture or retraining its knowledge.
Key takeaway

Alignment tax reduces the model’s freedom to express some capabilities; it does not necessarily erase the underlying knowledge.