10.5.2
Alignment Tax: Behavioural Constraint
Understand how preference optimisation and KL regularisation can suppress model capabilities without erasing the underlying knowledge.
What is it?
Alignment tax is a deliberate trade-off in which raw model capability can decrease because the policy is constrained to produce safer, better-formatted and preference-aligned outputs. The source explains that KL regularisation during DPO or PPO limits deviation from the reference policy. Knowledge is suppressed rather than erased.
Key points
Alignment tax is an expected behavioural constraint.
DPO and PPO can penalise deviation from a reference policy through KL divergence.
The model may lose access to high-variance reasoning paths.
Symptoms include over-refusal, stylistic stiffness and excessive caveats.
Raw capability benchmark scores may decline while preference benchmark scores improve.
The model may still contain the knowledge but produce safer, more generic responses.
Knowledge Suppressed by Alignment
A model can solve a complex reasoning problem before preference optimisation. After alignment training, it may produce cautious, generic answers or refuse a benign request.
The underlying knowledge may still exist. The aligned policy has been constrained so that some outputs are less likely to be generated.
This is behavioural suppression rather than necessarily permanent knowledge loss.
Real-world application
Track raw capability benchmarks alongside preference and safety evaluations during alignment training. Investigate whether performance loss is caused by behavioural constraints before changing the model architecture or retraining its knowledge.
Key takeaway
Alignment tax reduces the model’s freedom to express some capabilities; it does not necessarily erase the underlying knowledge.