1.17.4
The Helpfulness-Safety Tradeoff
What is it?
The helpfulness-safety tradeoff concerns how to make an AI model useful while preventing harmful behaviour. Excessive safety restrictions can cause over-refusal, where the model declines legitimate requests. The objective is to maximize helpfulness while satisfying defined safety constraints.
Key points
1. Over-refusal occurs when a model rejects harmless requests.
2. Safety controls should not unnecessarily block legitimate tasks.
3. The source describes optimizing helpfulness subject to a minimum safety constraint.
4. A weighted reward can combine helpfulness and safety signals.
5. The source gives a typical helpfulness weight alpha of 0.6–0.8.
6. The balance requires careful evaluation and tuning.
Balancing helpfulness and safety
A user asks an assistant for an educational explanation of historical violence. A model with excessive restrictions may refuse the request even though the context is legitimate. A better policy allows appropriate educational discussion while continuing to block harmful instructions.
Real-world application
Evaluate both harmful responses and unnecessary refusals when configuring an AI assistant. Safety controls should protect users without making the product unusable for legitimate work.
Key takeaway
A good AI system should be maximally helpful within clearly defined safety constraints.