1.17.1
Threat Taxonomy
What is it?
LLM safety begins by identifying the threats that can arise when language models generate content or interact with users and systems. The source identifies six major categories: harmful content, bias and discrimination, privacy violations, jailbreaking, misinformation and dual-use risks.
Key points
1. Harmful content: Generating toxic, violent or illegal instructions.
2. Bias and discrimination: Perpetuating stereotypes or treating groups unfairly.
3. Privacy violations: Exposing personal information or memorized training data.
4. Jailbreaking: Using adversarial prompts to bypass safety guardrails.
5. Misinformation: Generating convincing but false claims.
6. Dual-use: Using legitimate capabilities for harmful purposes.
Identifying safety threats in an AI assistant
An AI assistant may generate harmful instructions, reproduce biased language, expose private information, follow a jailbreak prompt, produce false claims or provide technical capabilities that can be misused. Each case represents a different safety threat and requires an appropriate control.
Real-world application
Before deploying an AI assistant, identify the types of information it handles, the actions it can perform and the ways its capabilities could be misused. Use this threat classification to plan safeguards.
Key takeaway
AI safety starts with understanding the different ways a model can cause harm or be misused.