1.17.2
Safety Training Pipeline
What is it?
The safety training pipeline applies safeguards throughout model development rather than relying on a single final check. The source highlights data filtering during pretraining, refusal examples during supervised fine-tuning, safety-specific reward models during reinforcement learning from human feedback, and iterative red-teaming.
Key points
1. Safety begins with filtering training data.
2. Pretraining data filtering removes toxic, biased and personally identifiable information-containing text.
3. Supervised fine-tuning introduces examples of appropriate refusals.
4. RLHF can use safety-specific reward models.
5. Red-teaming repeatedly tests the model for weaknesses.
6. Safety is applied across multiple training stages.
Safety checks throughout model training
During pretraining, unsafe or sensitive material can be filtered from training data. During SFT, the model learns examples of appropriate refusals. During RLHF, safety-specific reward signals help shape behaviour. Red-team exercises then test the resulting model for weaknesses.
Real-world application
When developing an AI product, consider safety during data preparation, model adaptation, evaluation and deployment instead of adding safeguards only after the system is built.
Key takeaway
Safety must be incorporated throughout the model training pipeline, not treated as a final feature.