4.3.4
Stage 4: Evaluation and Iteration
Understand how evaluation identifies failure cases and guides further training iterations.
What is it?
Evaluation and Iteration is the fourth stage of the RLHF pipeline. The aligned model is evaluated, new failure cases are collected, and the training process is refined through further iterations.
Key points
1. Evaluate the aligned language model.
2. Check response quality and instruction following.
3. Identify failure cases.
4. Collect new examples from these failures.
5. Use the findings to improve the training process.
6. Repeat evaluation and iteration.
7. The process is not complete after one optimization cycle.
Example: Improving a support assistant through evaluation
An aligned assistant is tested using customer-support questions.
Evaluation identifies that it sometimes provides incorrect refund information.
The team collects these failure cases, reviews the training data and improves the reward or training process.
The updated model is evaluated again.
This feedback cycle helps improve the aligned model over time.
Real-world application
Evaluation and iteration are important for detecting weaknesses that may not appear during initial training.
Key takeaway
Evaluate the aligned model, collect failure cases and iterate to improve it.