8.5.1
Why Best-of-N Sampling Needs No RL Training
Understand how Best-of-N improves response quality by selecting among generated candidates without updating model weights.
What is it?
Best-of-N improves response quality without changing model weights during the RL phase. The method generates multiple candidate responses and selects the best one using a reward model. It can also serve as the starting point for Rejection Sampling Fine-Tuning.
Key points
No model-weight updates during inference-time selection; generate multiple candidates; score candidates with a reward model; select the highest-scoring response; optional fine-tuning on selected responses.
Selecting a Better Answer Without Training
For a prompt, generate 16 candidate answers. A reward model scores each answer. The system returns the candidate with the highest reward score. The original model weights remain unchanged.
Real-world application
Use Best-of-N when you need stronger outputs at inference time without running a new RL training process.
Key takeaway
Best-of-N improves output selection by generating alternatives and choosing the highest-reward candidate.