8.5.2
The Best-of-N Algorithm: Generate, Score and Select
Learn the candidate generation, reward scoring, selection and optional fine-tuning steps.
What is it?
Best-of-N generates N responses for each prompt, scores every response with a reward model and selects the highest-scoring candidate. Selected responses can optionally become supervised fine-tuning data for a later training iteration.
Key points
Generate N responses; score each response using a reward model; select the maximum-scoring response; optionally use selected responses as SFT data; typical N range is 4–64.
The Best-of-N Selection Algorithm
For a prompt x, generate candidates y1 through yN. Calculate each reward score rφ(x, yi). Select y* = arg max rφ(x, yi). The selected answer is the response with the highest reward score.
Real-world application
Build a candidate-generation and reward-scoring pipeline for tasks where response quality can be evaluated reliably.
Key takeaway
Best-of-N is a generate, score and select process, with optional iterative fine-tuning.