8.5.4
Best-of-N Scaling Laws: Quality Gain vs Compute Cost
Compare candidate counts, response quality gains and compute costs, and understand Best-of-N as an RL baseline.
What is it?
Increasing the number of candidates can improve Best-of-N response quality, but generation cost also increases. The source compares candidate counts from 1 to 256 and recommends using Best-of-N as a baseline when evaluating other RL methods under the same compute budget.
Key points
N=1 is the baseline; N=4 gives approximately 5–8% win-rate gain at 4× cost; N=16 gives 10–15% at 16× cost; N=64 gives 15–20% at 64× cost; N=256 gives 18–22% at 256× cost; diminishing returns increase with N.
Choosing the Number of Candidates
The source reports these illustrative scaling results:
N=1: Baseline, 1× cost.
N=4: +5–8% win-rate, 4× cost.
N=16: +10–15% win-rate, 16× cost.
N=64: +15–20% win-rate, 64× cost.
N=256: +18–22% win-rate, 256× cost.
The source advises comparing an RL method with Best-of-N using the same compute budget.
Real-world application
Use Best-of-N as a benchmark before investing in more complex RL training. Choose N by balancing quality improvement against generation cost.
Key takeaway
A fair RL evaluation compares methods against Best-of-N at an equivalent compute budget.