8.5.3
Implementing Best-of-N and Rejection Sampling Fine-Tuning
Explore inference-time Best-of-N and iterative supervised fine-tuning using selected responses.
What is it?
Best-of-N can be implemented at inference time using a text-generation pipeline and a reward model. Rejection Sampling Fine-Tuning extends the approach: generate candidates, score them, retain high-quality responses, perform SFT on the selected data and repeat with the updated model.
Key points
Use multiple sampled generations; score candidates; select the highest reward; apply a quality threshold; collect selected prompt-completion pairs; fine-tune using SFTTrainer; repeat iteratively.
Best-of-N and Rejection Sampling Fine-Tuning
from transformers import pipeline
import numpy as np
gen_pipeline = pipeline(
"text-generation",
model=model,
tokenizer=tokenizer
)
def best_of_n(prompt, n=16, temperature=0.8):
candidates = gen_pipeline(
prompt,
num_return_sequences=n,
temperature=temperature,
do_sample=True,
max_new_tokens=512
)
scores = [
reward_model.score(prompt, c["generated_text"])
for c in candidates
]
return candidates[np.argmax(scores)]["generated_text"]
best_response = best_of_n(prompt, n=16)
# Rejection Sampling Fine-Tuning
from trl import SFTConfig, SFTTrainer
all_responses = []
for prompt in prompts:
candidates = [generate(prompt, temp=0.9) for _ in range(16)]
scores = [reward_model.score(prompt, c) for c in candidates]
best_idx = np.argmax(scores)
if scores[best_idx] > threshold:
all_responses.append({
"prompt": prompt,
"completion": candidates[best_idx]
})
sft_config = SFTConfig(
output_dir="./rft_output",
learning_rate=2e-5,
num_train_epochs=2,
max_seq_length=2048
)
trainer = SFTTrainer(
model=model,
args=sft_config,
train_dataset=all_responses,
tokenizer=tokenizer
)
trainer.train()
Real-world application
Use inference-time selection when you need better answers without retraining. Use rejection sampling fine-tuning when you want to turn selected high-quality outputs into training data for iterative model improvement.
Key takeaway
Rejection Sampling Fine-Tuning converts high-reward generated responses into supervised training data.