OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Preference Optimization Variants / Best-of-N Sampling (Rejection Sampling)
Lesson 348 of 415 8 min read
8.5.3

Implementing Best-of-N and Rejection Sampling Fine-Tuning

Explore inference-time Best-of-N and iterative supervised fine-tuning using selected responses.

What is it?

Best-of-N can be implemented at inference time using a text-generation pipeline and a reward model. Rejection Sampling Fine-Tuning extends the approach: generate candidates, score them, retain high-quality responses, perform SFT on the selected data and repeat with the updated model.

Key points

Use multiple sampled generations; score candidates; select the highest reward; apply a quality threshold; collect selected prompt-completion pairs; fine-tune using SFTTrainer; repeat iteratively.

Best-of-N and Rejection Sampling Fine-Tuning

from transformers import pipeline import numpy as np gen_pipeline = pipeline( "text-generation", model=model, tokenizer=tokenizer ) def best_of_n(prompt, n=16, temperature=0.8): candidates = gen_pipeline( prompt, num_return_sequences=n, temperature=temperature, do_sample=True, max_new_tokens=512 ) scores = [ reward_model.score(prompt, c["generated_text"]) for c in candidates ] return candidates[np.argmax(scores)]["generated_text"] best_response = best_of_n(prompt, n=16) # Rejection Sampling Fine-Tuning from trl import SFTConfig, SFTTrainer all_responses = [] for prompt in prompts: candidates = [generate(prompt, temp=0.9) for _ in range(16)] scores = [reward_model.score(prompt, c) for c in candidates] best_idx = np.argmax(scores) if scores[best_idx] > threshold: all_responses.append({ "prompt": prompt, "completion": candidates[best_idx] }) sft_config = SFTConfig( output_dir="./rft_output", learning_rate=2e-5, num_train_epochs=2, max_seq_length=2048 ) trainer = SFTTrainer( model=model, args=sft_config, train_dataset=all_responses, tokenizer=tokenizer ) trainer.train()

Real-world application

Use inference-time selection when you need better answers without retraining. Use rejection sampling fine-tuning when you want to turn selected high-quality outputs into training data for iterative model improvement.
Key takeaway

Rejection Sampling Fine-Tuning converts high-reward generated responses into supervised training data.