OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / SFT Best Practices and Techniques / Connection to RL – SFT Quality Determines RL Ceiling
Lesson 404 of 415 9 min read
10.6.4

Evaluating SFT Readiness with Pass@k

Estimate pass@1 and pass@k by sampling multiple responses from the SFT model before starting reinforcement learning.

What is it?

Pass@k evaluation estimates whether an SFT model can produce correct responses when sampling multiple completions for each prompt. Pass@1 estimates single-sample success, while pass@k measures whether at least one of k sampled responses is correct.

Key points

Sample multiple completions for each evaluation prompt. Compare each completion with the ground-truth answer. Pass@1 estimates the response success rate. Pass@k measures whether at least one of k responses is correct. The source states that pass@1 below 5% suggests RL will likely fail. The source states that pass@k below 20% suggests RL will struggle. The example uses k=8 and evaluates 100 examples.

Estimating Pass@k Before RL

import numpy as np from tqdm import tqdm def estimate_pass_at_k(model, tokenizer, dataset, k=8, n_samples=100): pass_at_1_scores = [] pass_at_k_scores = [] for example in tqdm(dataset.select(range(n_samples))): prompt = example["prompt"] ground_truth = example["answer"] inputs = tokenizer( prompt, return_tensors="pt" ).to(model.device) outputs = model.generate( **inputs, max_new_tokens=512, do_sample=True, temperature=0.8, num_return_sequences=k ) correct = 0 for output in outputs: response = tokenizer.decode( output, skip_special_tokens=True ) if ground_truth in response: correct += 1 pass_at_1_scores.append(correct / k) pass_at_k_scores.append(correct >= 1) print(f"Pass@1: {np.mean(pass_at_1_scores):.2%}") print(f"Pass@{k}: {np.mean(pass_at_k_scores):.2%}")

Real-world application

Run pass@k on a representative evaluation dataset before RL. Use a task-appropriate correctness check for the ground-truth answer and inspect both single-sample and multi-sample success rates.
Key takeaway

Pass@k checks whether the SFT model has the potential to produce a correct response before reinforcement learning begins.