10.6.4
Evaluating SFT Readiness with Pass@k
Estimate pass@1 and pass@k by sampling multiple responses from the SFT model before starting reinforcement learning.
What is it?
Pass@k evaluation estimates whether an SFT model can produce correct responses when sampling multiple completions for each prompt. Pass@1 estimates single-sample success, while pass@k measures whether at least one of k sampled responses is correct.
Key points
Sample multiple completions for each evaluation prompt.
Compare each completion with the ground-truth answer.
Pass@1 estimates the response success rate.
Pass@k measures whether at least one of k responses is correct.
The source states that pass@1 below 5% suggests RL will likely fail.
The source states that pass@k below 20% suggests RL will struggle.
The example uses k=8 and evaluates 100 examples.
Estimating Pass@k Before RL
import numpy as np
from tqdm import tqdm
def estimate_pass_at_k(model, tokenizer, dataset, k=8, n_samples=100):
pass_at_1_scores = []
pass_at_k_scores = []
for example in tqdm(dataset.select(range(n_samples))):
prompt = example["prompt"]
ground_truth = example["answer"]
inputs = tokenizer(
prompt,
return_tensors="pt"
).to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.8,
num_return_sequences=k
)
correct = 0
for output in outputs:
response = tokenizer.decode(
output,
skip_special_tokens=True
)
if ground_truth in response:
correct += 1
pass_at_1_scores.append(correct / k)
pass_at_k_scores.append(correct >= 1)
print(f"Pass@1: {np.mean(pass_at_1_scores):.2%}")
print(f"Pass@{k}: {np.mean(pass_at_k_scores):.2%}")
Real-world application
Run pass@k on a representative evaluation dataset before RL. Use a task-appropriate correctness check for the ground-truth answer and inspect both single-sample and multi-sample success rates.
Key takeaway
Pass@k checks whether the SFT model has the potential to produce a correct response before reinforcement learning begins.