5.8.4
Generating Responses and Reward Scoring
Understand how PPO generates responses from prompts and obtains reward scores for generated outputs.
What is it?
The training loop generates responses from batches of prompts and evaluates each response using a reward model. The generated token sequences are decoded into text before being passed to the reward model. The resulting reward scores are supplied to PPOTrainer.
Key points
1. PPOTrainer provides the training dataloader.
2. Read prompt token IDs from batch["input_ids"].
3. Generate responses with a maximum of 512 new tokens.
4. Use temperature = 0.7 and top_p = 0.9.
5. Decode generated tokens into text.
6. Score each prompt-response pair using the reward model.
7. Convert reward scores into tensors.
8. Supply the responses and rewards to the PPO update step.
Example: Generating and Scoring Responses
for batch in ppo_trainer.dataloader:
query_tensors = batch["input_ids"]
response_tensors = ppo_trainer.generate(
query_tensors,
max_new_tokens=512,
temperature=0.7,
top_p=0.9,
do_sample=True
)
texts = [
tokenizer.decode(r, skip_special_tokens=True)
for r in response_tensors
]
rewards = [
torch.tensor(reward_model.score(q, r))
for q, r in zip(batch["query"], texts)
]
The reward model evaluates generated responses and produces the reward signals for PPO.
Real-world application
This pattern connects prompt datasets, model generation and reward-model evaluation in an RLHF training pipeline.
Key takeaway
PPO training uses generated responses and reward-model scores as inputs to policy optimization.