OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / TRL Implementation
Lesson 194 of 415 7 min read
5.8.4

Generating Responses and Reward Scoring

Understand how PPO generates responses from prompts and obtains reward scores for generated outputs.

What is it?

The training loop generates responses from batches of prompts and evaluates each response using a reward model. The generated token sequences are decoded into text before being passed to the reward model. The resulting reward scores are supplied to PPOTrainer.

Key points

1. PPOTrainer provides the training dataloader. 2. Read prompt token IDs from batch["input_ids"]. 3. Generate responses with a maximum of 512 new tokens. 4. Use temperature = 0.7 and top_p = 0.9. 5. Decode generated tokens into text. 6. Score each prompt-response pair using the reward model. 7. Convert reward scores into tensors. 8. Supply the responses and rewards to the PPO update step.

Example: Generating and Scoring Responses

for batch in ppo_trainer.dataloader: query_tensors = batch["input_ids"] response_tensors = ppo_trainer.generate( query_tensors, max_new_tokens=512, temperature=0.7, top_p=0.9, do_sample=True ) texts = [ tokenizer.decode(r, skip_special_tokens=True) for r in response_tensors ] rewards = [ torch.tensor(reward_model.score(q, r)) for q, r in zip(batch["query"], texts) ] The reward model evaluates generated responses and produces the reward signals for PPO.

Real-world application

This pattern connects prompt datasets, model generation and reward-model evaluation in an RLHF training pipeline.
Key takeaway

PPO training uses generated responses and reward-model scores as inputs to policy optimization.