OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / PPO for RLHF: The Full Loop
Lesson 180 of 415 5 min read
5.6.1

The Components of the PPO RLHF Loop

Understand how prompts, policy generation, reward scoring and optimization connect.

What is it?

The first step in PPO-based RLHF is generating responses from the current policy. The example uses a batch of 128 prompts with a Llama-3-70B policy and a maximum generation length of 512 tokens. Responses are sampled using temperature 0.7 and top-p 0.9.

Key points

1. Model: Llama-3-70B policy. 2. Batch size: 128 prompts. 3. Maximum generation length: 512 tokens. 4. Temperature: 0.7. 5. Top-p: 0.9. 6. The policy generates one response for each prompt. 7. Generation consumes approximately 60% of the training time in this example.

Example: Generating 128 Responses

The training dataset supplies 128 prompts. The Llama-3-70B policy generates a response for each prompt using temperature 0.7 and top-p 0.9. Each response can contain up to 512 tokens. The resulting prompt-response pairs are passed to the reward model for scoring.

Real-world application

Generation systems such as vLLM can help produce responses in batches and improve GPU utilization during RLHF training.
Key takeaway

PPO begins by generating fresh responses using the current policy.