5.6.1
The Components of the PPO RLHF Loop
Understand how prompts, policy generation, reward scoring and optimization connect.
What is it?
The first step in PPO-based RLHF is generating responses from the current policy. The example uses a batch of 128 prompts with a Llama-3-70B policy and a maximum generation length of 512 tokens. Responses are sampled using temperature 0.7 and top-p 0.9.
Key points
1. Model: Llama-3-70B policy.
2. Batch size: 128 prompts.
3. Maximum generation length: 512 tokens.
4. Temperature: 0.7.
5. Top-p: 0.9.
6. The policy generates one response for each prompt.
7. Generation consumes approximately 60% of the training time in this example.
Example: Generating 128 Responses
The training dataset supplies 128 prompts.
The Llama-3-70B policy generates a response for each prompt using temperature 0.7 and top-p 0.9.
Each response can contain up to 512 tokens.
The resulting prompt-response pairs are passed to the reward model for scoring.
Real-world application
Generation systems such as vLLM can help produce responses in batches and improve GPU utilization during RLHF training.
Key takeaway
PPO begins by generating fresh responses using the current policy.