5.8.2
Configuring the PPO Trainer
Understand the configuration of the PPO trainer and its training components.
What is it?
The tokenizer converts text prompts into token IDs and decodes generated token sequences back into text. The book also demonstrates LoRA configuration through PEFT to adapt selected attention projection layers without fully fine-tuning all model weights.
Key points
1. Load AutoTokenizer from Transformers.
2. Use the same model identifier for tokenizer and policy model.
3. LoRA rank r = 64.
4. LoRA alpha = 16.
5. Target modules: q_proj, v_proj, k_proj and o_proj.
6. PEFT adds trainable low-rank adapters to selected model layers.
Example: Tokenizer and LoRA Setup
from transformers import AutoTokenizer
from peft import LoraConfig
tokenizer = AutoTokenizer.from_pretrained(
"meta-llama/Llama-3.1-8B-Instruct"
)
lora_config = LoraConfig(
r=64,
lora_alpha=16,
target_modules=["q_proj", "v_proj", "k_proj", "o_proj"]
)
The tokenizer prepares text for the model, while LoRA defines which projection layers receive trainable adapters.
Real-world application
LoRA reduces the number of trainable parameters and can make large-model fine-tuning more practical with limited GPU memory.
Key takeaway
The tokenizer handles text conversion, while LoRA enables parameter-efficient model adaptation.