5.9.7
Generation Temperature (temperature)
Understand how generation temperature affects exploration, response diversity and the noise in advantage estimates.
What is it?
Generation temperature controls the randomness of token sampling. It influences the diversity of generated responses and therefore the quality and variability of the reward and advantage signals.
Key points
1. Typical range: 0.7–1.0.
2. Lower temperature reduces exploration and response diversity.
3. Higher temperature produces more varied outputs.
4. Excessively high temperature can make advantage estimates noisier.
5. The book’s TRL example uses temperature = 0.7.
Example: Generation Temperature
temperature = 0.7
The model samples responses using a temperature of 0.7.
Lower temperature makes generation more focused, while higher temperature generally increases diversity.
Real-world application
Experiment with temperature while monitoring response quality, reward scores and the variability of training signals.
Key takeaway
Temperature controls the exploration-diversity trade-off during response generation.