1.4.4
Head Selection Summary
What is it?
Different prediction heads support different model outputs. A language modeling head produces vocabulary logits for token prediction. A value head produces numerical estimates used in reinforcement learning. Conditional generation uses the language modeling output while conditioning generation on an input prompt. Selecting the appropriate head depends on the training objective and task.
Key points
1. Language modeling head: predicts scores for vocabulary tokens.
2. Conditional generation: generates output based on an input context.
3. Value head: estimates a numerical value for a state or representation.
4. Language generation typically uses vocabulary logits.
5. Reinforcement learning workflows may use both policy and value outputs.
6. The required output determines the appropriate model configuration.
Example: Choosing the output head
Task: Generate an answer to a question.
Required output: A sequence of text tokens.
Relevant component: Language modeling head.
Task: Estimate the expected return of a state during PPO training.
Required output: A numerical value.
Relevant component: Value head.
The two outputs serve different roles in the learning process.
Real-world application
Understanding prediction heads helps developers configure language models, build reinforcement learning pipelines and distinguish generation outputs from value estimates.
Key takeaway
Choose the prediction head according to the output required: vocabulary scores for generation or numerical values for value estimation.