9.2.3
The Scalar Regression Head
Understand how a linear regression head maps a pooled hidden representation to a single reward score.
What is it?
The regression head maps the pooled hidden representation to one scalar reward value. It replaces the language modelling head used for next-token prediction.
Key points
The regression head is a linear layer W ∈ R^(d×1). The input is the pooled hidden state of dimension d. The output is a single scalar reward. The reward score can then be used to compare responses or provide a training signal for reinforcement learning.
From Hidden State to Scalar Reward
Suppose the pooled hidden representation has 4096 dimensions. A linear layer maps those 4096 values to one output, such as 1.7. This number is the reward model score for the prompt-response pair.
Real-world application
Use a scalar regression head when the training objective requires one reward score per response.
Key takeaway
The regression head transforms a high-dimensional language representation into a single reward value.