9.2.2
Pooling the Final Hidden State
Learn how the final token or CLS representation is selected for reward prediction.
What is it?
Pooling selects one hidden representation from the sequence so that the reward model can produce a single score for the complete prompt-response pair.
Key points
For decoder-only models, the standard architecture extracts the hidden state at the last token position. For encoder models, it can use the CLS token representation. The selected representation is passed to the regression head.
Selecting the Representation to Score
A response contains many tokens, each with a hidden representation. The model selects the final token hidden state for a decoder-only architecture and uses it as the compact representation for predicting the response reward.
Real-world application
Check which pooling strategy is appropriate for the model architecture being used.
Key takeaway
Pooling converts a sequence of hidden states into one representation for reward prediction.