OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Reward Model Training / Reward Model Architectures
Lesson 360 of 415 5 min read
9.2.2

Pooling the Final Hidden State

Learn how the final token or CLS representation is selected for reward prediction.

What is it?

Pooling selects one hidden representation from the sequence so that the reward model can produce a single score for the complete prompt-response pair.

Key points

For decoder-only models, the standard architecture extracts the hidden state at the last token position. For encoder models, it can use the CLS token representation. The selected representation is passed to the regression head.

Selecting the Representation to Score

A response contains many tokens, each with a hidden representation. The model selects the final token hidden state for a decoder-only architecture and uses it as the compact representation for predicting the response reward.

Real-world application

Check which pooling strategy is appropriate for the model architecture being used.
Key takeaway

Pooling converts a sequence of hidden states into one representation for reward prediction.