OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Reward Model Training / Reward Model Architectures
Lesson 361 of 415 5 min read
9.2.3

The Scalar Regression Head

Understand how a linear regression head maps a pooled hidden representation to a single reward score.

What is it?

The regression head maps the pooled hidden representation to one scalar reward value. It replaces the language modelling head used for next-token prediction.

Key points

The regression head is a linear layer W ∈ R^(d×1). The input is the pooled hidden state of dimension d. The output is a single scalar reward. The reward score can then be used to compare responses or provide a training signal for reinforcement learning.

From Hidden State to Scalar Reward

Suppose the pooled hidden representation has 4096 dimensions. A linear layer maps those 4096 values to one output, such as 1.7. This number is the reward model score for the prompt-response pair.

Real-world application

Use a scalar regression head when the training objective requires one reward score per response.
Key takeaway

The regression head transforms a high-dimensional language representation into a single reward value.