1.4.3
Value Head (Regression for RL)
What is it?
A value head is an additional neural network component that predicts a numerical value from a Transformer representation. In reinforcement learning workflows, this value estimate can represent the expected future return or reward associated with a state. It helps estimate how beneficial a state or partial response is expected to be.
Key points
1. A value head produces a numerical prediction rather than vocabulary logits.
2. It commonly uses the Transformer hidden representation as input.
3. In reinforcement learning, it can estimate expected future return.
4. The estimate is used in value-based learning and policy optimization methods.
5. The value head and language modeling head serve different purposes.
6. The exact value target depends on the reinforcement learning algorithm.
Example: Estimating response value
Imagine an AI assistant generating a response to a customer.
A value head may estimate the expected future reward associated with the current state or partial response.
For illustration:
Estimated value: 0.75
This number is a model prediction, not a guaranteed quality score or a direct measurement of correctness.
Real-world application
Value heads are useful in reinforcement learning workflows such as PPO, where value estimates help calculate advantages and guide policy updates.
Key takeaway
A value head predicts an estimated numerical value that can help guide reinforcement learning updates.