OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Reward Model Training / Reward Model Architectures
Lesson 359 of 415 5 min read
9.2.1

The Pretrained LLM as a Reward Model Backbone

Understand how a pretrained language model encodes a prompt-response pair into hidden representations.

What is it?

A reward model can use a pretrained large language model as its backbone. The backbone processes the prompt and response together and produces contextual hidden representations that contain information about the input.

Key points

The backbone is a pretrained LLM such as Llama or Mistral. It encodes the prompt-response pair into a sequence of hidden states. The standard reward model architecture replaces the language modelling head with a scalar regression head.

Using an LLM as a Reward Model Backbone

A pretrained Llama model receives a user prompt and an assistant response. Instead of predicting the next vocabulary token, the reward model architecture uses the resulting hidden representations to estimate how good the response is.

Real-world application

Reuse a pretrained language model as the representation backbone when building a reward model from preference data.
Key takeaway

The pretrained LLM provides contextual representations; the reward head converts them into a reward score.