9.2.1
The Pretrained LLM as a Reward Model Backbone
Understand how a pretrained language model encodes a prompt-response pair into hidden representations.
What is it?
A reward model can use a pretrained large language model as its backbone. The backbone processes the prompt and response together and produces contextual hidden representations that contain information about the input.
Key points
The backbone is a pretrained LLM such as Llama or Mistral. It encodes the prompt-response pair into a sequence of hidden states. The standard reward model architecture replaces the language modelling head with a scalar regression head.
Using an LLM as a Reward Model Backbone
A pretrained Llama model receives a user prompt and an assistant response. Instead of predicting the next vocabulary token, the reward model architecture uses the resulting hidden representations to estimate how good the response is.
Real-world application
Reuse a pretrained language model as the representation backbone when building a reward model from preference data.
Key takeaway
The pretrained LLM provides contextual representations; the reward head converts them into a reward score.