4.1.4
The Shared Foundation of LLM Reinforcement Learning
Understand the common policy, reward, KL constraint and policy-gradient machinery.
What is it?
Although RLHF/DPO and RLVR have different goals and reward sources, they share the same core machinery: a language-model policy, a reward signal, a KL constraint against a reference policy and policy-gradient optimization.
Key points
1. Policy πθ: The LLM generates text autoregressively.
2. Reward signal r(x,y): Learned from preferences or computed through verification.
3. KL constraint: Keeps the policy close to a reference policy and helps prevent degenerate solutions.
4. Policy-gradient optimization: Updates the model toward higher reward.
5. PPO and GRPO are optimization methods identified by the source.
6. The reward source changes between the paradigms, but the shared optimization foundation remains.
Example: Two reward sources, one training foundation
An LLM generates an answer to a prompt.
Alignment training:
A learned reward model evaluates whether the answer matches human preferences.
Capability training:
A verifier checks whether the answer is mathematically correct.
In both cases, the LLM policy generates the text, receives a reward signal, stays constrained relative to a reference policy and is updated using policy optimization.
Real-world application
Understanding the shared foundation helps engineers reuse policy optimization concepts across preference alignment and verifiable-reward training.
Key takeaway
Different reward sources can use the same core policy, KL constraint and policy-gradient machinery.