OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / RL Foundations for Language Models / Two Paradigms for RL in LLMs
Lesson 143 of 415 6 min read
4.1.4

The Shared Foundation of LLM Reinforcement Learning

Understand the common policy, reward, KL constraint and policy-gradient machinery.

What is it?

Although RLHF/DPO and RLVR have different goals and reward sources, they share the same core machinery: a language-model policy, a reward signal, a KL constraint against a reference policy and policy-gradient optimization.

Key points

1. Policy πθ: The LLM generates text autoregressively. 2. Reward signal r(x,y): Learned from preferences or computed through verification. 3. KL constraint: Keeps the policy close to a reference policy and helps prevent degenerate solutions. 4. Policy-gradient optimization: Updates the model toward higher reward. 5. PPO and GRPO are optimization methods identified by the source. 6. The reward source changes between the paradigms, but the shared optimization foundation remains.

Example: Two reward sources, one training foundation

An LLM generates an answer to a prompt. Alignment training: A learned reward model evaluates whether the answer matches human preferences. Capability training: A verifier checks whether the answer is mathematically correct. In both cases, the LLM policy generates the text, receives a reward signal, stays constrained relative to a reference policy and is updated using policy optimization.

Real-world application

Understanding the shared foundation helps engineers reuse policy optimization concepts across preference alignment and verifiable-reward training.
Key takeaway

Different reward sources can use the same core policy, KL constraint and policy-gradient machinery.