OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / Derivation of the PPO Gradient and Update Rule
Lesson 171 of 415 5 min read
5.4.1

Step 1: The RL Objective

Understand the objective of maximizing expected cumulative reward.

What is it?

The RL objective defines what a reinforcement learning agent tries to achieve: maximize the expected cumulative reward generated by following its policy. The objective is expressed in terms of policy parameters θ and trajectories τ.

Key points

1. The objective is to maximize expected cumulative reward. 2. θ represents the policy parameters. 3. τ represents a trajectory generated by the policy. 4. A trajectory contains the sequence of states, actions and rewards. 5. Equation (5.3): J(θ) = E_{τ~πθ}[Σ from t=0 to T of r_t].

Example: Maximizing Cumulative Reward

An AI agent performs a sequence of actions and receives rewards of 2, 3 and 5. Total cumulative reward = 2 + 3 + 5 = 10. The RL objective is to learn policy parameters that produce higher expected cumulative rewards across trajectories.

Real-world application

The RL objective provides the optimization target for PPO. Training attempts to improve the policy so that its generated trajectories achieve higher rewards.
Key takeaway

Reinforcement learning optimizes policy parameters to maximize expected cumulative reward.