5.4.1
Step 1: The RL Objective
Understand the objective of maximizing expected cumulative reward.
What is it?
The RL objective defines what a reinforcement learning agent tries to achieve: maximize the expected cumulative reward generated by following its policy. The objective is expressed in terms of policy parameters θ and trajectories τ.
Key points
1. The objective is to maximize expected cumulative reward.
2. θ represents the policy parameters.
3. τ represents a trajectory generated by the policy.
4. A trajectory contains the sequence of states, actions and rewards.
5. Equation (5.3): J(θ) = E_{τ~πθ}[Σ from t=0 to T of r_t].
Example: Maximizing Cumulative Reward
An AI agent performs a sequence of actions and receives rewards of 2, 3 and 5.
Total cumulative reward = 2 + 3 + 5 = 10.
The RL objective is to learn policy parameters that produce higher expected cumulative rewards across trajectories.
Real-world application
The RL objective provides the optimization target for PPO. Training attempts to improve the policy so that its generated trajectories achieve higher rewards.
Key takeaway
Reinforcement learning optimizes policy parameters to maximize expected cumulative reward.