3.2.2
Return and Cumulative Discounted Reward
Understand how rewards are accumulated over time and how the discount factor affects future rewards.
What is it?
Return is the cumulative reward collected over time. Discounted return is expressed as Gₜ = Σₖ₌₀∞ γᵏ rₜ₊ₖ. The discount factor controls the importance of future rewards.
Key points
1. Return represents cumulative reward.
2. Rewards are collected at each step.
3. Future rewards are discounted using γ.
4. Generally, 0 ≤ γ ≤ 1.
5. Smaller γ emphasizes immediate rewards.
6. Larger γ gives greater importance to future rewards.
Example: Calculating discounted return
Suppose an agent receives:
r₀ = 10
r₁ = 5
r₂ = 2
Let γ = 0.9.
G₀ = 10 + (0.9 × 5) + (0.9² × 2)
G₀ = 10 + 4.5 + 1.62
G₀ = 16.12
Real-world application
Discounted return defines the objective for agents balancing immediate and long-term rewards.
Key takeaway
Return measures accumulated reward; the discount factor controls the importance of future rewards.