9.6.2
Weighted Sum and Reward Scaling
Combine rewards using weights and understand why differences in scale can affect the result.
What is it?
The weighted-sum method combines reward signals by multiplying each reward by its weight and adding the results. It is simple, but sensitive to differences in reward scale.
Key points
Formula: r = Σ(w_n × r_n).
Each reward signal receives a weight.
The weighted rewards are added together.
A reward with a larger numerical scale can dominate the combined result, even when its assigned weight is smaller.
Weighted Reward Calculation
Suppose correctness has reward 0.8 with weight 1.0, format has reward 0.5 with weight 0.3, and length has reward 0.2 with weight 0.1. The combined reward is (1.0 × 0.8) + (0.3 × 0.5) + (0.1 × 0.2) = 0.97.
Real-world application
Use weighted sums when reward scales are understood and the relative importance of each objective can be expressed through weights. Check whether one reward dominates because of its scale.
Key takeaway
Weighted sums are straightforward, but reward scale can strongly influence the result.