OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Reward Model Training / Multi-Objective Rewards – Combination Strategies
Lesson 371 of 415 5 min read
9.6.2

Weighted Sum and Reward Scaling

Combine rewards using weights and understand why differences in scale can affect the result.

What is it?

The weighted-sum method combines reward signals by multiplying each reward by its weight and adding the results. It is simple, but sensitive to differences in reward scale.

Key points

Formula: r = Σ(w_n × r_n). Each reward signal receives a weight. The weighted rewards are added together. A reward with a larger numerical scale can dominate the combined result, even when its assigned weight is smaller.

Weighted Reward Calculation

Suppose correctness has reward 0.8 with weight 1.0, format has reward 0.5 with weight 0.3, and length has reward 0.2 with weight 0.1. The combined reward is (1.0 × 0.8) + (0.3 × 0.5) + (0.1 × 0.2) = 0.97.

Real-world application

Use weighted sums when reward scales are understood and the relative importance of each objective can be expressed through weights. Check whether one reward dominates because of its scale.
Key takeaway

Weighted sums are straightforward, but reward scale can strongly influence the result.