OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 314 of 415 6 min read
7.5.9.4

Binary and Top-K Divergence Masks

Understand the binary and top-K token masks used to approximate divergence-based constraints.

What is it?

DPPO approximates divergence constraints with token-level binary or top-K masks. The source describes binary masks based on TV or KL contributions and top-K masks that retain tokens with the largest contributions.

Key points

binary_tv masks tokens using absolute probability difference.|binary_kl masks tokens using a KL contribution.|topk_tv retains tokens with the highest TV contribution.|topk_kl retains tokens with the highest KL contribution.

Four token-level divergence masks

The source lists four approaches: binary_tv, binary_kl, topk_tv, and topk_kl. Binary approaches mask tokens that exceed a divergence threshold. Top-K approaches retain tokens with the largest contributions under the selected divergence measure.

Real-world application

Use these mask types as a conceptual guide when implementing divergence-based token selection.
Key takeaway

DPPO approximates direct divergence constraints using binary or top-K token masks.