OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 312 of 415 6 min read
7.5.7.4

Sequence-Level vs Token-Level Importance Sampling

Compare sequence-level geometric-mean correction with token-level importance sampling, including their variance and bias trade-offs.

What is it?

TIS and MIS can operate at sequence level or token level. Sequence-level importance sampling uses the geometric mean of token ratios and is theoretically preferred but has higher variance. Token-level correction has lower variance but is biased relative to the sequence correction.

Key points

Sequence-level correction uses the geometric mean across tokens.|Sequence-level correction is theoretically correct but higher variance.|Token-level correction calculates a separate ratio for each token.|Token-level correction is biased but lower variance.

Choosing the importance-sampling level

For a generated sequence containing multiple tokens, sequence-level correction combines token ratios using their geometric mean. Token-level correction evaluates each token separately. The two approaches trade theoretical correctness against variance.

Real-world application

Choose the correction level based on whether theoretical sequence-level correction or lower-variance token-level correction is more important.
Key takeaway

Sequence-level correction is theoretically preferred; token-level correction is a practical lower-variance compromise.