OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / SFT Best Practices and Techniques / Sequence Packing for Efficiency
Lesson 382 of 415 6 min read
10.1.3

Attention Masking and Packing Efficiency

Understand block-diagonal attention masks, packing efficiency and the risk of cross-example attention.

What is it?

Packed examples require careful attention masking. A block-diagonal attention mask prevents tokens belonging to one example from attending to tokens belonging to another, avoiding cross-example contamination.

Key points

Typical packing efficiency: 85–95%. Typical efficiency with padding: 20–50%. The source reports a 2–4× speedup for datasets with high length variance. Memory use is similar to padding because the total tokens per batch remain the same. Careful attention masking is essential to prevent cross-example attention.

Preventing Cross-Example Attention

Imagine two examples concatenated into one sequence. Without a suitable attention mask, tokens from the second example could attend to tokens from the first. A block-diagonal attention mask keeps the attention computation separated between the original examples.

Real-world application

Verify that the model and training implementation correctly support block-diagonal attention masks when enabling sequence packing. Monitor packing efficiency and training throughput.
Key takeaway

Packing improves token utilisation, but incorrect attention masking can contaminate training examples.