10.1.3
Attention Masking and Packing Efficiency
Understand block-diagonal attention masks, packing efficiency and the risk of cross-example attention.
What is it?
Packed examples require careful attention masking. A block-diagonal attention mask prevents tokens belonging to one example from attending to tokens belonging to another, avoiding cross-example contamination.
Key points
Typical packing efficiency: 85–95%.
Typical efficiency with padding: 20–50%.
The source reports a 2–4× speedup for datasets with high length variance.
Memory use is similar to padding because the total tokens per batch remain the same.
Careful attention masking is essential to prevent cross-example attention.
Preventing Cross-Example Attention
Imagine two examples concatenated into one sequence. Without a suitable attention mask, tokens from the second example could attend to tokens from the first. A block-diagonal attention mask keeps the attention computation separated between the original examples.
Real-world application
Verify that the model and training implementation correctly support block-diagonal attention masks when enabling sequence packing. Monitor packing efficiency and training throughput.
Key takeaway
Packing improves token utilisation, but incorrect attention masking can contaminate training examples.