OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / SFT Best Practices and Techniques / Sequence Packing for Efficiency
Lesson 380 of 415 5 min read
10.1.1

The Padding Problem in SFT Batches

Understand how padding variable-length examples wastes training computation.

What is it?

Standard SFT batches pad every sequence to the length of the longest sequence in the batch. When examples have very different lengths, a large proportion of computation is spent processing padding tokens rather than useful training data.

Key points

Batches are padded to the longest sequence. High length variance increases padding waste. The source estimates 50–80% of compute may be wasted on padding tokens. Sequence packing is designed to reduce this waste.

Short Instructions and Long Documents

Consider a batch containing several short instructions and one long document. Standard padding extends every short instruction to the length of the long document. Most positions in the shorter examples are padding, consuming compute without adding useful training tokens.

Real-world application

Inspect the sequence-length distribution in an SFT dataset. Datasets with high length variance are strong candidates for sequence packing.
Key takeaway

Padding wastes computation when short examples are extended to match much longer examples.