10.1.1
The Padding Problem in SFT Batches
Understand how padding variable-length examples wastes training computation.
What is it?
Standard SFT batches pad every sequence to the length of the longest sequence in the batch. When examples have very different lengths, a large proportion of computation is spent processing padding tokens rather than useful training data.
Key points
Batches are padded to the longest sequence.
High length variance increases padding waste.
The source estimates 50–80% of compute may be wasted on padding tokens.
Sequence packing is designed to reduce this waste.
Short Instructions and Long Documents
Consider a batch containing several short instructions and one long document. Standard padding extends every short instruction to the length of the long document. Most positions in the shorter examples are padding, consuming compute without adding useful training tokens.
Real-world application
Inspect the sequence-length distribution in an SFT dataset. Datasets with high length variance are strong candidates for sequence packing.
Key takeaway
Padding wastes computation when short examples are extended to match much longer examples.