10.1.2
How Sequence Packing Works
Learn how short examples are concatenated into fixed-length sequences separated by EOS tokens.
What is it?
Sequence packing concatenates multiple short training examples into a single sequence up to max_seq_length. EOS tokens separate the examples, while attention masking prevents tokens from different examples from attending to each other.
Key points
Sort examples by length optionally to improve packing efficiency.
Greedily place examples into bins of size max_seq_length.
Concatenate examples using EOS separators.
Use attention masking to prevent cross-example attention.
Compute loss only on non-padding tokens.
Packing Several Examples into One Sequence
Suppose a training sequence has a maximum length of 4,096 tokens. Several shorter examples can be placed into the same sequence, separated by EOS tokens, instead of padding each example independently. The packed sequence uses more of its available token positions for training data.
Real-world application
Prepare examples, optionally sort them by length, greedily pack them into fixed-size bins and preserve example boundaries through EOS separators and attention masks.
Key takeaway
Sequence packing increases useful token utilisation by combining multiple short examples into one sequence.