OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / SFT Best Practices and Techniques / Sequence Packing for Efficiency
Lesson 381 of 415 5 min read
10.1.2

How Sequence Packing Works

Learn how short examples are concatenated into fixed-length sequences separated by EOS tokens.

What is it?

Sequence packing concatenates multiple short training examples into a single sequence up to max_seq_length. EOS tokens separate the examples, while attention masking prevents tokens from different examples from attending to each other.

Key points

Sort examples by length optionally to improve packing efficiency. Greedily place examples into bins of size max_seq_length. Concatenate examples using EOS separators. Use attention masking to prevent cross-example attention. Compute loss only on non-padding tokens.

Packing Several Examples into One Sequence

Suppose a training sequence has a maximum length of 4,096 tokens. Several shorter examples can be placed into the same sequence, separated by EOS tokens, instead of padding each example independently. The packed sequence uses more of its available token positions for training data.

Real-world application

Prepare examples, optionally sort them by length, greedily pack them into fixed-size bins and preserve example boundaries through EOS separators and attention masks.
Key takeaway

Sequence packing increases useful token utilisation by combining multiple short examples into one sequence.