10.3.4
Completion Masking Pitfalls
Recognise response-template tokenisation errors, short-response problems and masking requirements for multi-turn conversations.
What is it?
Completion-only masking can fail when the response template does not exactly match the tokenised sequence. Very short responses may leave too few training tokens, while multi-turn conversations require masking every non-assistant turn.
Key points
The response template must match its tokenised form exactly.
Tokenisation off-by-one errors can apply the mask incorrectly.
Very short responses may provide insufficient gradient signal.
Consider a minimum response-length threshold.
For multi-turn conversations, mask all non-assistant turns, not only the first prompt.
Three Completion Masking Pitfalls
Pitfall 1: Template mismatch
The configured assistant header differs from the actual tokenised header, so the masking boundary is incorrect.
Pitfall 2: Very short responses
After prompt masking, too few response tokens remain to provide meaningful training signal.
Pitfall 3: Multi-turn conversations
Masking only the first prompt can leave later user or system turns included in the loss.
Real-world application
Validate the response-template boundary on tokenised examples before training. Inspect multi-turn samples and consider filtering examples whose assistant responses are too short.
Key takeaway
Correct masking depends on exact template matching, sufficient response tokens and handling every non-assistant turn.