OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / SFT Best Practices and Techniques / Completion-Only Masking
Lesson 388 of 415 5 min read
10.3.1

Why Mask the Prompt?

Understand why instruction fine-tuning should calculate loss on assistant responses rather than system prompts and user messages.

What is it?

In instruction fine-tuning, the model should learn to generate the assistant response, not predict the system prompt or user question. Computing loss on prompt tokens uses training signal on text that the model is meant to condition on. Completion-only masking sets the loss to zero for non-assistant tokens.

Key points

The prompt provides context for the response. The assistant response is the target to learn. Prompt-token loss can waste gradient signal and encourage prompt memorisation. Completion-only masking excludes non-assistant tokens from the loss calculation.

Prompt Tokens vs Completion Tokens

System: You are a helpful assistant. User: What is sequence packing? Assistant: Sequence packing combines multiple short examples into a longer sequence to reduce padding. With completion-only masking, the system and user text provide context, while the assistant response contributes to the training loss.

Real-world application

Use completion-only masking when training instruction-following models on prompt-response datasets. Ensure the target loss is applied to the assistant response rather than the prompt.
Key takeaway

Train the model to produce the answer, while using the prompt as context.