10.3.2
Completion-Only Masking in TRL
Configure DataCollatorForCompletionOnlyLM to apply loss only to tokens after the assistant response template.
What is it?
TRL provides DataCollatorForCompletionOnlyLM to mask prompt tokens during supervised fine-tuning. The response template identifies where the assistant response begins, allowing the collator to apply loss only to subsequent tokens.
Key points
Load the model-compatible tokenizer.
Define the assistant response template.
Create DataCollatorForCompletionOnlyLM with the response template and tokenizer.
The collator masks tokens before the response template.
The response tokens remain available for loss calculation.
Creating a Completion-Only Collator
from transformers import AutoTokenizer
from trl import DataCollatorForCompletionOnlyLM
tokenizer = AutoTokenizer.from_pretrained(
"meta-llama/Llama-3.1-8B-Instruct"
)
response_template = "<|start_header_id|>assistant<|end_header_id|>"
collator = DataCollatorForCompletionOnlyLM(
response_template=response_template,
tokenizer=tokenizer
)
Real-world application
Configure the response template to match the assistant header used by the selected model. Pass the collator to the training pipeline so prompt tokens are excluded from loss.
Key takeaway
The response template tells the collator where assistant-response loss should begin.