10.2.1
Why Chat Templates Matter
Understand how chat templates distinguish system prompts, user messages and assistant responses.
What is it?
Chat templates define how a conversation is represented as a sequence of tokens. Instruction-following models need to distinguish system instructions, user messages and assistant responses. Using an incorrect template, or omitting it, can significantly degrade performance at inference time.
Key points
Chat templates encode conversation structure into tokens.
System messages provide instructions.
User messages contain requests or input.
Assistant messages contain model responses.
The model should use a compatible template during training and inference.
A Conversation with Three Roles
System: You are a helpful assistant.
User: Explain sequence packing.
Assistant: Sequence packing combines multiple short training examples into a longer sequence to reduce padding.
Real-world application
Prepare instruction datasets with clearly separated system, user and assistant roles. Ensure the same model-compatible chat structure is used when preparing training examples and serving requests.
Key takeaway
A chat template teaches the model where each message begins, which role is speaking and where a message ends.