10.2.2
ChatML and Llama 3 Conversation Formats
Compare ChatML and Llama 3 special-token formats for representing conversations.
What is it?
ChatML and Llama 3 use special tokens to represent conversation roles and message boundaries. Both formats separate system, user and assistant messages, but their token conventions differ.
Key points
ChatML uses <|im_start|> to begin a message and <|im_end|> to end it.
Llama 3 uses <|begin_of_text|> to begin the sequence.
Llama 3 uses <|start_header_id|> and <|end_header_id|> to delimit role headers.
Llama 3 uses <|eot_id|> to mark the end of a message.
The model tokenizer and its expected template determine which format to use.
ChatML and Llama 3 Formats
ChatML:
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
Explain AI agents.<|im_end|>
<|im_start|>assistant
AI agents use models and tools to perform tasks.<|im_end|>
Llama 3:
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>
Explain AI agents.<|eot_id|><|start_header_id|>assistant<|end_header_id|>
AI agents use models and tools to perform tasks.<|eot_id|>
Real-world application
Inspect the tokenizer associated with a pretrained model before formatting an SFT dataset. Do not assume that ChatML and Llama 3 templates are interchangeable.
Key takeaway
Different models can use different chat templates. Use the format expected by the selected model.