10.2.3
Applying Chat Templates with Tokenizers
Use tokenizer.apply_chat_template to convert structured messages into formatted training text.
What is it?
Transformers tokenizers provide apply_chat_template() to convert structured conversation messages into the format expected by a model. Messages are supplied as a list of role and content dictionaries.
Key points
Load the tokenizer associated with the selected model.
Represent each message using role and content fields.
Pass the messages to tokenizer.apply_chat_template().
Use tokenize=False when the formatting function should return formatted text.
Use add_generation_prompt=False when formatting a complete conversation that already contains the assistant response.
Applying a Chat Template
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
"meta-llama/Llama-3.1-8B-Instruct"
)
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain AI agents."},
{"role": "assistant", "content": "AI agents use models and tools to perform tasks."}
]
formatted_text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=False
)
Real-world application
Use apply_chat_template() inside dataset preprocessing to consistently convert role-based conversation records into model-compatible training text.
Key takeaway
Use the tokenizer to apply the model-specific conversation format instead of manually constructing special-token strings.