10.2.4
Formatting SFT Datasets with TRL
Build a formatting function and connect it to SFTConfig and SFTTrainer.
What is it?
TRL SFTTrainer can use a formatting_func to convert each dataset example into a formatted conversation. The function builds role-based messages and applies the tokenizer chat template before training.
Key points
Import SFTConfig and SFTTrainer from TRL.
Load the tokenizer for the target model.
Create a formatting_func that accepts a dataset example.
Build system, user and assistant messages.
Return tokenizer.apply_chat_template() output.
Pass formatting_func to SFTTrainer.
Formatting an SFT Dataset with TRL
from transformers import AutoTokenizer
from trl import SFTConfig, SFTTrainer
tokenizer = AutoTokenizer.from_pretrained(
"meta-llama/Llama-3.1-8B-Instruct"
)
def formatting_func(example):
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": example["instruction"]},
{"role": "assistant", "content": example["response"]}
]
return tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=False
)
config = SFTConfig(
max_seq_length=2048,
output_dir="sft_model"
)
trainer = SFTTrainer(
model=model,
tokenizer=tokenizer,
args=config,
train_dataset=dataset,
formatting_func=formatting_func
)
trainer.train()
Real-world application
Use formatting_func when the source dataset stores instructions and responses in separate columns. This keeps dataset conversion consistent and connects directly to the TRL supervised fine-tuning workflow.
Key takeaway
A formatting function connects structured training records to the model-specific chat template used by SFTTrainer.