1.7.1
Training Objective
What is it?
Causal Language Modeling (CLM) trains a decoder-only language model to predict the next token using the tokens that came before it. The model learns by comparing its predicted probability for the actual next token with the correct answer across a sequence.
Key points
1. The model predicts one token at a time.
2. Each prediction uses only preceding tokens.
3. Training minimizes the negative log-probability of the correct next token.
4. Large-scale training can produce capabilities without explicit labels for each capability.
Next-token prediction
Input: The capital of France is
Expected next token: Paris
The model calculates the probability of possible next tokens and learns to assign a higher probability to the correct continuation.
Real-world application
Use causal language modeling when pretraining decoder-only models on large text collections. Prepare token sequences and train the model to predict each next token.
Key takeaway
Pretraining uses next-token prediction as its central learning objective.