1.4.1
Language Modeling Head (Pretraining)
What is it?
A language modeling head converts the final hidden representation of a Transformer into scores for possible next tokens. These scores are called logits. Applying softmax converts them into a probability distribution over the model vocabulary. During autoregressive generation, the model uses this distribution to select the next token.
Key points
1. The Transformer produces hidden representations.
2. The language modeling head projects these representations into vocabulary-sized logits.
3. Each logit corresponds to a possible vocabulary token.
4. Softmax can convert logits into probabilities.
5. A decoding strategy selects the next token.
6. The process repeats during text generation.
Example: Predicting the next token
Input prompt:
"The capital of France is"
The Transformer processes the prompt and produces a final hidden representation.
The language modeling head calculates scores for possible next tokens.
Illustrative candidates:
Paris: 0.90
London: 0.04
Berlin: 0.03
Other tokens: 0.03
These probabilities are only illustrative. Actual model outputs depend on the model and input context.
Real-world application
Language modeling heads are used in text generation, chatbots, code completion and other applications where a model predicts the next token.
Key takeaway
The language modeling head converts Transformer representations into scores over the vocabulary so the model can predict the next token.