OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / LLM Architecture and Optimization Methods / Prediction Heads: What Transformers Output
Lesson 23 of 415 6 min read
1.4.1

Language Modeling Head (Pretraining)

What is it?

A language modeling head converts the final hidden representation of a Transformer into scores for possible next tokens. These scores are called logits. Applying softmax converts them into a probability distribution over the model vocabulary. During autoregressive generation, the model uses this distribution to select the next token.

Key points

1. The Transformer produces hidden representations. 2. The language modeling head projects these representations into vocabulary-sized logits. 3. Each logit corresponds to a possible vocabulary token. 4. Softmax can convert logits into probabilities. 5. A decoding strategy selects the next token. 6. The process repeats during text generation.

Example: Predicting the next token

Input prompt: "The capital of France is" The Transformer processes the prompt and produces a final hidden representation. The language modeling head calculates scores for possible next tokens. Illustrative candidates: Paris: 0.90 London: 0.04 Berlin: 0.03 Other tokens: 0.03 These probabilities are only illustrative. Actual model outputs depend on the model and input context.

Real-world application

Language modeling heads are used in text generation, chatbots, code completion and other applications where a model predicts the next token.
Key takeaway

The language modeling head converts Transformer representations into scores over the vocabulary so the model can predict the next token.