1.1.3
What is next-token prediction?
What is it?
Next-token prediction is the central mechanism used by autoregressive language models to generate text. Given a sequence of previous tokens, the model estimates the probability of possible next tokens. It then selects one token and uses the updated sequence to predict the next one.
Key points
1. A token can represent a word, part of a word or another text unit.
2. The model receives the tokens that came before.
3. It calculates a probability distribution over possible next tokens.
4. A decoding method determines which token is selected.
5. The selected token becomes part of the input for the next prediction.
Example: Predicting the next token
Consider this input:
"Machine learning is a branch of"
Possible next tokens might include:
• Artificial
• Computer
• Data
The model assigns probabilities to possible continuations. The selected token is then used to continue generating the sentence.
The actual prediction depends on the model, context and decoding settings.
Real-world application
Next-token prediction is used in text completion, conversational assistants, code generation, document drafting and other language-generation tasks.
Key takeaway
Next-token prediction means estimating what token is most suitable to come next, based on the tokens already provided.