1.13.1
In-Context Learning (ICL)
What is it?
In-Context Learning (ICL) is the ability of a language model to learn a task at inference time from examples included in the prompt. The model identifies the pattern in the examples and applies it to new inputs without updating its weights.
Key points
ICL uses examples inside the prompt. No gradient updates or fine-tuning are required. The model infers the task from input-output patterns. The source discusses implicit Bayesian inference, induction heads and task vectors as explanations for how ICL works. ICL generally improves with model scale.
Learning a Pattern from Examples
Prompt examples: Input: happy, Output: positive. Input: terrible, Output: negative. New input: excellent. The model identifies the demonstrated sentiment classification pattern and predicts positive.
Real-world application
Use ICL when a model needs to follow a particular task format or pattern without changing the model through fine-tuning.
Key takeaway
In-context learning allows a model to adapt its behaviour from examples provided directly in the prompt.