1.3.4
Embeddings: From Discrete Tokens to Continuous Space
What is it?
An embedding converts a discrete token ID into a numerical vector. These vectors are learned during model training and provide a continuous representation that neural network layers can process. Contextual representations are then developed as the token vectors pass through Transformer layers.
Key points
1. A tokenizer converts text into token IDs.
2. An embedding layer maps each token ID to a vector.
3. The vector contains numerical values.
4. Embeddings are learned during training.
5. Transformer layers update representations using context.
6. Similarity in embedding space can reflect learned relationships, but is not a guarantee of identical meaning.
Example: From token ID to vector
Suppose the tokenizer converts "AI" into token ID 1250.
An embedding layer maps ID 1250 to a vector such as:
[0.21, -0.45, 0.78, 0.12, ...]
The actual vector is learned by the model and usually has many more dimensions.
The Transformer processes this vector along with other token representations.
Real-world application
Embeddings are used in language models, semantic search, retrieval systems, recommendation systems and document similarity applications.
Key takeaway
Embeddings transform discrete token IDs into numerical vectors that neural networks can process.