OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / LLM Architecture and Optimization Methods / Tokenization
Lesson 7 of 415 6 min read
1.2.3

Other Tokenization Methods

What is it?

BPE is not the only way to create subword tokens. Other approaches include WordPiece, Unigram Language Model tokenization and byte-level tokenization. These methods use different strategies to construct vocabularies and split text. The choice of tokenizer affects how text is represented and how many tokens are required.

Key points

1. WordPiece selects subword units using a vocabulary-learning approach based on likelihood. 2. Unigram tokenization starts with a larger vocabulary and removes less useful pieces according to a probabilistic model. 3. Byte-level tokenization represents text using bytes and can handle arbitrary input text. 4. Different tokenizers may split the same sentence differently. 5. Tokenization affects sequence length and model input cost.

Example: Different tokenizers, different splits

Consider the word "unhappiness". One tokenizer might split it as: un | happiness Another might split it as: un | happi | ness A character-based representation might split it into individual characters. These are illustrative possibilities. Actual output depends on the tokenizer vocabulary and algorithm.

Real-world application

Understanding tokenizer differences is useful when selecting a model, estimating token usage, preparing multilingual datasets and comparing model input limits.
Key takeaway

Different tokenization algorithms create different token boundaries. The tokenizer used by a model matters when preparing its input.