1.2.3
Other Tokenization Methods
What is it?
BPE is not the only way to create subword tokens. Other approaches include WordPiece, Unigram Language Model tokenization and byte-level tokenization. These methods use different strategies to construct vocabularies and split text. The choice of tokenizer affects how text is represented and how many tokens are required.
Key points
1. WordPiece selects subword units using a vocabulary-learning approach based on likelihood.
2. Unigram tokenization starts with a larger vocabulary and removes less useful pieces according to a probabilistic model.
3. Byte-level tokenization represents text using bytes and can handle arbitrary input text.
4. Different tokenizers may split the same sentence differently.
5. Tokenization affects sequence length and model input cost.
Example: Different tokenizers, different splits
Consider the word "unhappiness".
One tokenizer might split it as:
un | happiness
Another might split it as:
un | happi | ness
A character-based representation might split it into individual characters.
These are illustrative possibilities. Actual output depends on the tokenizer vocabulary and algorithm.
Real-world application
Understanding tokenizer differences is useful when selecting a model, estimating token usage, preparing multilingual datasets and comparing model input limits.
Key takeaway
Different tokenization algorithms create different token boundaries. The tokenizer used by a model matters when preparing its input.