1.2.1
Why Not Characters or Words?
What is it?
Language models need to convert text into a numerical representation before processing it. One approach would be to split text into individual characters or complete words. However, character-level tokenization creates very long sequences, while word-level tokenization struggles with unfamiliar words, spelling variations and large vocabularies. Subword tokenization provides a practical balance.
Key points
1. Character tokenization splits text into individual characters.
2. Word tokenization treats complete words as individual units.
3. Character-level sequences can become very long.
4. Word-level vocabularies can become extremely large.
5. Unseen words create difficulties for word-level tokenizers.
6. Subword tokenization combines smaller units to represent many words.
Example: Three ways to tokenize a sentence
Sentence: "Learning models are powerful."
Character-level:
L | e | a | r | n | i | n | g | ...
Word-level:
Learning | models | are | powerful
Subword-level:
Learn | ing | model | s | are | power | ful
The exact subword divisions depend on the tokenizer.
Real-world application
Subword tokenization helps language models process unfamiliar names, technical vocabulary, different word forms and multilingual text without requiring a separate vocabulary entry for every possible word.
Key takeaway
Tokenization must balance vocabulary size and sequence length. Subword methods provide a practical compromise between character-level and word-level approaches.