OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / LLM Architecture and Optimization Methods / Tokenization
Lesson 5 of 415 5 min read
1.2.1

Why Not Characters or Words?

What is it?

Language models need to convert text into a numerical representation before processing it. One approach would be to split text into individual characters or complete words. However, character-level tokenization creates very long sequences, while word-level tokenization struggles with unfamiliar words, spelling variations and large vocabularies. Subword tokenization provides a practical balance.

Key points

1. Character tokenization splits text into individual characters. 2. Word tokenization treats complete words as individual units. 3. Character-level sequences can become very long. 4. Word-level vocabularies can become extremely large. 5. Unseen words create difficulties for word-level tokenizers. 6. Subword tokenization combines smaller units to represent many words.

Example: Three ways to tokenize a sentence

Sentence: "Learning models are powerful." Character-level: L | e | a | r | n | i | n | g | ... Word-level: Learning | models | are | powerful Subword-level: Learn | ing | model | s | are | power | ful The exact subword divisions depend on the tokenizer.

Real-world application

Subword tokenization helps language models process unfamiliar names, technical vocabulary, different word forms and multilingual text without requiring a separate vocabulary entry for every possible word.
Key takeaway

Tokenization must balance vocabulary size and sequence length. Subword methods provide a practical compromise between character-level and word-level approaches.