1.2.4
Tokenization Best Practices
What is it?
Good tokenization practices help ensure that text is processed consistently and efficiently. Developers should use the tokenizer associated with the selected model, inspect how important text is split, and check token counts before sending large inputs. They should also consider multilingual content, special tokens and the model context limit.
Key points
1. Use the tokenizer designed for the selected model.
2. Check token counts rather than estimating from word counts.
3. Test technical terms, names, numbers and punctuation.
4. Check multilingual text where relevant.
5. Account for special tokens and prompt structure.
6. Keep input length within the model context limit.
7. Avoid assuming different models share the same tokenizer.
Example: Checking input length
Suppose an application sends a long customer-support conversation to an LLM.
Before sending it, the application can:
1. Count the tokens.
2. Check the model context limit.
3. Remove irrelevant conversation history if necessary.
4. Preserve important instructions and recent messages.
5. Send the prepared input to the model.
Real-world application
These practices are useful when building chatbots, document-processing systems, retrieval applications and AI agents that handle long conversations.
Key takeaway
Always measure token usage with the actual model tokenizer instead of relying on word counts.