OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / LLM Architecture and Optimization Methods / Pretraining: Best Practices
Lesson 46 of 415 6 min read
1.7.2

Data Pipeline

What is it?

A pretraining data pipeline collects, filters, deduplicates and combines text from different sources before training. Data quality and domain balance are important because the model learns patterns from the data it receives.

Key points

1. Collect data from web pages, code, books, research papers and curated sources. 2. Remove duplicates using methods such as MinHash and exact substring matching. 3. Apply quality filters, including language, length, toxicity and perplexity-based checks. 4. Mix domains using sampling strategies. 5. Give additional weight to code and mathematics when reasoning capability is a priority.

Building a pretraining dataset

Illustrative data mixture from the book: Web crawl: 80% Code: 10% Books and papers: 5% Curated data: 5% The book also describes frontier pretraining datasets ranging from 1 to 15 trillion tokens.

Real-world application

Design a repeatable data pipeline that records data sources, removes duplicates, applies quality checks and controls the sampling proportions used during training.
Key takeaway

A large dataset is not automatically a good dataset. Quality, diversity and data mixing matter.