OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / SFT Best Practices and Techniques / Data Mixing Strategies for Multi-Task SFT
Lesson 393 of 415 6 min read
10.4.2

Proportional Dataset Mixing

Calculate dataset sampling probabilities according to each dataset size.

What is it?

Proportional mixing samples examples according to each dataset’s share of the total number of examples. The probability of selecting dataset k is its size divided by the combined size of all datasets.

Key points

The probability is pk = Nk / ΣNj. Nk is the number of examples in dataset k. The denominator is the total number of examples across all datasets. Larger datasets contribute more samples. This is the default approach in many frameworks and works well when dataset quality is similar.

Calculating Proportional Sampling

Suppose the training datasets contain: Math: 5,000 examples Code: 3,000 examples General: 2,000 examples Total: 10,000 examples Math probability = 5,000 / 10,000 = 0.50 Code probability = 3,000 / 10,000 = 0.30 General probability = 2,000 / 10,000 = 0.20 The sampling proportions are 50%, 30% and 20%.

Real-world application

Use proportional mixing when datasets have reasonably similar quality and their relative sizes reflect the desired training contribution.
Key takeaway

Proportional mixing gives larger datasets a larger share of the training samples.