10.4.2
Proportional Dataset Mixing
Calculate dataset sampling probabilities according to each dataset size.
What is it?
Proportional mixing samples examples according to each dataset’s share of the total number of examples. The probability of selecting dataset k is its size divided by the combined size of all datasets.
Key points
The probability is pk = Nk / ΣNj.
Nk is the number of examples in dataset k.
The denominator is the total number of examples across all datasets.
Larger datasets contribute more samples.
This is the default approach in many frameworks and works well when dataset quality is similar.
Calculating Proportional Sampling
Suppose the training datasets contain:
Math: 5,000 examples
Code: 3,000 examples
General: 2,000 examples
Total: 10,000 examples
Math probability = 5,000 / 10,000 = 0.50
Code probability = 3,000 / 10,000 = 0.30
General probability = 2,000 / 10,000 = 0.20
The sampling proportions are 50%, 30% and 20%.
Real-world application
Use proportional mixing when datasets have reasonably similar quality and their relative sizes reflect the desired training contribution.
Key takeaway
Proportional mixing gives larger datasets a larger share of the training samples.