1.5.10
Mixed Precision Training
What is it?
Mixed precision training uses more than one numerical precision during model training. It can use lower-precision formats for selected computations while retaining higher precision where needed. This can reduce memory consumption and improve computational efficiency on compatible hardware, while requiring care to maintain numerical stability.
Key points
1. Lower-precision arithmetic can reduce memory usage.
2. Some operations benefit from higher precision.
3. FP16 and BF16 are common reduced-precision formats.
4. FP32 is used for higher-precision computation and storage where needed.
5. Loss scaling can help prevent underflow in FP16 training.
6. Hardware and software support influence performance.
Example: Mixed precision computation
A training workflow may use:
Matrix multiplication: BF16
Selected sensitive operations: FP32
Optimizer states: Depending on the training implementation
The exact precision assignment depends on the model, framework and hardware.
The goal is to improve efficiency without unacceptable numerical errors.
Real-world application
Mixed precision is useful when training large models under GPU memory and compute constraints.
Key takeaway
Mixed precision balances computational efficiency and numerical stability by using different precisions for different operations.