1.9.3
LoRA Variants
What is it?
LoRA variants extend the original low-rank adaptation method to improve memory efficiency, optimization, generalization or parameter efficiency. Different variants address different limitations of standard LoRA.
Key points
1. QLoRA: Uses a 4-bit quantized frozen base model with trainable LoRA adapters, commonly using NF4 and double quantization.
2. DoRA: Separates weight magnitude and direction, applying the low-rank update to direction.
3. LoRA+: Uses different learning rates for matrices A and B.
4. AdaLoRA: Dynamically allocates rank across layers using importance estimates.
5. rsLoRA: Uses alpha divided by square root of rank for improved stability at high ranks.
6. VeRA: Uses shared frozen random matrices and trains scaling vectors.
7. LoRA-FA: Freezes matrix A after initialization and trains matrix B.
Comparing LoRA variants
QLoRA: Useful when GPU memory is limited. The book describes a 70B model fine-tuned on a single 48GB GPU using QLoRA.
DoRA: Separates weight magnitude from direction to improve adaptation.
LoRA+: Uses a higher learning rate for B than A; the book describes a ratio of approximately 16:1.
AdaLoRA: Allocates rank dynamically across layers.
rsLoRA: Uses alpha/√r scaling, particularly useful at high ranks.
VeRA: Reduces trainable parameters by sharing frozen random matrices.
LoRA-FA: Reduces memory by freezing A.
Real-world application
Select a LoRA variant based on the main constraint. Consider QLoRA for limited GPU memory, AdaLoRA for adaptive rank allocation and VeRA for very small task-specific adapters.
Key takeaway
LoRA variants improve different aspects of adaptation. Select the variant according to memory, capacity and optimization requirements.