1.9.1
The LoRA Insight
What is it?
LoRA (Low-Rank Adaptation) reduces fine-tuning cost by freezing the original model weights and learning a small low-rank update using two smaller matrices, A and B. Instead of updating the entire weight matrix, only these additional matrices are trained.
Key points
1. The original weight matrix W remains frozen.
2. Two trainable matrices A and B represent the weight update.
3. The update is scaled using alpha divided by rank.
4. Trainable parameters are significantly reduced.
5. The learned update can be merged into the original weights during inference.
6. LoRA exploits the low intrinsic dimensionality of many fine-tuning tasks.
Understanding the LoRA weight update
Original weight matrix: W
LoRA update: ΔW = (α / r) × BA
Updated weight: W′ = W + (α / r) × BA
Suppose d = 4096 and rank r = 16.
Full matrix parameters: 4096 × 4096 = 16,777,216
LoRA parameters: 2 × 4096 × 16 = 131,072
Only the smaller A and B matrices need to be trained.
Real-world application
Use LoRA when adapting a large pretrained model without updating all its parameters. Freeze the base model and train adapter matrices for the target task.
Key takeaway
LoRA reduces the number of trainable parameters by learning a low-rank update instead of changing the complete weight matrix.