OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / LLM Architecture and Optimization Methods / LoRA and Parameter-Efficient Fine-Tuning
Lesson 58 of 415 8 min read
1.9.3

LoRA Variants

What is it?

LoRA variants extend the original low-rank adaptation method to improve memory efficiency, optimization, generalization or parameter efficiency. Different variants address different limitations of standard LoRA.

Key points

1. QLoRA: Uses a 4-bit quantized frozen base model with trainable LoRA adapters, commonly using NF4 and double quantization. 2. DoRA: Separates weight magnitude and direction, applying the low-rank update to direction. 3. LoRA+: Uses different learning rates for matrices A and B. 4. AdaLoRA: Dynamically allocates rank across layers using importance estimates. 5. rsLoRA: Uses alpha divided by square root of rank for improved stability at high ranks. 6. VeRA: Uses shared frozen random matrices and trains scaling vectors. 7. LoRA-FA: Freezes matrix A after initialization and trains matrix B.

Comparing LoRA variants

QLoRA: Useful when GPU memory is limited. The book describes a 70B model fine-tuned on a single 48GB GPU using QLoRA. DoRA: Separates weight magnitude from direction to improve adaptation. LoRA+: Uses a higher learning rate for B than A; the book describes a ratio of approximately 16:1. AdaLoRA: Allocates rank dynamically across layers. rsLoRA: Uses alpha/√r scaling, particularly useful at high ranks. VeRA: Reduces trainable parameters by sharing frozen random matrices. LoRA-FA: Reduces memory by freezing A.

Real-world application

Select a LoRA variant based on the main constraint. Consider QLoRA for limited GPU memory, AdaLoRA for adaptive rank allocation and VeRA for very small task-specific adapters.
Key takeaway

LoRA variants improve different aspects of adaptation. Select the variant according to memory, capacity and optimization requirements.