1.9.2
LoRA Hyperparameters
What is it?
LoRA hyperparameters determine the capacity, scaling and training behaviour of adapters. Rank controls the size of the update, alpha controls its scaling, target modules determine where adapters are applied, and dropout provides regularization.
Key points
1. Rank (r): Common values are 8, 16, 32 and 64. Higher rank provides more capacity but uses more memory.
2. Alpha: Controls update magnitude through alpha/r scaling.
3. Target modules: Common choices include q_proj, k_proj, v_proj and o_proj.
4. Dropout: Common range is 0.0–0.1; 0.05 is often used for small datasets.
5. Bias: The common setting is none.
6. Learning rate: The book gives a typical range of 1e-4 to 3e-4.
7. Start with rank 16 for general-purpose fine-tuning.
Choosing a LoRA rank
Example rank selection:
Rank 8: Simple classification or single-domain chat.
Rank 16: General-purpose fine-tuning.
Rank 32–64: More complex mathematics, coding and multi-turn reasoning tasks.
Rank 128+: Diminishing returns may make full fine-tuning or higher-rank QLoRA worth evaluating.
Example configuration:
r = 16
lora_alpha = 32
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj"]
lora_dropout = 0.05
bias = "none"
Real-world application
Begin with rank 16 and evaluate training loss and task performance. Increase rank if the adapter appears too constrained. Tune alpha, target modules and learning rate alongside rank.
Key takeaway
Choose LoRA hyperparameters according to task complexity, available memory and training results.