OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / LLM Architecture and Optimization Methods / LoRA and Parameter-Efficient Fine-Tuning
Lesson 57 of 415 7 min read
1.9.2

LoRA Hyperparameters

What is it?

LoRA hyperparameters determine the capacity, scaling and training behaviour of adapters. Rank controls the size of the update, alpha controls its scaling, target modules determine where adapters are applied, and dropout provides regularization.

Key points

1. Rank (r): Common values are 8, 16, 32 and 64. Higher rank provides more capacity but uses more memory. 2. Alpha: Controls update magnitude through alpha/r scaling. 3. Target modules: Common choices include q_proj, k_proj, v_proj and o_proj. 4. Dropout: Common range is 0.0–0.1; 0.05 is often used for small datasets. 5. Bias: The common setting is none. 6. Learning rate: The book gives a typical range of 1e-4 to 3e-4. 7. Start with rank 16 for general-purpose fine-tuning.

Choosing a LoRA rank

Example rank selection: Rank 8: Simple classification or single-domain chat. Rank 16: General-purpose fine-tuning. Rank 32–64: More complex mathematics, coding and multi-turn reasoning tasks. Rank 128+: Diminishing returns may make full fine-tuning or higher-rank QLoRA worth evaluating. Example configuration: r = 16 lora_alpha = 32 target_modules = ["q_proj", "k_proj", "v_proj", "o_proj"] lora_dropout = 0.05 bias = "none"

Real-world application

Begin with rank 16 and evaluate training loss and task performance. Increase rank if the adapter appears too constrained. Tune alpha, target modules and learning rate alongside rank.
Key takeaway

Choose LoRA hyperparameters according to task complexity, available memory and training results.