What are target modules in LoRA
LoRA (Low-Rank Adaptation), target modules are the specific neural network layers or weight matrices where low-rank update matrices are injected to adapt a pretrained model efficiently. These modules are chosen because modifying them allows fine-tuning with fewer parameters and less compute while preserving the original model weights.Target modules in LoRA are specific neural network layers that receive low-rank adaptation matrices to enable efficient fine-tuning of large language models.How it works
LoRA fine-tunes large models by freezing the original weights and injecting small, trainable low-rank matrices into selected target modules. These modules are typically weight matrices in transformer layers, such as query, key, value, or feed-forward layers. By updating only these low-rank matrices, LoRA drastically reduces the number of trainable parameters.
Think of the original model as a complex machine with many gears (weights). Instead of replacing or adjusting all gears, LoRA adds small auxiliary gears (target modules) that subtly change the machine's behavior without rebuilding it.
Concrete example
In a transformer model, common target modules for LoRA are the weight matrices of the attention mechanism's query and value projections. For example, if the original query weight matrix is W_q, LoRA adds two low-rank matrices A and B such that the adapted weight is W_q + BA.
import os
from openai import OpenAI
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
# Pseudocode for specifying target modules in LoRA fine-tuning config
lora_config = {
"target_modules": ["q_proj", "v_proj"], # query and value projection layers
"r": 8, # low-rank dimension
"alpha": 16,
"dropout": 0.1
}
print("Target modules for LoRA:", lora_config["target_modules"])Target modules for LoRA: ['q_proj', 'v_proj']
When to use it
Use target modules in LoRA when you want to fine-tune large language models efficiently with limited compute and memory. Selecting appropriate target modules (like attention projections or feed-forward layers) balances adaptation capacity and parameter efficiency.
Do not use LoRA target modules if you need full model retraining or if the model architecture does not expose modular weight matrices suitable for low-rank updates.
Key terms
| Term | Definition |
|---|---|
| LoRA | Low-Rank Adaptation technique for efficient fine-tuning by injecting low-rank matrices. |
| Target modules | Specific neural network layers where LoRA inserts trainable low-rank matrices. |
| Low-rank matrices | Small parameter matrices (A and B) used to adapt pretrained weights. |
| Query projection (q_proj) | Weight matrix projecting input to query vectors in attention. |
| Value projection (v_proj) | Weight matrix projecting input to value vectors in attention. |
Key Takeaways
- Target modules are the neural network layers where LoRA applies low-rank updates to adapt a model efficiently.
- Choosing query and value projection layers as target modules is common for transformer-based models.
- LoRA fine-tunes only these modules, keeping the original model weights frozen to save compute and memory.