Comparison intermediate · 7 min read

LoRA vs QLoRA: which adapter fine-tuning method should you use?

Quick pick

Use LoRA if you have 24GB+ VRAM and want maximum training speed and quality. Use QLoRA if you're on consumer GPUs (8-16GB) or need to fit larger models on limited hardware.

VERDICT

Use LoRA for production fine-tuning when you have adequate GPU memory: it trains 2-3x faster than QLoRA with identical final model quality. Use QLoRA if you're constrained by hardware (consumer GPU, edge device) or need to fine-tune 70B models on a single A100; QLoRA uses 75% less VRAM at the cost of 20-40% slower training speed. For most practitioners on consumer hardware, QLoRA enables fine-tuning that would otherwise be impossible.

Side-by-side comparison

Dimension	LoRA	QLoRA	Winner
Memory (7B model)	~16GB VRAM	~6-8GB VRAM	QLoRA
Training speed (7B)	~100 steps/min (A100)	~60 steps/min (A100)	LoRA
Max model size (single GPU)	13B-30B	70B+	QLoRA
Quantization required	No (FP16/BF16)	Yes (4-bit NF4)	LoRA
Final model quality	Identical to full fine-tune	Identical to full fine-tune	Tie
Ease of implementation	Standard transformers library	Requires bitsandbytes + peft	LoRA
Inference speed	Same as base model	Same as base model	Tie
Hardware flexibility	High-end GPUs required	Works on consumer GPUs (RTX 4090, M2 Pro)	QLoRA

Performance benchmarks

Memory usage (Llama 2 7B, rank=8)

LoRA ~16GB VRAM (FP16 weights)

QLoRA ~5-6GB VRAM (4-bit NF4)

QLoRA achieves 70% memory reduction via 4-bit quantization + paged optimizers. Measured on A100 with batch_size=4.

Training throughput (7B model, A100 40GB)

LoRA ~100-120 steps/min

QLoRA ~50-70 steps/min

LoRA is 2-3x faster due to no dequantization overhead per forward pass. QLoRA dequantizes on-the-fly, causing latency.

Max model size per single 80GB A100

LoRA ~30B parameters

QLoRA ~70B+ parameters

QLoRA's 4-bit quantization enables fine-tuning models that won't fit in full precision. Llama 2 70B fits on single A100 with QLoRA.

Final model quality (MMLU benchmark)

LoRA Identical to full fine-tune baseline

QLoRA Identical to full fine-tune baseline

Both methods preserve the full fine-tuning capability when using identical rank/alpha. Quality difference is unmeasurable (<0.1%).

When to use each

LoRA

✓ You have access to 24GB+ VRAM (A100 40GB, RTX 6000) and prioritize training speed: LoRA trains 2-3x faster with zero quantization overhead.
✓ Fine-tuning models up to 30B parameters where you want to minimize training complexity and avoid quantization mechanics.
✓ Production pipelines that require deterministic, reproducible training without quantization artifacts or paged optimizer quirks.
✓ Research or benchmarking work where you need maximum training throughput and can justify the hardware cost.
✓ You're already running full-precision models and want the simplest adapter setup with standard bitsandbytes optimizer.

QLoRA

✓ You're on a consumer GPU (RTX 4090, RTX 4080, M2 Pro max) with 16GB or less VRAM: QLoRA is the only viable fine-tuning method.
✓ Fine-tuning large models (70B+) where LoRA would require multi-GPU setup or cluster: QLoRA fits on a single A100 80GB.
✓ Resource-constrained environments: laptops, edge devices, or cheap cloud instances where VRAM <24GB is your limiting factor.
✓ Cost-sensitive fine-tuning where you want to use cheaper H100 PCIe (40GB) or single A100 instead of multiple GPUs.
✓ You need to fine-tune multiple models simultaneously on one GPU: QLoRA's smaller footprint enables this.

Common misconceptions

LoRA

✗ LoRA is 'full fine-tuning' and will make the model exactly as good as full fine-tuning.

✓ LoRA is a low-rank approximation. On challenging benchmarks (math, coding), LoRA can underperform full fine-tuning by 2-5%: this is most visible with rank < 16. Rank 64+ minimizes this gap but increases VRAM.

✗ LoRA works on all hardware and doesn't require special setup.

✓ LoRA still requires enough VRAM for gradient checkpointing and optimizer states. A 7B model needs ~16GB even with LoRA. If you have <8GB VRAM, LoRA won't fit: you need QLoRA.

✗ You can use LoRA with any quantized base model automatically.

✓ LoRA on quantized base models (4-bit, 8-bit) requires extra dequantization steps or custom code. Standard LoRA works best with FP16/BF16 weights. If you want LoRA + quantization, use QLoRA's integrated approach instead.

QLoRA

✗ QLoRA produces 'nearly identical' models to full fine-tuning, so there's no quality tradeoff.

✓ QLoRA produces identical *convergence* but the 4-bit quantization adds subtle noise. On some benchmarks (especially math/coding), QLoRA fine-tuned models score 1-3% lower than LoRA on the same hardware. The gap is usually acceptable but measurable.

✗ QLoRA is as fast as LoRA but just uses less memory.

✓ QLoRA is 20-40% slower per step due to on-the-fly dequantization of base model weights. Total wall-clock time to convergence is meaningfully higher. If training speed matters, LoRA is still faster.

✗ You can just install QLoRA and it 'just works' like standard LoRA.

✓ QLoRA requires bitsandbytes (CUDA-compiled, platform-specific), peft v0.4+, and careful optimizer configuration (paged_adamw_8bit vs adamw_bnb_8bit). On Windows or non-standard setups, bitsandbytes installation is a common blocker. LoRA has zero such dependencies.

Code examples

Task: Load a 7B model and set up LoRA adapters for supervised fine-tuning.

LoRA: basic fine-tuning setup

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import get_peft_model, LoraConfig, TaskType
import torch

model_name = "meta-llama/Llama-2-7b-hf"
tokenizer = AutoTokenizer.from_pretrained(model_name)
# Load base model in FP16 (standard approach for LoRA)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,  # LoRA uses full precision, no quantization
    device_map="auto"
)

# Configure LoRA adapters
lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
    task_type=TaskType.CAUSAL_LM,
    lora_dropout=0.05
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()  # LoRA-only params

# Standard training loop with HuggingFace trainer
from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="./lora_output",
    per_device_train_batch_size=4,
    num_train_epochs=3,
    learning_rate=5e-4
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset
)
trainer.train()

LoRA loads the base model in standard FP16 precision without quantization, then applies low-rank adapter matrices to just q_proj/v_proj layers. This is the simplest setup: no bitsandbytes, no 4-bit NF4 conversion.

QLoRA: fine-tuning on consumer GPU

python

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import get_peft_model, LoraConfig, TaskType
import torch

model_name = "meta-llama/Llama-2-7b-hf"
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Configure 4-bit quantization (QLoRA requires bitsandbytes)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,  # QLoRA-specific: 4-bit NF4 quantization
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4"
)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,  # Load in 4-bit, not FP16
    device_map="auto"
)

# LoRA config identical to standard LoRA
lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
    task_type=TaskType.CAUSAL_LM,
    lora_dropout=0.05
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

# Training with paged_adamw_8bit optimizer (QLoRA best practice)
from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="./qlora_output",
    per_device_train_batch_size=4,
    optim="paged_adamw_8bit",  # QLoRA-specific: memory-efficient optimizer
    num_train_epochs=3,
    learning_rate=5e-4
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset
)
trainer.train()

QLoRA loads the base model in 4-bit NF4 quantization (via BitsAndBytesConfig) before applying LoRA adapters. The paged_adamw_8bit optimizer and double_quant further reduce memory. This setup trades training speed for 75% VRAM savings.

Migration path

Migrating from LoRA to QLoRA (or vice versa) requires minimal code changes, as both use the same PEFT adapter interface: **LoRA → QLoRA:** 1. Add BitsAndBytesConfig with load_in_4bit=True to your model loading step. 2. Change the model.from_pretrained() call to include quantization_config parameter. 3. Update training_args.optim to "paged_adamw_8bit" (optional but recommended for memory). 4. Everything else: LoRA config, training loop, save/load: stays identical. 5. Adapters trained with QLoRA can be loaded and merged with a quantized base model; adapters are format-agnostic. **QLoRA → LoRA:** 1. Remove BitsAndBytesConfig entirely. 2. Load model with torch_dtype=torch.float16 and device_map="auto" (no quantization_config). 3. Change optimizer back to default "adamw_torch" or "adamw_8bit" (non-paged). 4. Keep LoRA config unchanged. 5. Training will be 2-3x faster; VRAM requirement increases from 6GB to ~16GB. Note: Pre-trained QLoRA adapters merge identically into both quantized and full-precision base models. You can train on QLoRA, then deploy merged weights on a full-precision model without quality loss.

RECOMMENDATION

Use LoRA if you have 24GB+ VRAM and want maximum training speed (2-3x faster) with zero quantization complexity. Use QLoRA if you're on consumer hardware (RTX 4090, M2 Pro) or need to fine-tune large models (70B+): the 20-40% training slowdown is worth fitting work that otherwise wouldn't fit at all. For most practitioners constrained by GPU memory, QLoRA is the practical choice despite being slower.

Verified 2026-04

Verify ↗

Community Notes

No notes yetBe the first to share a version-specific fix or tip.