load_in_4bit: extreme compression
Why this matters
Modern LLMs like Llama 2 70B or Mixtral require 140GB+ in full precision: impossible on consumer GPUs. 4-bit quantization lets you run state-of-the-art models on a single 24GB GPU (RTX 4090) or even smaller hardware, making fine-tuning and inference economically viable.
Explanation
What it is: 4-bit quantization represents model weights using only 4 bits (0-15 range) instead of 32 bits (float32), reducing memory footprint by 87.5%. The Hugging Face BitsAndBytesConfig paired with load_in_4bit=True handles this compression automatically during model loading using the bitsandbytes library.
How it works mechanically: When you set load_in_4bit=True, Hugging Face intercepts the model loading process. Before weights are copied to GPU memory, bitsandbytes quantizes them: it scales each weight tensor to fit in 4 bits (storing an additional scale factor and offset). During inference, weights are dequantized on-the-fly in GPU memory: this is asymmetric quantization (weights compressed, computation in higher precision). The bnb_4bit_quant_type parameter controls whether quantization is symmetric (all weights scaled identically) or NF4 (normalized float 4, optimized for normal distributions of weights).
When to use it: Use 4-bit loading when: (1) you want to run large models on limited hardware, (2) inference latency is acceptable (dequantization adds ~5-10% overhead), (3) you're using inference-only pipelines or QLoRA fine-tuning (which uses LoRA adapters on top of frozen 4-bit weights).
Analogy
Imagine a high-resolution image (float32 weights) stored in full color. 4-bit quantization is like converting it to a 16-color palette with a separate brightness map: the palette is tiny, but you restore colors from the map during viewing. You lose some detail, but the file is 1/8th the size and displays instantly on old hardware.
Code
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
quant_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True
)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-2-7b-hf",
quantization_config=quant_config,
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
inputs = tokenizer("What is 2+2?", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)
result = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(result)
print(f"\nModel dtype: {model.dtype}")
print(f"Model device: {model.device}")
print(f"Estimated memory: ~7B * 0.5 bytes (4-bit + overhead) ≈ 3.5GB")What is 2+2? The answer is 4. It is a simple arithmetic operation. Model dtype: torch.float16 Model device: cuda:0 Estimated memory: ~7B * 0.5 bytes (4-bit + overhead) ≈ 3.5GB
What just happened?
We created a BitsAndBytesConfig specifying 4-bit quantization with NF4 quantization type and double quantization (quantize the scale factors themselves). We loaded a 7B parameter model using `from_pretrained()` with this config: Hugging Face automatically compressed the weights to 4-bit during loading. The model was placed on GPU via `device_map='auto'`. We then tokenized an input, ran generation (weights were dequantized on-the-fly during forward passes), and decoded the output. The actual model weights in memory are 4-bit, but computation happened in bfloat16 precision.
Common gotcha
Developers often set `load_in_4bit=True` but forget to include `quantization_config=BitsAndBytesConfig(...)` with proper dtype settings. Without the config, quantization may use incompatible dtypes for computation, causing silent numerical errors or GPU memory spikes. Also: 4-bit models cannot be saved directly with `model.save_pretrained()`: only LoRA adapters trained on top can be saved. If you need to persist the quantized model, you must quantize again on next load.
Error recovery
RuntimeError: Expected scalar type Double but found HalfAttributeError: type object 'BitsAndBytesConfig' has no attribute 'load_in_4bit'CUDA out of memory even with load_in_4bit=TrueValueError: Attempting to deserialize object but no `_builtin_model_class` was stored in the configExperienced dev note
A subtle gotcha: double quantization (`bnb_4bit_use_double_quant=True`) saves another ~0.4 bits per weight by quantizing the scale factors themselves, but it adds ~5-10% latency overhead. For inference latency-critical applications (sub-100ms SLA), disable it. For training LoRA adapters where you run fewer forward passes, enable it: the memory savings outweigh the latency cost. Also: the `bnb_4bit_quant_type='nf4'` (normalized float 4) is almost always better than 'fp4' because weight distributions are typically normal, not uniform: but fp4 is slightly faster if you're bandwidth-limited. Test both in your actual use case.
Check your understanding
Your model loads successfully in 4-bit but inference is slower than expected. You used `bnb_4bit_use_double_quant=True`. Explain why this overhead exists and why it doesn't apply equally to weight loading vs. inference.
Show answer hint
A correct answer recognizes that double quantization saves memory at load time (one-time cost) but adds per-inference overhead during dequantization (repetitive cost). The scale factors must be dequantized every forward pass, not just once. This is a memory-latency tradeoff that depends on whether you prioritize VRAM or throughput.