ValueError
bitsandbytes.nn.modules.ValueError
Stack trace
ValueError: load_in_4bit must be set to True to load a 4-bit quantized model with bitsandbytes
File "/usr/local/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 123, in from_pretrained
raise ValueError("load_in_4bit must be True for 4-bit quantized models") Why it happens
BitsAndBytes requires the load_in_4bit parameter explicitly set to True when loading 4-bit quantized models. If this flag is missing or set to False, the library cannot properly configure the quantization layers, causing a ValueError. This often happens when upgrading bitsandbytes or using incompatible model loading code.
Detection
Check for ValueError exceptions during model loading that mention load_in_4bit. Add logging around model loading calls to verify the presence and value of load_in_4bit in the config or kwargs.
Causes & fixes
The load_in_4bit parameter is not set or set to False when loading a 4-bit quantized model.
Explicitly pass load_in_4bit=True in the model loading function, e.g., from_pretrained(..., load_in_4bit=True).
Using an outdated bitsandbytes version that does not support the current load_in_4bit API.
Upgrade bitsandbytes to the latest stable version (>=0.39.0) that supports load_in_4bit parameter.
Model checkpoint is not actually quantized to 4-bit but load_in_4bit=True is forced.
Verify the model checkpoint supports 4-bit quantization before setting load_in_4bit=True to avoid config mismatch.
Code: broken vs fixed
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained('model-name') # triggers ValueErrorimport os
from transformers import AutoModelForCausalLM
os.environ['BITSANDBYTES_NOWELCOME'] = '1' # optional to suppress warnings
model = AutoModelForCausalLM.from_pretrained('model-name', load_in_4bit=True) # fixed by adding load_in_4bit
print('Model loaded with 4-bit quantization')Workaround
If you cannot set load_in_4bit=True immediately, load the model without quantization and then apply manual quantization steps or convert the model offline before loading.
Prevention
Always verify your model checkpoint's quantization format and explicitly set load_in_4bit=True when loading 4-bit quantized models with bitsandbytes to avoid config errors.