RuntimeError
torch.nn.modules.module.RuntimeError
Stack trace
RuntimeError: Error(s) in loading state_dict for PeftModel: Missing key(s) in state_dict: "lora_adapter.weight", "lora_adapter.bias"... Unexpected key(s) in state_dict: "base_model.weight"... size mismatch for lora_adapter.weight: copying a param with shape [64, 64] from checkpoint, the shape in current model is [32, 64].
Why it happens
The PEFT adapter load state dict error happens because the adapter checkpoint's parameter keys or shapes do not align with the base model's architecture. This mismatch can occur if the adapter was trained on a different model version, or if the base model has changed since the adapter was saved.
Detection
Check the error logs for missing or unexpected keys and size mismatches when loading the adapter state dict. Validate that the adapter and base model versions match before loading.
Causes & fixes
Adapter checkpoint was saved from a different base model architecture or version.
Ensure the base model used to load the adapter matches exactly the model used during adapter training, including version and configuration.
Using a PEFT adapter checkpoint incompatible with the current PEFT or transformers library version.
Upgrade or downgrade the PEFT and transformers libraries to the versions compatible with the adapter checkpoint or re-export the adapter with the current library versions.
Partial or corrupted adapter checkpoint missing required keys.
Verify the adapter checkpoint file integrity and re-download or re-save the adapter checkpoint if corrupted.
Loading the adapter state dict into the full base model instead of the PEFT wrapper model.
Load the adapter checkpoint using the PEFT model wrapper (e.g., PeftModel.from_pretrained) rather than directly into the base model.
Code: broken vs fixed
from transformers import AutoModel
from peft import PeftModel
model = AutoModel.from_pretrained('bert-base-uncased')
adapter_path = './adapter_checkpoint'
# This line causes the error
model.load_state_dict(torch.load(adapter_path))import os
import torch
from transformers import AutoModel
from peft import PeftModel
os.environ['TRANSFORMERS_CACHE'] = '/path/to/cache'
base_model = AutoModel.from_pretrained('bert-base-uncased')
adapter_path = './adapter_checkpoint'
# Correctly load adapter using PeftModel wrapper
model = PeftModel.from_pretrained(base_model, adapter_path)
print('Adapter loaded successfully')Workaround
Catch the RuntimeError and manually filter or map the state dict keys to match the base model, but this is error-prone and not recommended for production.
Prevention
Always save and load PEFT adapters using the PeftModel API with matching base model versions and keep PEFT and transformers libraries in sync to avoid state dict mismatches.