RuntimeError
torch._C._RuntimeError
Stack trace
RuntimeError: int8 quantization is not supported on this CPU or the quantization backend is not set correctly.
File "model_quant.py", line 42, in <module>
model = torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8) # triggers error
File "/usr/local/lib/python3.9/site-packages/torch/quantization/quantize.py", line 123, in quantize_dynamic
raise RuntimeError("int8 quantization CPU not supported error") Why it happens
Int8 quantization requires CPU support for specific instructions like AVX2 or VNNI. If the CPU lacks these or if PyTorch's quantization backend is not set to a supported one (e.g., fbgemm for x86), this error is raised. It can also happen if the environment or PyTorch build does not include quantization support.
Detection
Check CPU capabilities using tools like lscpu or cpuinfo and verify PyTorch quantization backend settings before running quantized models to catch unsupported configurations early.
Causes & fixes
CPU hardware does not support int8 quantization instructions (e.g., lacks AVX2 or VNNI)
Run the model on a CPU that supports these instructions or switch to a GPU backend that supports quantization.
PyTorch quantization backend is not set or set incorrectly (e.g., backend is 'qnnpack' on x86 CPU)
Set the quantization backend to 'fbgemm' for x86 CPUs by calling torch.backends.quantized.engine = 'fbgemm' before quantization.
PyTorch installation does not include quantization support or is outdated
Upgrade PyTorch to a version that supports quantization and ensure it is installed with CPU quantization capabilities.
Code: broken vs fixed
import torch
model = torch.nn.Linear(10, 5)
# This line triggers the error on unsupported CPU
quantized_model = torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)import os
import torch
# Set quantization backend to fbgemm for CPU support
torch.backends.quantized.engine = 'fbgemm'
model = torch.nn.Linear(10, 5)
quantized_model = torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8) # fixed
print("Quantization successful on supported CPU")Workaround
If you cannot change hardware or backend, run the model without quantization or use float32 precision as a temporary fallback.
Prevention
Verify CPU instruction support and explicitly set the quantization backend in your environment before deploying quantized models to avoid unsupported runtime errors.