How to Intermediate · 4 min read

INT8 quantization with ONNX

Q: INT8 quantization with ONNX

Use onnxruntime and onnxruntime.quantization tools to apply INT8 quantization on ONNX models. The quantize_dynamic or quantize_static functions convert model weights and activations to INT8, improving inference speed and reducing model size.

Quick answer

Use onnxruntime and onnxruntime.quantization tools to apply INT8 quantization on ONNX models. The quantize_dynamic or quantize_static functions convert model weights and activations to INT8, improving inference speed and reducing model size.

PREREQUISITES

Python 3.8+
pip install onnx onnxruntime onnxruntime-tools

Setup

Install the required packages for ONNX quantization using pip. You need onnx for model handling and onnxruntime with quantization tools for applying INT8 quantization.

bash

pip install onnx onnxruntime onnxruntime-tools

Step by step

This example shows how to apply dynamic INT8 quantization to an existing ONNX model using onnxruntime.quantization.quantize_dynamic. Dynamic quantization converts weights to INT8 and quantizes activations on the fly during inference.

python

import os
from onnxruntime.quantization import quantize_dynamic, QuantType

# Path to your original ONNX model
original_model_path = "model.onnx"
# Path to save the quantized INT8 model
quantized_model_path = "model_int8.onnx"

# Apply dynamic quantization
quantize_dynamic(
    model_input=original_model_path,
    model_output=quantized_model_path,
    weight_type=QuantType.QInt8  # INT8 quantization for weights
)

print(f"Quantized model saved to: {quantized_model_path}")

output

Quantized model saved to: model_int8.onnx

Common variations

Besides dynamic quantization, you can perform static quantization which requires calibration data to quantize activations more precisely. Use quantize_static with a calibration data loader.

Example for static quantization:

python

from onnxruntime.quantization import quantize_static, CalibrationDataReader
import onnx

class DummyCalibrationDataReader(CalibrationDataReader):
    def get_next(self):
        # Return input data dictionary for calibration or None when done
        return None

# Load your ONNX model
model_fp32 = onnx.load("model.onnx")

# Create calibration data reader
calibration_data_reader = DummyCalibrationDataReader()

# Apply static quantization
quantize_static(
    model_input="model.onnx",
    model_output="model_int8_static.onnx",
    calibration_data_reader=calibration_data_reader,
    quant_format=QuantType.QOperator  # INT8 quantization
)

print("Static quantized model saved to model_int8_static.onnx")

output

Static quantized model saved to model_int8_static.onnx

Troubleshooting

If you get errors about missing calibration data during static quantization, ensure your CalibrationDataReader provides valid input samples.
Check that your ONNX model is compatible with quantization by verifying opset version & supported operators.
For performance issues, try dynamic quantization first as it requires no calibration and is simpler.

✅

Key Takeaways

Use onnxruntime.quantization.quantize_dynamic for quick INT8 quantization without calibration.
Static quantization with calibration improves accuracy but requires input data samples.
Verify ONNX model compatibility and opset version before quantization.
INT8 quantization reduces model size and speeds up inference on supported hardware.

Verified 2026-04 · onnxruntime.quantization.quantize_dynamic, onnxruntime.quantization.quantize_static

Verify ↗