ValueError
vllm.LLMError: Quantization method not supported
Stack trace
Traceback (most recent call last):
File "app.py", line 15, in <module>
llm = LLM(model="my-model", quantization_method="q4_0") # triggers error
File "/usr/local/lib/python3.9/site-packages/vllm/llm.py", line 120, in __init__
raise ValueError("Quantization method not supported")
ValueError: Quantization method not supported Why it happens
vLLM supports only specific quantization methods compatible with the loaded model and the installed vLLM version. Using an unsupported or misspelled quantization method causes this error during model initialization.
Detection
Check the quantization_method parameter passed to vLLM's LLM constructor; validate it against the supported methods listed in the vLLM documentation before instantiating the model.
Causes & fixes
Using a quantization method string not supported by the installed vLLM version.
Verify the supported quantization methods for your vLLM version and use only those exact strings, e.g., 'int8', 'int4', or omit the parameter if unsure.
Typo or incorrect casing in the quantization method name.
Correct the quantization method string to match exactly the supported method names, which are case-sensitive.
Attempting to quantize a model that does not support quantization or requires a different method.
Check the model documentation for compatible quantization methods and select one supported by both the model and vLLM.
Code: broken vs fixed
from vllm import LLM
llm = LLM(model="my-model", quantization_method="q4_0") # triggers ValueError
print(llm.generate("Hello"))import os
from vllm import LLM
# Removed unsupported quantization_method to fix error
llm = LLM(model=os.environ["VLLM_MODEL"])
print(llm.generate("Hello"))Workaround
Catch the ValueError when initializing LLM and fallback to creating the model without quantization or with a default supported method.
Prevention
Always consult the vLLM documentation for supported quantization methods for your model and vLLM version before setting this parameter.