RuntimeError
litellm.errors.RuntimeError: router no healthy deployments available
Stack trace
Traceback (most recent call last):
File "app.py", line 42, in <module>
response = client.chat.completions.create(model="litellm-router", messages=messages)
File "/usr/local/lib/python3.10/site-packages/litellm/client.py", line 88, in create
raise RuntimeError("router no healthy deployments available")
litellm.errors.RuntimeError: router no healthy deployments available Why it happens
LiteLLM's router component manages routing requests to available model deployments. This error occurs when no deployments are registered as healthy or available, often due to deployment failures, network issues, or misconfiguration in the deployment registry.
Detection
Monitor router health endpoints and deployment status regularly; catch RuntimeError exceptions from the router client and log deployment health states before retrying.
Causes & fixes
No model deployments are currently registered or all are marked unhealthy in the LiteLLM router.
Ensure at least one model deployment is running and registered as healthy in the LiteLLM deployment registry or orchestrator.
Network connectivity issues prevent the router from reaching the deployments.
Verify network connectivity between the router and model deployment instances; fix firewall or DNS issues blocking communication.
Misconfiguration in the router's deployment registry or environment variables causing it to look for non-existent deployments.
Check and correct router configuration files or environment variables to point to valid, healthy deployment names.
Model deployment crashed or failed health checks after startup.
Inspect deployment logs, restart failed deployments, and ensure health check endpoints respond correctly.
Code: broken vs fixed
from litellm import LiteLLMClient
client = LiteLLMClient()
messages = [{"role": "user", "content": "Hello"}]
# This line raises RuntimeError: router no healthy deployments available
response = client.chat.completions.create(model="litellm-router", messages=messages)
print(response)import os
from litellm import LiteLLMClient
# Set environment variable to point to healthy deployment
os.environ["LITELLM_DEPLOYMENT"] = "healthy-deployment"
client = LiteLLMClient()
messages = [{"role": "user", "content": "Hello"}]
# Fixed: ensure deployment is healthy and configured
response = client.chat.completions.create(model="litellm-router", messages=messages)
print(response)Workaround
Catch the RuntimeError exception, then implement a retry with exponential backoff while alerting the ops team to restore healthy deployments.
Prevention
Implement robust deployment health monitoring and automatic failover to ensure the router always has healthy deployments available.