LoadBalancingCooldownError
litellm.errors.LoadBalancingCooldownError
Stack trace
litellm.errors.LoadBalancingCooldownError: Request rejected due to load balancing cooldown period. Please retry after cooldown expires.
Why it happens
LiteLLM's internal load balancer enforces a cooldown period between requests to avoid overloading backend resources. When requests come in too quickly, this error is raised to throttle traffic and maintain stability.
Detection
Monitor for LoadBalancingCooldownError exceptions in your request handling code and log the timestamps to detect frequent cooldown triggers before user impact.
Causes & fixes
Sending requests to LiteLLM too rapidly without respecting cooldown intervals
Implement client-side rate limiting or exponential backoff to space out requests and respect the cooldown period.
Multiple concurrent requests exceeding LiteLLM's load balancing capacity
Serialize or queue requests on the client side to reduce concurrency and avoid triggering cooldown.
Using an outdated LiteLLM client version that mishandles cooldown signals
Upgrade to the latest LiteLLM client version which properly handles cooldown errors and retries.
Code: broken vs fixed
from litellm import LiteLLM
client = LiteLLM(api_key='mykey')
response = client.generate('Hello world') # triggers LoadBalancingCooldownErrorimport os
from litellm import LiteLLM, LoadBalancingCooldownError
import time
client = LiteLLM(api_key=os.environ['LITELLM_API_KEY'])
try:
response = client.generate('Hello world')
except LoadBalancingCooldownError:
time.sleep(2) # wait for cooldown
response = client.generate('Hello world')
print(response)Workaround
Catch LoadBalancingCooldownError and parse the error message to extract cooldown duration, then sleep that duration before retrying the request.
Prevention
Implement client-side rate limiting and exponential backoff strategies to space out requests and avoid triggering LiteLLM's load balancing cooldown.