Fix Azure OpenAI 429 rate limit error
Quick answer
A 429 rate limit error from
AzureOpenAI occurs when your request rate exceeds the service quota. Add exponential backoff retry logic around your API calls to automatically handle RateLimitError and avoid immediate failures. ERROR TYPE
api_error ⚡ QUICK FIX
Add exponential backoff retry logic around your API call to handle
RateLimitError automatically.Why this happens
The 429 RateLimitError from AzureOpenAI indicates that your application is sending requests faster than the allowed quota or concurrency limits set by Azure. This can happen if you make many rapid calls without pacing or retrying on failure.
Typical error output looks like:
{"error":{"code":"429","message":"You have exceeded your request rate limit."}}Example of code triggering this error without retries:
from openai import AzureOpenAI
import os
client = AzureOpenAI(
api_key=os.environ["AZURE_OPENAI_API_KEY"],
azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
api_version="2024-02-01"
)
response = client.chat.completions.create(
model=os.environ["AZURE_OPENAI_DEPLOYMENT"],
messages=[{"role": "user", "content": "Hello"}]
)
print(response.choices[0].message.content) output
openai.error.RateLimitError: You have exceeded your request rate limit.
The fix
Wrap your AzureOpenAI API call in a retry loop with exponential backoff to handle RateLimitError. This delays and retries requests, preventing immediate failure and respecting rate limits.
Below is a robust example using time.sleep and catching the RateLimitError exception:
from openai import AzureOpenAI, RateLimitError
import os
import time
client = AzureOpenAI(
api_key=os.environ["AZURE_OPENAI_API_KEY"],
azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
api_version="2024-02-01"
)
def call_azure_openai_with_retries(messages, max_retries=5):
delay = 1 # initial delay in seconds
for attempt in range(max_retries):
try:
response = client.chat.completions.create(
model=os.environ["AZURE_OPENAI_DEPLOYMENT"],
messages=messages
)
return response.choices[0].message.content
except RateLimitError:
if attempt == max_retries - 1:
raise
time.sleep(delay)
delay *= 2 # exponential backoff
# Usage
messages = [{"role": "user", "content": "Hello"}]
result = call_azure_openai_with_retries(messages)
print(result) output
Hello! How can I assist you today?
Preventing it in production
- Implement exponential backoff retries with jitter to avoid synchronized retries.
- Monitor your Azure OpenAI usage quotas and request limits in the Azure portal.
- Use client-side rate limiting to pace requests below your quota.
- Consider fallback strategies or queueing requests during high load.
- Log and alert on repeated
RateLimitErroroccurrences to adjust usage or request quota increases.
Key Takeaways
- Always implement exponential backoff retries to handle Azure OpenAI 429 errors gracefully.
- Monitor and respect your Azure OpenAI service quotas to avoid hitting rate limits.
- Use client-side rate limiting and logging to prevent and diagnose rate limit issues.