How to debug FastAPI LLM endpoint
code_error Why this happens
Common issues in FastAPI LLM endpoints arise from improper async handling, missing or invalid API keys, incorrect model names, or unhandled exceptions during the API call. For example, forgetting to await the asynchronous call to client.chat.completions.create() can cause the endpoint to hang or return incomplete responses. Errors often manifest as 500 Internal Server Errors or timeouts.
Typical broken code example:
from fastapi import FastAPI
import os
from openai import OpenAI
app = FastAPI()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
@app.get("/chat")
async def chat_endpoint():
# Missing await causes coroutine object returned instead of response
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello"}]
)
return {"response": response.choices[0].message.content}500 Internal Server Error or JSON serialization error due to coroutine object
The fix
Fix the endpoint by properly awaiting asynchronous calls, adding try-except blocks to catch API errors, and logging exceptions for easier debugging. Also, validate environment variables and model names before making requests.
Corrected code example:
from fastapi import FastAPI, HTTPException
import os
import logging
from openai import OpenAI
app = FastAPI()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
@app.get("/chat")
async def chat_endpoint():
try:
response = await client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello"}]
)
return {"response": response.choices[0].message.content}
except Exception as e:
logging.error(f"LLM API call failed: {e}")
raise HTTPException(status_code=500, detail="LLM service error"){"response": "Hello! How can I assist you today?"} Preventing it in production
Implement exponential backoff retry logic around your API calls to handle transient RateLimitError or network issues. Use structured logging to capture request and response details. Validate environment variables and model names at startup. Consider fallback responses or circuit breakers to maintain service availability.
import asyncio
from fastapi import FastAPI, HTTPException
import os
import logging
from openai import OpenAI
app = FastAPI()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
async def call_llm_with_retries(messages, retries=3, delay=1):
for attempt in range(retries):
try:
return await client.chat.completions.create(
model="gpt-4o",
messages=messages
)
except Exception as e:
logging.warning(f"Attempt {attempt+1} failed: {e}")
if attempt == retries - 1:
raise
await asyncio.sleep(delay * 2 ** attempt)
@app.get("/chat")
async def chat_endpoint():
try:
response = await call_llm_with_retries([
{"role": "user", "content": "Hello"}
])
return {"response": response.choices[0].message.content}
except Exception as e:
logging.error(f"LLM API call failed after retries: {e}")
raise HTTPException(status_code=500, detail="LLM service error"){"response": "Hello! How can I assist you today?"} Key Takeaways
- Always await asynchronous LLM API calls in FastAPI endpoints to avoid coroutine errors.
- Use try-except blocks with logging to capture and diagnose API call failures.
- Implement retries with exponential backoff to handle transient rate limits and network issues.