How to beginner · 3 min read

How to add logging middleware to FastAPI LLM app

Quick answer

Use FastAPI's middleware system to create a custom logging middleware that intercepts requests and responses. Implement a middleware class or function that logs relevant details before passing control to the LLM endpoint handler using FastAPI and OpenAI or other SDKs.

PREREQUISITES

Python 3.8+
FastAPI installed (pip install fastapi)
Uvicorn installed for running the server (pip install uvicorn)
OpenAI API key (or other LLM API key)
pip install openai>=1.0

Setup

Install fastapi and uvicorn for the web server, and openai for the LLM client.

Set your OpenAI API key as an environment variable:

export OPENAI_API_KEY="your_api_key_here"

bash

pip install fastapi uvicorn openai>=1.0

Step by step

Create a FastAPI app with a logging middleware that logs incoming requests and outgoing responses. Then add a simple LLM endpoint using the OpenAI SDK.

python

import os
import logging
from fastapi import FastAPI, Request
from fastapi.responses import Response
from openai import OpenAI

# Configure logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

app = FastAPI()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

# Logging middleware
@app.middleware("http")
async def log_requests(request: Request, call_next):
    logger.info(f"Incoming request: {request.method} {request.url}")
    response: Response = await call_next(request)
    logger.info(f"Response status: {response.status_code}")
    return response

# LLM endpoint
@app.post("/chat")
async def chat_endpoint(request: Request):
    data = await request.json()
    messages = data.get("messages", [{"role": "user", "content": "Hello"}])

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=messages
    )
    text = response.choices[0].message.content
    logger.info(f"LLM response: {text}")
    return {"response": text}

# To run: uvicorn filename:app --reload

Common variations

Use async logging libraries like loguru for more advanced logging.
Log request and response bodies carefully to avoid sensitive data leaks.
Use different LLM SDKs like anthropic or gemini by adapting the client initialization and call.
Implement streaming responses by integrating FastAPI's StreamingResponse with the LLM streaming API.

Troubleshooting

If logs do not appear, ensure the logging level is set to INFO or lower.
If the middleware blocks requests, verify call_next(request) is awaited and returned properly.
For missing API key errors, confirm OPENAI_API_KEY is set in your environment.

✅

Key Takeaways

Use FastAPI's @app.middleware("http") decorator to add logging middleware.
Log request method, URL, and response status for effective tracing.
Always await and return call_next(request) in middleware to avoid blocking.
Use environment variables for API keys to keep credentials secure.
Adapt logging detail level to avoid exposing sensitive information.

Verified 2026-04 · gpt-4o

Verify ↗