How to beginner · 3 min read

How to test LLM applications

Q: How to test LLM applications

To test LLM applications, use the OpenAI SDK to send controlled prompts and verify responses programmatically. Implement unit tests by asserting expected outputs or response properties, and use streaming or async calls to test real-time behavior.

Quick answer

To test LLM applications, use the OpenAI SDK to send controlled prompts and verify responses programmatically. Implement unit tests by asserting expected outputs or response properties, and use streaming or async calls to test real-time behavior.

PREREQUISITES

Python 3.8+
OpenAI API key (free tier works)
pip install openai>=1.0

Setup

Install the official openai Python SDK and set your API key as an environment variable for secure authentication.

bash

pip install openai>=1.0

output

Collecting openai
  Downloading openai-1.x.x-py3-none-any.whl
Installing collected packages: openai
Successfully installed openai-1.x.x

Step by step

Use the OpenAI client to send a test prompt and assert the response content. This example demonstrates a simple synchronous test for an LLM completion.

python

import os
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

# Define a test prompt
messages = [{"role": "user", "content": "Say hello in a friendly way."}]

# Call the chat completion endpoint
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages
)

# Extract the generated text
text = response.choices[0].message.content
print("LLM response:", text)

# Simple assertion for testing
assert "hello" in text.lower(), "Response does not contain 'hello'"

output

LLM response: Hello! How can I assist you today?

Common variations

Test LLMs asynchronously or with streaming to handle real-time outputs. You can also test different models by changing the model parameter.

python

import asyncio
import os
from openai import OpenAI

async def async_test():
    client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
    messages = [{"role": "user", "content": "Give me a quick joke."}]

    # Async call
    response = await client.chat.completions.acreate(
        model="gpt-4o-mini",
        messages=messages
    )
    print("Async LLM response:", response.choices[0].message.content)

asyncio.run(async_test())

output

Async LLM response: Why did the scarecrow win an award? Because he was outstanding in his field!

Troubleshooting

If you get authentication errors, verify your OPENAI_API_KEY environment variable is set correctly.
For rate limit errors, implement retries with exponential backoff.
If responses are empty or unexpected, check your prompt formatting and model availability.

✅

Key Takeaways

Use the official OpenAI SDK with environment-based API keys for secure testing.
Write assertions on LLM outputs to automate validation of expected behavior.
Test asynchronously and with streaming to cover real-time application scenarios.

Verified 2026-04 · gpt-4o-mini

Verify ↗