How to intermediate · 4 min read

How to validate LLM output with Pydantic

Q: How to validate LLM output with Pydantic

Use Pydantic models to define the expected schema of your LLM output, then parse and validate the raw text response by converting it to JSON and loading it into the model. This ensures structured, type-safe data from unstructured LLM outputs.

Quick answer

Use Pydantic models to define the expected schema of your LLM output, then parse and validate the raw text response by converting it to JSON and loading it into the model. This ensures structured, type-safe data from unstructured LLM outputs.

PREREQUISITES

Python 3.8+
OpenAI API key (free tier works)
pip install openai>=1.0 pydantic

Setup

Install the required packages and set your OpenAI API key as an environment variable.

Install packages: pip install openai pydantic
Set environment variable: export OPENAI_API_KEY='your_api_key' (Linux/macOS) or setx OPENAI_API_KEY "your_api_key" (Windows)

bash

pip install openai pydantic

Step by step

Define a Pydantic model matching the expected LLM output structure, then call the LLM, parse its JSON response, and validate it with the model.

python

import os
import json
from openai import OpenAI
from pydantic import BaseModel, ValidationError

# Define Pydantic model for expected output
class Person(BaseModel):
    name: str
    age: int
    email: str

# Initialize OpenAI client
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

# Prompt the LLM to output JSON matching the Person model
prompt = (
    "Generate a JSON object with fields: name (string), age (int), email (string)."
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}]
)

raw_text = response.choices[0].message.content
print("Raw LLM output:", raw_text)

try:
    # Parse the raw text as JSON
    data = json.loads(raw_text)
    # Validate and parse with Pydantic
    person = Person(**data)
    print("Validated output:", person)
except json.JSONDecodeError as e:
    print("Failed to parse JSON:", e)
except ValidationError as e:
    print("Validation error:", e)

output

Raw LLM output: {"name": "Alice", "age": 30, "email": "alice@example.com"}
Validated output: name='Alice' age=30 email='alice@example.com'

Common variations

You can adapt this pattern for asynchronous calls, different LLM providers like Anthropic Claude, or streaming outputs. For example, use anthropic.Anthropic client with system= parameter and messages=[{{"role": "user", "content": ...}}]. Also, adjust the Pydantic model for nested or list data.

python

import os
import json
import anthropic
from pydantic import BaseModel

class Person(BaseModel):
    name: str
    age: int
    email: str

client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=500,
    system="You are a helpful assistant.",
    messages=[{"role": "user", "content": "Generate a JSON object with name, age, email."}]
)

raw_text = response.content[0].text
print("Raw Claude output:", raw_text)

try:
    data = json.loads(raw_text)
    person = Person(**data)
    print("Validated output:", person)
except Exception as e:
    print("Error:", e)

output

Raw Claude output: {"name": "Bob", "age": 25, "email": "bob@example.com"}
Validated output: name='Bob' age=25 email='bob@example.com'

Troubleshooting

If JSON parsing fails, ensure the LLM prompt explicitly requests JSON output and consider using json.loads with strict=False or regex extraction.
If validation errors occur, check your Pydantic model matches the expected fields and types exactly.
For partial or malformed outputs, implement fallback parsing or prompt engineering to improve output consistency.

✅

Key Takeaways

Define strict Pydantic models to enforce structured LLM output validation.
Parse LLM text output as JSON before loading into Pydantic models.
Use explicit prompts to guide LLMs to produce valid JSON for easier validation.
Adapt validation for different LLM providers by adjusting client and prompt syntax.
Handle parsing and validation errors gracefully with try-except blocks.

Verified 2026-04 · gpt-4o, claude-3-5-sonnet-20241022

Verify ↗