How to beginner · 3 min read

How to get JSON output from LLM

Quick answer

Use prompt engineering to instruct the LLM to respond in JSON format and parse the response string as JSON in Python. With the openai SDK, send a prompt requesting JSON output, then convert the response.choices[0].message.content to a Python dictionary using json.loads().

PREREQUISITES

Python 3.8+
OpenAI API key (free tier works)
pip install openai>=1.0

Setup

Install the official openai Python SDK and set your API key as an environment variable.

Install SDK: pip install openai
Set environment variable in your shell: export OPENAI_API_KEY='your_api_key_here'

bash

pip install openai

Step by step

Send a prompt explicitly requesting JSON output and parse the response string into a Python dictionary.

python

import os
import json
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

prompt = (
    "Generate a JSON object with keys 'name', 'age', and 'city' describing a person."
    " Respond ONLY with valid JSON."
)

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": prompt}]
)

json_text = response.choices[0].message.content

try:
    data = json.loads(json_text)
    print("Parsed JSON output:", data)
except json.JSONDecodeError:
    print("Failed to parse JSON. Raw output:", json_text)

output

Parsed JSON output: {'name': 'John Doe', 'age': 30, 'city': 'New York'}

Common variations

You can use other models like claude-3-5-haiku-20241022 with the Anthropic SDK or handle streaming responses. For Anthropic, use the system= parameter and parse message.content[0].text. Async calls require async client methods.

python

import os
import json
import anthropic

client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

system_prompt = "You are a helpful assistant that outputs only JSON."
user_prompt = "Generate a JSON object with keys 'name', 'age', and 'city'."

message = client.messages.create(
    model="claude-3-5-haiku-20241022",
    max_tokens=500,
    system=system_prompt,
    messages=[{"role": "user", "content": user_prompt}]
)

json_text = message.content[0].text

try:
    data = json.loads(json_text)
    print("Parsed JSON output:", data)
except json.JSONDecodeError:
    print("Failed to parse JSON. Raw output:", json_text)

output

Parsed JSON output: {'name': 'Jane Smith', 'age': 28, 'city': 'San Francisco'}

Troubleshooting

If JSON parsing fails, verify the prompt explicitly requests JSON only.
Use strict prompt instructions like "Respond ONLY with valid JSON."
Check for trailing text or explanations in the output and adjust prompt or parse accordingly.
Use try-except blocks to handle parsing errors gracefully.

✅

Key Takeaways

Always instruct the LLM clearly to output only valid JSON to simplify parsing.
Use json.loads() in Python to convert the LLM's JSON string response into a dictionary.
Handle JSON parsing errors with try-except to avoid runtime crashes.
Anthropic and OpenAI SDKs have different parameter conventions but both support JSON output workflows.
Prompt engineering is critical to ensure the LLM outputs well-formed JSON.

Verified 2026-04 · gpt-4o-mini, claude-3-5-haiku-20241022

Verify ↗