How to debug structured output failures
model_behavior Why this happens
Structured output failures happen when the AI model returns text that does not conform to the expected JSON or structured format. This often occurs because the prompt lacks clear instructions or constraints, or the model generates extra commentary or formatting errors. For example, a prompt asking for JSON without specifying strict formatting can lead to invalid JSON output, causing parsing errors in your code.
Typical error output includes JSONDecodeError or unexpected tokens when parsing the response.
from openai import OpenAI
import os
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "user", "content": "Return a JSON object with keys 'name' and 'age'."}
]
)
text = response.choices[0].message.content
import json
# This will fail if the output is not valid JSON
parsed = json.loads(text)
print(parsed)json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The fix
Fix structured output failures by explicitly instructing the model to respond with strict JSON only, without extra text or formatting. Use a system prompt or user prompt that enforces JSON schema compliance. Then validate the output before parsing to handle errors gracefully.
from openai import OpenAI
import os
import json
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
system_prompt = (
"You are a JSON generator. Respond ONLY with a valid JSON object with keys 'name' (string) and 'age' (integer)."
)
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": "Generate the JSON."}
]
)
text = response.choices[0].message.content
try:
parsed = json.loads(text)
print(parsed)
except json.JSONDecodeError as e:
print(f"JSON parsing failed: {e}"){'name': 'Alice', 'age': 30} Preventing it in production
Implement retries with exponential backoff for transient failures and validate AI responses with JSON schema validators before processing. Use fallback logic if parsing fails, such as requesting the model to regenerate output or logging errors for manual review. Always include clear prompt instructions and consider using structured output features if supported by the API.
import time
max_retries = 3
for attempt in range(max_retries):
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": "Generate the JSON."}
]
)
text = response.choices[0].message.content
try:
parsed = json.loads(text)
print(parsed)
break
except json.JSONDecodeError:
if attempt < max_retries - 1:
time.sleep(2 ** attempt) # exponential backoff
continue
else:
print("Failed to parse JSON after retries."){'name': 'Alice', 'age': 30} Key Takeaways
- Always instruct the model explicitly to output strict JSON for structured data.
- Validate and parse AI responses with error handling to catch malformed outputs.
- Use retries with exponential backoff to handle transient generation errors.
- Implement fallback or regeneration logic when structured output parsing fails.