OutputParserException
langchain_core.exceptions.OutputParserException
Stack trace
langchain_core.exceptions.OutputParserException: Could not parse LLM output: ```json\n{"key": "value"}\n```
at llamaindex.tools.base_tool.BaseTool._parse_response (base_tool.py:123)
at llamaindex.tools.base_tool.BaseTool._call (base_tool.py:98)
at llamaindex.tools.base_tool.BaseTool.__call__ (base_tool.py:75)
at main.py:45 Why it happens
LlamaIndex expects the LLM to return a strictly formatted JSON string without markdown fences or extra text. When the LLM returns JSON wrapped in markdown code blocks or with additional commentary, the JSON parser fails, raising OutputParserException.
Detection
Catch OutputParserException around tool calls and log the raw LLM output to detect unexpected formatting or markdown fences before parsing.
Causes & fixes
LLM response includes markdown fences (e.g., ```json) around the JSON output
Modify the prompt to instruct the model to return raw JSON only, or use a parser that strips markdown fences automatically.
The prompt does not clearly specify the exact JSON schema expected by LlamaIndex
Update the prompt to explicitly define the JSON structure and fields expected, ensuring the LLM outputs matching JSON.
Using a base LLM model that ignores output format instructions and returns free-form text
Switch to an instruction-tuned model like gpt-4o-mini or claude-3-5-haiku-20241022 that reliably follows output format instructions.
Code: broken vs fixed
from llama_index import LLMPredictor, ServiceContext, Tool
llm_predictor = LLMPredictor(model_name="gpt-4o")
service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor)
tool = Tool(service_context=service_context)
response = tool("Get data in JSON format") # Raises OutputParserException due to markdown fences
print(response)import os
from llama_index import LLMPredictor, ServiceContext, Tool
from langchain.output_parsers import JsonOutputParser
llm_predictor = LLMPredictor(model_name="gpt-4o-mini")
service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor)
tool = Tool(service_context=service_context, output_parser=JsonOutputParser()) # Added JsonOutputParser to strip fences
response = tool("Return ONLY raw JSON, no markdown fences") # Prompt updated
print(response)Workaround
Wrap the tool call in try/except OutputParserException, then extract JSON substring manually using regex and parse it with json.loads() as a fallback.
Prevention
Use structured output features like OpenAI's response_format or Anthropic's tool use to enforce schema-valid JSON responses at the API level, avoiding parser errors.