json.JSONDecodeError
json.decoder.JSONDecodeError
Stack trace
Traceback (most recent call last):
File "app.py", line 42, in save_chat_history
json_data = json.dumps(chat_history) # <-- error here
File "/usr/lib/python3.9/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
File "/usr/lib/python3.9/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python3.9/json/encoder.py", line 257, in iterencode
raise JSONDecodeError("Invalid control character at", s, err.value) from None
json.decoder.JSONDecodeError: Invalid control character at: line 1 column 15 (char 14) Why it happens
Chat history often contains user input or model output with special characters or non-serializable objects. When attempting to serialize or deserialize this data as JSON, invalid control characters or unsupported types cause the JSONDecodeError. This typically happens if the chat history includes raw binary data, unescaped control characters, or circular references.
Detection
Monitor serialization calls with try/except blocks catching JSONDecodeError and log the raw chat history content causing failure to identify problematic entries before crashing.
Causes & fixes
Chat history contains unescaped control characters like newlines or tabs in strings
Sanitize or escape control characters in chat messages before serialization using json.dumps with ensure_ascii=True or manual string cleaning
Chat history includes non-serializable Python objects such as datetime, bytes, or custom classes
Convert non-serializable objects to JSON-compatible types (e.g., ISO strings for datetime) before serialization
Corrupted or partial chat history data causing malformed JSON structure
Validate chat history data integrity before serialization and implement fallback recovery or data cleansing routines
Using incompatible JSON libraries or versions that handle encoding differently
Use Python's built-in json module consistently and ensure environment uses supported Python version (3.7+)
Code: broken vs fixed
import json
def save_chat_history(chat_history):
# This line raises JSONDecodeError if chat_history has invalid chars
json_data = json.dumps(chat_history) # <-- error here
with open('history.json', 'w') as f:
f.write(json_data)
chat_history = {'messages': ['Hello\x0cWorld', 'Test']}
save_chat_history(chat_history)import json
import os
def sanitize_for_json(obj):
if isinstance(obj, str):
return obj.encode('unicode_escape').decode('utf-8')
if isinstance(obj, dict):
return {k: sanitize_for_json(v) for k, v in obj.items()}
if isinstance(obj, list):
return [sanitize_for_json(i) for i in obj]
return obj
def save_chat_history(chat_history):
sanitized = sanitize_for_json(chat_history) # sanitize control chars
json_data = json.dumps(sanitized, ensure_ascii=True) # fixed: ensure_ascii escapes control chars
with open('history.json', 'w') as f:
f.write(json_data)
chat_history = {'messages': ['Hello\x0cWorld', 'Test']}
save_chat_history(chat_history) # works without JSONDecodeErrorWorkaround
Catch JSONDecodeError during serialization, then manually clean or remove problematic entries from chat history strings before retrying serialization.
Prevention
Design chat history storage to only include JSON-serializable types and sanitize all user/model text inputs on ingestion to avoid invalid characters or unsupported types.