How to handle optional fields in extraction
response_model with optional fields defined as Optional in Pydantic models or handle missing keys gracefully in your extraction logic. Always validate the presence of optional fields before accessing them to avoid runtime errors.model_behavior Optional in your Pydantic extraction model and check for their presence before use.Why this happens
When extracting structured data from AI responses, optional fields may be missing if the model does not provide them. For example, a Pydantic BaseModel expecting a field without Optional will raise validation errors if that field is absent. Similarly, accessing missing keys in raw JSON extraction causes KeyError. This happens because the AI output varies and optional fields are not guaranteed.
Example broken code:
from pydantic import BaseModel
class User(BaseModel):
name: str
age: int # Required field
# AI response missing 'age'
response = {'name': 'Alice'}
user = User(**response) # Raises ValidationError because 'age' is missingpydantic.error_wrappers.ValidationError: 1 validation error for User age field required (type=value_error.missing)
The fix
Mark optional fields with Optional and provide default values or None. This allows Pydantic to accept missing fields without error. Alternatively, when parsing raw JSON, use dict.get() to safely access optional keys.
This works because the model or code explicitly allows absence of those fields, preventing validation or runtime errors.
from typing import Optional
from pydantic import BaseModel
import os
from openai import OpenAI
class User(BaseModel):
name: str
age: Optional[int] = None # Optional field
# Example AI extraction call using instructor for structured extraction
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Extract user info: Alice"}],
response_model=User
)
user = response.choices[0].message
print(f"Name: {user.name}, Age: {user.age}") # Age may be None safelyName: Alice, Age: None
Preventing it in production
Implement validation and fallback logic for optional fields. Use Pydantic models with Optional fields and default values. When not using models, always check for key existence with dict.get() or in before access.
Also, log missing optional fields for monitoring and consider prompting the model to provide defaults or explicit nulls. Use retries or clarifications if critical optional data is missing.
Key Takeaways
- Use Pydantic
Optionalfields to handle missing extraction data gracefully. - Always validate or check for optional fields before accessing to avoid runtime errors.
- Log and monitor missing optional fields to improve extraction prompts and data quality.