DeepEvalSchemaError
deepeval.exceptions.DeepEvalSchemaError
Stack trace
deepeval.exceptions.DeepEvalSchemaError: Test case schema validation failed: missing required field 'input'
File "/app/deepeval_runner.py", line 42, in run_test_case
deepeval.validate_test_case(test_case)
File "/usr/local/lib/python3.9/site-packages/deepeval/validation.py", line 88, in validate_test_case
raise DeepEvalSchemaError(f"Test case schema validation failed: {error_msg}") Why it happens
DeepEval expects test cases to strictly follow a predefined JSON schema specifying required fields and types. If the test case JSON is missing required fields, has extra unexpected fields, or has type mismatches, DeepEval raises this schema error to prevent invalid test execution.
Detection
Validate test cases against the DeepEval JSON schema before running tests, or catch DeepEvalSchemaError exceptions and log the invalid test case data for debugging.
Causes & fixes
Test case JSON is missing required fields such as 'input' or 'expected_output'.
Ensure all required fields defined in the DeepEval test case schema are present in your test case JSON before submission.
Field types in the test case do not match the schema, e.g., 'input' is a string but expected to be an object.
Correct the data types in your test case JSON to exactly match the schema definitions, including nested fields.
Test case JSON contains unexpected extra fields not defined in the schema.
Remove any extraneous fields from the test case JSON or update the schema if those fields are intended.
Code: broken vs fixed
import os
from deepeval import validate_test_case
# Missing required 'input' field triggers DeepEvalSchemaError
test_case = {
"expected_output": "42"
}
validate_test_case(test_case) # This line raises DeepEvalSchemaErrorimport os
from deepeval import validate_test_case
# Added required 'input' field to fix schema error
test_case = {
"input": {"question": "What is 6 * 7?"},
"expected_output": "42"
}
validate_test_case(test_case) # Now passes schema validation
print("Test case validated successfully.")Workaround
Catch DeepEvalSchemaError exceptions, extract the raw test case JSON, and manually fix missing or mismatched fields before retrying validation.
Prevention
Integrate schema validation as a pre-commit hook or CI step to ensure all DeepEval test cases conform to the schema before deployment.