EvaluationConfigError
promptfoo.errors.EvaluationConfigError
Stack trace
promptfoo.errors.EvaluationConfigError: Invalid evaluation config: missing or malformed fields
File "/app/eval_runner.py", line 42, in run_evaluation
evaluation = promptfoo.evaluate(config)
File "/usr/local/lib/python3.10/site-packages/promptfoo/__init__.py", line 120, in evaluate
raise EvaluationConfigError("Invalid evaluation config: missing or malformed fields") Why it happens
Promptfoo requires a well-formed evaluation configuration specifying prompts, metrics, and models. This error occurs when the config file is missing required keys, has syntax errors, or uses unsupported parameter values. The evaluation cannot proceed without a valid config.
Detection
Validate your evaluation config schema before running promptfoo.evaluate(), and log config loading errors to catch issues early.
Causes & fixes
Missing required fields like 'prompts' or 'metrics' in the evaluation config
Add all required fields to the config file according to promptfoo's schema documentation.
Syntax errors or invalid YAML/JSON formatting in the config file
Use a YAML/JSON linter or parser to validate the config file syntax before running evaluations.
Using unsupported or misspelled keys in the config
Check the promptfoo config reference and correct any invalid or misspelled keys.
Config references models or metrics not installed or registered
Ensure all referenced models and metrics are installed and properly registered in your environment.
Code: broken vs fixed
import promptfoo
config = {
# Missing 'prompts' key
"metrics": ["accuracy"]
}
# This line triggers EvaluationConfigError
result = promptfoo.evaluate(config)import os
import promptfoo
config = {
"prompts": ["What is the capital of France?"], # Added required 'prompts' key
"metrics": ["accuracy"]
}
# Now runs without EvaluationConfigError
result = promptfoo.evaluate(config)
print("Evaluation succeeded", result)Workaround
Wrap promptfoo.evaluate() in try/except EvaluationConfigError, log the config contents, and fix errors manually before retrying.
Prevention
Use schema validation tools or promptfoo's config validation utilities to verify your evaluation config before execution to avoid runtime errors.