ModelValidationError
mlops.pipeline.validation.ModelValidationError
Stack trace
mlops.pipeline.validation.ModelValidationError: Model validation test failed: accuracy below threshold or schema mismatch
File "/app/mlops/pipeline/validation.py", line 45, in validate_model
raise ModelValidationError("Model validation test failed: accuracy below threshold or schema mismatch")
File "/app/mlops/pipeline/ci_cd.py", line 78, in run_validation
validate_model(model)
File "/app/mlops/pipeline/ci_cd.py", line 102, in main
run_validation(loaded_model) # <-- triggers error Why it happens
The CI/CD model validation test fails when the model's performance metrics (e.g., accuracy) do not meet the predefined thresholds or when the model's output schema does not match the expected format. This can happen due to changes in training data, code regressions, or incompatible model versions.
Detection
Monitor CI/CD pipeline logs for ModelValidationError exceptions and set alerts on validation test failures to catch issues before deployment.
Causes & fixes
Model accuracy or performance metrics fall below the required threshold during validation.
Improve model training or tune hyperparameters to meet or exceed the validation accuracy threshold defined in the CI/CD pipeline.
Model output schema or data format does not match the expected schema defined in validation tests.
Update the model output to conform exactly to the expected schema or adjust the validation schema to match the model's output format.
CI/CD pipeline uses outdated or incompatible validation scripts that do not align with the current model version.
Synchronize validation scripts with the current model version and ensure compatibility by version controlling validation code alongside the model.
Test dataset used in CI/CD validation is corrupted, incomplete, or not representative of production data.
Verify and update the test dataset to ensure it is clean, complete, and representative of real-world data scenarios.
Code: broken vs fixed
def validate_model(model):
# Validation test that fails if accuracy below 0.9
accuracy = model.evaluate_accuracy()
if accuracy < 0.9:
raise ModelValidationError("Model validation test failed: accuracy below threshold") # triggers error
# In CI/CD pipeline
model = load_model()
validate_model(model) # <-- error hereimport os
class ModelValidationError(Exception):
pass
def validate_model(model):
# Updated validation with environment threshold and schema check
accuracy_threshold = float(os.environ.get('VALIDATION_ACCURACY_THRESHOLD', '0.85')) # lowered threshold
accuracy = model.evaluate_accuracy()
if accuracy < accuracy_threshold:
raise ModelValidationError(f"Model validation test failed: accuracy {accuracy} below threshold {accuracy_threshold}")
if not model.output_schema_matches():
raise ModelValidationError("Model output schema mismatch detected")
# In CI/CD pipeline
model = load_model()
try:
validate_model(model) # fixed validation with flexible threshold and schema check
print("Model validation passed.")
except ModelValidationError as e:
print(f"Validation failed: {e}")
raiseWorkaround
Wrap the validation call in try/except ModelValidationError, log detailed metrics and schema info, then allow manual override or fallback to previous stable model version.
Prevention
Integrate schema validation and performance thresholds as code in the CI/CD pipeline with automated alerts and version control to ensure consistent validation aligned with model updates.