On-call for ML systems

apiVersion: v1
kind: ConfigMap
metadata:
  name: ml-on-call-runbook
  namespace: ml-platform
data:
  runbook.md: |
    # ML Model On-Call Runbook
    
    ## Incident: Prediction Latency > 500ms
    1. Check pod CPU/memory: `kubectl top pod -l app=inference-service`
    2. Check model serving logs: `kubectl logs -l app=inference-service --tail=100`
    3. Verify feature service is healthy: `curl http://feature-service:8000/health`
    4. If feature service down: page feature-platform team
    5. If model OOM: scale replicas or reduce batch size
    6. Rollback to last known good: `kubectl set image deployment/inference-service inference-service=registry.example.com/inference:v2.1.4`
    
    ## Incident: Prediction Drift Detected
    1. Check MLflow: run anomaly detection notebook at https://mlflow.example.com/notebook/drift-check
    2. Inspect raw predictions vs expected distribution from past 7 days
    3. Check data schema: did any upstream data sources change?
    4. If feature values out of range: escalate to data-engineering team
    5. If model predictions unexpected: trigger retraining in staging
    6. Do NOT immediately roll back: model may be correct, data may have legitimately shifted
    
    ## Incident: OOM Killed Pod
    1. Check memory request/limit: `kubectl describe pod <pod-name>`
    2. Check batch size in config: `kubectl get configmap inference-config -o yaml | grep batch_size`
    3. Reduce batch_size by 50%, apply: `kubectl set env deployment/inference-service BATCH_SIZE=8`
    4. Monitor memory for 10 minutes
    5. If stable, update deployment YAML and redeploy
    6. Create ticket to profile model memory usage
    
    ## Escalation
    - Latency + memory pressure: ML Platform On-Call
    - Data quality issues: Data Engineering On-Call
    - Predictions don't match business logic: ML Team Lead
    - Cannot contact anyone after 5 min: @ml-platform-channel in Slack
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: ml-model-alerts
  namespace: ml-platform
spec:
  groups:
  - name: ml-inference
    interval: 30s
    rules:
    - alert: InferencePodCrashLooping
      expr: rate(kube_pod_container_status_restarts_total{pod=~"inference-.*"}[15m]) > 0.1
      for: 5m
      annotations:
        summary: "Inference pod restarting {{ $labels.pod }}"
        runbook_url: "https://wiki.example.com/ml-oncall#pod-crashloop"
    - alert: PredictionLatencyHigh
      expr: histogram_quantile(0.99, rate(inference_request_duration_seconds_bucket[5m])) > 0.5
      for: 10m
      annotations:
        summary: "p99 latency > 500ms for model {{ $labels.model_name }}"
    - alert: FeatureServiceDown
      expr: up{job="feature-service"} == 0
      for: 1m
      annotations:
        summary: "Feature service unreachable"
        runbook_url: "https://wiki.example.com/ml-oncall#feature-service-down"
    - alert: PredictionDriftDetected
      expr: ml_prediction_drift_score{model="production"} > 0.15
      for: 1h
      annotations:
        summary: "Prediction drift > 0.15 for {{ $labels.model }}"
        dashboard: "https://mlflow.example.com/experiments/{{$labels.experiment_id}}"
    - alert: InputDataMissing
      expr: rate(feature_missing_value_total{job="inference-service"}[5m]) > 0.01
      for: 5m
      annotations:
        summary: "{{ $value | humanizePercentage }} of features missing"