Exception
runpod.worker.exceptions.UnhandledException
Stack trace
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/runpod/worker/worker.py", line 123, in run
result = self.handle_task(task)
File "/usr/local/lib/python3.9/site-packages/runpod/worker/worker.py", line 98, in handle_task
output = self.task_handler(task)
File "/app/worker.py", line 45, in task_handler
raise ValueError("Invalid input data")
ValueError: Invalid input data
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/runpod/worker/worker.py", line 130, in run
self.report_error(e)
File "/usr/local/lib/python3.9/site-packages/runpod/worker/worker.py", line 75, in report_error
raise UnhandledException("Worker failed with unhandled error") from e
runpod.worker.exceptions.UnhandledException: Worker failed with unhandled error Why it happens
This error occurs when the RunPod worker code raises an exception that is not caught or handled within the task processing logic. The worker framework then wraps this as an unhandled exception and crashes the worker process. Common causes include invalid input data, missing environment variables, or bugs in the user-defined task handler.
Detection
Monitor worker logs for tracebacks and unhandled exceptions. Implement try/except blocks around task processing to catch errors and log raw inputs for diagnosis before the worker crashes.
Causes & fixes
User task handler raises an exception due to invalid or unexpected input data
Add input validation and exception handling inside the task handler function to catch and handle invalid inputs gracefully.
Missing or incorrect environment variables required by the worker code
Ensure all required environment variables are set correctly before starting the RunPod worker.
Unhandled exceptions in asynchronous or background threads within the worker
Wrap asynchronous code with try/except blocks and propagate errors back to the main worker thread for proper handling.
Code: broken vs fixed
def task_handler(task):
# This line raises an unhandled exception
result = 1 / 0 # ZeroDivisionError
return {'result': result}
# RunPod worker setup
import runpod
runpod.worker.start(task_handler=task_handler) # This triggers unhandled errorimport os
import runpod
def task_handler(task):
try:
result = 1 / 0 # This will raise ZeroDivisionError
except ZeroDivisionError as e:
return {'error': 'Division by zero encountered'} # Handle error gracefully
return {'result': result}
# Use environment variable for API key (example)
os.environ['RUNPOD_API_KEY'] = os.getenv('RUNPOD_API_KEY')
runpod.worker.start(task_handler=task_handler) # Fixed: exceptions handledWorkaround
Wrap the entire task handler call in a try/except block that catches all exceptions, logs the error and input data, and returns a safe error response to prevent worker crashes.
Prevention
Implement comprehensive input validation and error handling in all worker task code. Use monitoring and alerting on worker logs to detect and fix exceptions early. Consider adding automated retries with backoff for transient errors.