RuntimeError
torch.cuda.runtime.RuntimeError
Stack trace
Traceback (most recent call last):
File "app.py", line 42, in <module>
output = model(input_tensor.cuda()) # triggers CUDA runtime error
File "/usr/local/lib/python3.9/site-packages/torch/cuda/runtime.py", line 123, in _check_driver
raise RuntimeError("CUDA driver version is insufficient for CUDA runtime version")
RuntimeError: CUDA driver version is insufficient for CUDA runtime version Why it happens
RunPod instances may have pre-installed CUDA drivers that are older or incompatible with the CUDA runtime version bundled with your PyTorch or AI framework. This mismatch causes the CUDA runtime to fail initializing GPU resources.
Detection
Check the CUDA driver and runtime versions at startup by running 'nvidia-smi' and 'torch.version.cuda' to detect mismatches before GPU code executes.
Causes & fixes
RunPod instance has an outdated NVIDIA GPU driver incompatible with the CUDA runtime version used by PyTorch.
Update the NVIDIA GPU driver on the RunPod instance to a version compatible with your CUDA runtime, typically by running 'sudo apt-get update && sudo apt-get install --only-upgrade nvidia-driver-XXX'.
Your Python environment uses a PyTorch version built against a newer CUDA runtime than the installed driver supports.
Install a PyTorch version compatible with the installed CUDA driver version, e.g., use 'pip install torch==<version>+cuXXX' matching your driver.
Multiple CUDA versions installed on the RunPod instance causing environment conflicts.
Clean up conflicting CUDA installations and ensure environment variables like CUDA_HOME and LD_LIBRARY_PATH point to the correct CUDA version.
Code: broken vs fixed
import torch
model = torch.nn.Linear(10, 5).cuda()
input_tensor = torch.randn(1, 10)
output = model(input_tensor.cuda()) # RuntimeError: CUDA driver version mismatch here
print(output)import os
import torch
# Ensure environment variables and driver are compatible
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
model = torch.nn.Linear(10, 5).cuda()
input_tensor = torch.randn(1, 10).cuda()
output = model(input_tensor) # Fixed: driver and runtime versions matched
print(output)Workaround
If you cannot update the driver immediately, switch to a CPU-only PyTorch build by setting 'device = torch.device("cpu")' and running your model on CPU until the driver is updated.
Prevention
Use RunPod instance images with pre-validated CUDA driver and runtime compatibility or containerize your environment with matching CUDA versions to avoid driver/runtime mismatches.