TimeoutError
asyncio.exceptions.TimeoutError
Stack trace
Traceback (most recent call last):
File "app/reranker.py", line 45, in rerank_candidates
reranked = await asyncio.wait_for(reranker.rerank(candidates), timeout=5.0) # TimeoutError here
File "/usr/lib/python3.9/asyncio/tasks.py", line 481, in wait_for
raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError Why it happens
Reranking involves calling an LLM or external service to score or reorder candidates. If the response takes longer than the configured timeout, asyncio raises a TimeoutError. This can happen due to network delays, overloaded services, or inefficient reranking logic.
Detection
Monitor reranking call durations and catch asyncio.TimeoutError exceptions to log latency spikes before they cause full request failures.
Causes & fixes
Reranking API call or LLM request exceeds the configured timeout duration
Increase the timeout parameter in asyncio.wait_for or the HTTP client to allow more time for reranking to complete.
Reranking model or service is overloaded or slow due to high traffic or resource constraints
Scale up reranking service resources or implement rate limiting and backoff to reduce load and latency.
Inefficient reranking code or large candidate sets causing slow processing
Optimize reranking logic to reduce complexity or limit the number of candidates passed for reranking.
Network instability or connectivity issues causing delayed responses
Implement retry logic with exponential backoff and monitor network health to mitigate transient delays.
Code: broken vs fixed
import asyncio
async def rerank_candidates(candidates, reranker):
# This line causes TimeoutError if reranking takes longer than 5 seconds
reranked = await asyncio.wait_for(reranker.rerank(candidates), timeout=5.0)
return rerankedimport os
import asyncio
async def rerank_candidates(candidates, reranker):
# Increased timeout from 5 to 15 seconds to avoid TimeoutError
rerank_timeout = float(os.environ.get('RERANK_TIMEOUT', '15.0'))
reranked = await asyncio.wait_for(reranker.rerank(candidates), timeout=rerank_timeout)
return reranked
# Example usage with environment variable set
# os.environ['RERANK_TIMEOUT'] = '15.0'Workaround
Wrap the reranking call in try/except asyncio.TimeoutError, log the timeout event, and fallback to returning the original candidate order or a cached reranking result.
Prevention
Design reranking pipelines with scalable infrastructure, set realistic timeouts based on observed latencies, and implement retries with backoff to handle transient delays gracefully.