LiteLLMStreamingProxyError
litellm.errors.LiteLLMStreamingProxyError
Stack trace
litellm.errors.LiteLLMStreamingProxyError: Streaming response incomplete: proxy connection closed unexpectedly
File "/app/main.py", line 42, in generate_stream
for chunk in client.stream_generate(prompt):
File "/usr/local/lib/python3.9/site-packages/litellm/client.py", line 128, in stream_generate
raise LiteLLMStreamingProxyError("Streaming response incomplete: proxy connection closed unexpectedly") Why it happens
LiteLLM streams responses through a proxy that may prematurely close the connection due to network instability, timeout, or proxy misconfiguration. This causes the client to receive an incomplete streaming response, triggering the error.
Detection
Monitor streaming calls for LiteLLMStreamingProxyError exceptions and log partial response data to detect incomplete streams before downstream failures occur.
Causes & fixes
Proxy server closes the streaming connection early due to timeout or resource limits
Increase proxy timeout settings and ensure the proxy supports long-lived streaming connections without interruption.
Network instability causes dropped packets or TCP connection resets during streaming
Implement retry logic with exponential backoff on streaming failures and use a stable network environment.
LiteLLM client does not handle partial or interrupted streams gracefully
Upgrade LiteLLM client to the latest version which includes improved streaming error handling and automatic reconnection.
Code: broken vs fixed
from litellm import LiteLLMClient
client = LiteLLMClient()
prompt = "Tell me a story."
# This line raises LiteLLMStreamingProxyError on incomplete proxy stream
for chunk in client.stream_generate(prompt):
print(chunk)import os
from litellm import LiteLLMClient, LiteLLMStreamingProxyError
os.environ["LITELLM_API_KEY"] = os.environ.get("OPENAI_API_KEY", "your_api_key_here")
client = LiteLLMClient()
prompt = "Tell me a story."
try:
for chunk in client.stream_generate(prompt):
print(chunk)
except LiteLLMStreamingProxyError:
print("Stream interrupted, retrying...")
for chunk in client.stream_generate(prompt): # retry once
print(chunk) # Added retry on streaming proxy errorWorkaround
Catch LiteLLMStreamingProxyError exceptions, extract any partial streamed data received, and retry the streaming request after a short delay to recover from proxy interruptions.
Prevention
Configure proxies to support stable, long-lived streaming connections with appropriate timeouts and resource limits; implement client-side retries and exponential backoff for streaming calls.