How to debug RAG retrieval failures
config_error Why this happens
Retrieval failures in RAG pipelines often stem from misconfigured vector stores, embedding model mismatches, or incorrect retrieval parameters. For example, if your retriever uses a different embedding model than your index, similarity scores become meaningless, causing empty or irrelevant results. Code that does not check for empty retrieval results or silently swallows exceptions can also mask these issues.
Typical error outputs include empty document lists, low similarity scores, or unexpected exceptions like KeyError or ValueError during retrieval.
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
import os
# Broken example: embedding model mismatch
embeddings = OpenAIEmbeddings(model="text-embedding-ada-002", api_key=os.environ["OPENAI_API_KEY"])
vectorstore = FAISS.load_local("my_faiss_index", embeddings)
query = "Explain RAG retrieval failures"
results = vectorstore.similarity_search(query, k=5)
print(results) # Might return empty or irrelevant results[] # Empty list or irrelevant documents
The fix
Ensure your embedding model used for indexing and querying is the same. Add logging to inspect retrieved documents and similarity scores. Adjust retrieval parameters like k to get enough candidates. Validate that your vector store is properly loaded and indexed.
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
import os
embeddings = OpenAIEmbeddings(model="text-embedding-ada-002", api_key=os.environ["OPENAI_API_KEY"])
vectorstore = FAISS.load_local("my_faiss_index", embeddings)
query = "Explain RAG retrieval failures"
results = vectorstore.similarity_search(query, k=5)
for i, doc in enumerate(results):
print(f"Doc {i+1}:", doc.page_content[:200]) # Log snippet of retrieved docsDoc 1: Retrieval Augmented Generation (RAG) combines retrieval of documents with generation by a language model... Doc 2: Common causes of retrieval failures include embedding mismatches, empty indexes, or incorrect query preprocessing... ...
Preventing it in production
Implement retry logic with exponential backoff for transient API errors. Validate embeddings and index consistency during deployment. Use monitoring to track retrieval success rates and alert on anomalies. Provide fallback responses when retrieval returns empty results to maintain user experience.
Key Takeaways
- Always use the same embedding model for indexing and querying in RAG pipelines.
- Add logging to inspect retrieved documents and similarity scores for debugging.
- Validate vector store loading and indexing before querying to avoid silent failures.