How to Intermediate · 4 min read

How to do hybrid retrieval in Haystack

Q: How to do hybrid retrieval in Haystack

Use Haystack to combine dense vector retrieval with sparse keyword search by integrating EmbeddingRetriever and BM25Retriever in a Pipeline. This hybrid approach improves search relevance by leveraging both semantic and lexical matching.

Quick answer

Use Haystack to combine dense vector retrieval with sparse keyword search by integrating EmbeddingRetriever and BM25Retriever in a Pipeline. This hybrid approach improves search relevance by leveraging both semantic and lexical matching.

PREREQUISITES

Python 3.8+
pip install haystack-ai openai
OpenAI API key (free tier works)
Basic knowledge of document stores and retrievers

Setup

Install the latest Haystack AI package and set your OpenAI API key as an environment variable to enable embedding generation.

bash

pip install haystack-ai openai

Step by step

This example shows how to create a hybrid retrieval pipeline in Haystack by combining a BM25Retriever for sparse keyword search and an EmbeddingRetriever for dense semantic search. The pipeline merges results from both retrievers to improve accuracy.

python

import os
from haystack import Pipeline
from haystack.document_stores import InMemoryDocumentStore
from haystack.nodes import BM25Retriever, EmbeddingRetriever

# Ensure your OpenAI API key is set in environment
# os.environ["OPENAI_API_KEY"] should be set outside the script

# Initialize an in-memory document store
document_store = InMemoryDocumentStore()

# Sample documents
docs = [
    {"content": "Haystack is an open source NLP framework."},
    {"content": "OpenAI provides powerful language models."},
    {"content": "Hybrid retrieval combines sparse and dense methods."}
]

# Write documents to the store
document_store.write_documents(docs)

# Initialize retrievers
bm25_retriever = BM25Retriever(document_store=document_store)
embedding_retriever = EmbeddingRetriever(
    document_store=document_store,
    embedding_model="sentence-transformers/all-MiniLM-L6-v2",
    model_format="sentence_transformers"
)

# Update embeddings in the document store
document_store.update_embeddings(embedding_retriever)

# Create a pipeline that merges BM25 and embedding retrievers
pipeline = Pipeline()
pipeline.add_node(component=bm25_retriever, name="BM25Retriever", inputs=["Query"])
pipeline.add_node(component=embedding_retriever, name="EmbeddingRetriever", inputs=["Query"])

# Define a custom node to merge results
from haystack.nodes import BaseComponent

class HybridMergeNode(BaseComponent):
    def run(self, bm25_results, embedding_results):
        # Simple merge: combine and deduplicate by document id
        combined = {doc.id: doc for doc in bm25_results}
        for doc in embedding_results:
            combined[doc.id] = doc
        merged_results = list(combined.values())
        return {"documents": merged_results}, "output"

    def run_batch(self, bm25_results, embedding_results):
        return self.run(bm25_results, embedding_results)

merge_node = HybridMergeNode()
pipeline.add_node(component=merge_node, name="HybridMerge", inputs=["BM25Retriever", "EmbeddingRetriever"])

# Query the pipeline
query = "What is hybrid retrieval?"
result = pipeline.run(query=query)

# Print merged document contents
for doc in result["documents"]:
    print(f"- {doc.content}")

output

- Hybrid retrieval combines sparse and dense methods.
- Haystack is an open source NLP framework.
- OpenAI provides powerful language models.

Common variations

Use other document stores like FAISSDocumentStore for scalable vector search.
Replace sentence-transformers/all-MiniLM-L6-v2 with OpenAI embeddings by using OpenAIEmbeddingRetriever.
Adjust merging logic to rank or weight results from each retriever differently.
Implement asynchronous retrieval for improved performance in large-scale applications.

Troubleshooting

If embeddings are not updating, ensure your embedding model is compatible with the document store and call update_embeddings() after writing documents.
If results seem irrelevant, tune the merging strategy or increase retriever top_k parameters.
For large datasets, switch to persistent document stores like FAISSDocumentStore or ElasticsearchDocumentStore.

✅

Key Takeaways

Combine BM25Retriever and EmbeddingRetriever in a Pipeline for hybrid retrieval.
Update document embeddings after adding documents to enable dense retrieval.
Customize merging logic to balance sparse and dense results for best relevance.

Verified 2026-04 · sentence-transformers/all-MiniLM-L6-v2

Verify ↗