How to Intermediate · 3 min read

How to build RAG pipeline with Haystack

Q: How to build RAG pipeline with Haystack

Use haystack to build a RAG pipeline by combining a retriever like FAISS with a generator such as OpenAIGenerator. Index your documents in a DocumentStore, query with the retriever, and generate answers with the generator in a Pipeline.

Quick answer

Use haystack to build a RAG pipeline by combining a retriever like FAISS with a generator such as OpenAIGenerator. Index your documents in a DocumentStore, query with the retriever, and generate answers with the generator in a Pipeline.

PREREQUISITES

Python 3.8+
OpenAI API key (free tier works)
pip install haystack-ai openai faiss-cpu

Setup

Install the required packages and set your OpenAI API key as an environment variable.

bash

pip install haystack-ai openai faiss-cpu

Step by step

This example shows how to create a RAG pipeline with Haystack using InMemoryDocumentStore, FAISS retriever, and OpenAIGenerator. It indexes sample documents, then queries the pipeline to get a generated answer.

python

import os
from haystack import Pipeline
from haystack.document_stores import InMemoryDocumentStore
from haystack.nodes import FAISSRetriever, OpenAIGenerator

# Set your OpenAI API key in environment
# export OPENAI_API_KEY="your_api_key"

# Initialize document store
document_store = InMemoryDocumentStore()

# Sample documents to index
docs = [
    {"content": "Python is a programming language."},
    {"content": "Haystack is an open-source NLP framework."},
    {"content": "RAG stands for Retrieval-Augmented Generation."}
]

# Write documents to the store
document_store.write_documents(docs)

# Initialize retriever with FAISS
retriever = FAISSRetriever(document_store=document_store)

# Initialize generator with OpenAI
generator = OpenAIGenerator(api_key=os.environ["OPENAI_API_KEY"], model="gpt-4o-mini")

# Build pipeline
pipeline = Pipeline()
pipeline.add_node(component=retriever, name="Retriever", inputs=["Query"])
pipeline.add_node(component=generator, name="Generator", inputs=["Retriever"])

# Query the pipeline
query = "What does RAG mean?"
result = pipeline.run(query=query)

print("Generated answer:", result["answers"][0].answer)

output

Generated answer: RAG stands for Retrieval-Augmented Generation, a technique that combines document retrieval with language generation.

Common variations

Use FAISSDocumentStore for persistent FAISS indexes instead of InMemoryDocumentStore.
Switch to other retrievers like DPRRetriever or BM25Retriever based on your use case.
Use different generators such as OpenAIGenerator with other OpenAI models or HuggingFaceGenerator for local models.
Implement async querying by using Haystack's async pipeline methods.

Troubleshooting

If you see ImportError for faiss, ensure faiss-cpu is installed correctly.
If OpenAI API calls fail, verify your OPENAI_API_KEY environment variable is set and valid.
For slow retrieval, consider using a persistent FAISS index or a more efficient retriever.
If no answers are returned, check that documents are indexed properly and retriever is configured correctly.

✅

Key Takeaways

Use Haystack's Pipeline to combine retriever and generator for RAG.
Index documents in a DocumentStore and retrieve with FAISSRetriever.
Generate answers with OpenAIGenerator using your OpenAI API key.
Customize retrievers and generators based on your data and latency needs.
Set environment variables correctly to avoid authentication and import errors.

Verified 2026-04 · gpt-4o-mini

Verify ↗