How to use Haystack with Kubernetes
Quick answer
Use
Haystack by containerizing your pipeline components and deploying them as pods on Kubernetes. Configure persistent storage and services for document stores and retrievers, then orchestrate your AI pipeline with Kubernetes manifests or Helm charts.PREREQUISITES
Python 3.8+Docker installedkubectl configured with access to a Kubernetes clusterBasic knowledge of Kubernetes manifestspip install haystack-ai>=2.0
Setup
Install haystack-ai Python package and Docker to containerize your Haystack app. Ensure kubectl is configured to access your Kubernetes cluster. Prepare environment variables for API keys if using external LLMs.
pip install haystack-ai
# Verify kubectl access
gcloud container clusters get-credentials my-cluster --zone us-central1-a --project my-project
kubectl get nodes output
NAME STATUS ROLES AGE VERSION k8s-node-1.example.com Ready <none> 10d v1.26.0
Step by step
Build a Docker image for your Haystack pipeline, push it to a container registry, then deploy it on Kubernetes with a Deployment and Service manifest. Use a persistent volume for document storage and configure environment variables for API keys.
### Dockerfile
FROM python:3.10-slim
WORKDIR /app
COPY requirements.txt ./
RUN pip install -r requirements.txt
COPY . ./
CMD ["python", "app.py"]
---
# app.py example
from haystack import Pipeline
from haystack.nodes import OpenAIGenerator, InMemoryRetriever
import os
retriever = InMemoryRetriever()
generator = OpenAIGenerator(api_key=os.environ["OPENAI_API_KEY"], model="gpt-4o-mini")
pipeline = Pipeline()
pipeline.add_node(component=retriever, name="Retriever", inputs=["Query"])
pipeline.add_node(component=generator, name="Generator", inputs=["Retriever"])
if __name__ == "__main__":
query = "What is Kubernetes?"
result = pipeline.run(query=query)
print(result["answers"])
---
# Kubernetes deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: haystack-app
spec:
replicas: 2
selector:
matchLabels:
app: haystack
template:
metadata:
labels:
app: haystack
spec:
containers:
- name: haystack-container
image: gcr.io/my-project/haystack-app:latest
env:
- name: OPENAI_API_KEY
valueFrom:
secretKeyRef:
name: openai-secret
key: api-key
ports:
- containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
name: haystack-service
spec:
selector:
app: haystack
ports:
- protocol: TCP
port: 80
targetPort: 8000
type: LoadBalancer output
deployment.apps/haystack-app created service/haystack-service created # After deployment, access your service via external IP
Common variations
- Use
ElasticsearchDocumentStoreorFAISSDocumentStorewith persistent volumes for scalable document storage. - Deploy Haystack components separately as microservices for retriever, reader, and generator.
- Use Helm charts to manage complex deployments and upgrades.
- Integrate with cloud-managed Kubernetes services like GKE, EKS, or AKS for production readiness.
Troubleshooting
- If pods crash due to missing API keys, verify Kubernetes secrets are correctly mounted and environment variables set.
- For persistent storage issues, ensure your PersistentVolumeClaims are bound and accessible by pods.
- Check pod logs with
kubectl logs <pod-name>for runtime errors. - Network issues: verify Services and Ingress are properly configured for external access.
Key Takeaways
- Containerize Haystack pipelines with Docker for Kubernetes deployment.
- Use Kubernetes Deployments and Services to scale and expose your Haystack app.
- Persist document stores with Kubernetes PersistentVolumes for data durability.
- Separate Haystack components into microservices for flexible scaling.
- Use Kubernetes secrets to securely manage API keys and credentials.