How to store embeddings in Pinecone
Quick answer
To store embeddings in
Pinecone, first generate vector embeddings using an embedding model like text-embedding-3-small from OpenAI. Then, initialize a Pinecone index and use the index.upsert() method to insert vectors with unique IDs. This enables fast similarity search and retrieval.PREREQUISITES
Python 3.8+OpenAI API keyPinecone API keypip install openai>=1.0 pinecone-client
Setup
Install the required Python packages and set environment variables for your OPENAI_API_KEY and PINECONE_API_KEY. Initialize the Pinecone client and create or connect to an index.
pip install openai pinecone-client Step by step
This example shows how to generate embeddings with OpenAI, create a Pinecone index, and store embeddings with unique IDs.
import os
from openai import OpenAI
from pinecone import Pinecone
# Initialize OpenAI client
openai_client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
# Initialize Pinecone client
pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])
# Create or connect to an index
index_name = "example-embedding-index"
if index_name not in pc.list_indexes():
pc.create_index(index_name, dimension=1536) # dimension must match embedding size
index = pc.Index(index_name)
# Sample texts to embed
texts = ["Hello world", "Pinecone vector database", "Store embeddings efficiently"]
# Generate embeddings using OpenAI
response = openai_client.embeddings.create(
model="text-embedding-3-small",
input=texts
)
embeddings = [data.embedding for data in response.data]
# Prepare vectors for upsert (id, vector, metadata)
vectors = [(f"vec{i}", emb, {"text": texts[i]}) for i, emb in enumerate(embeddings)]
# Upsert vectors into Pinecone index
index.upsert(vectors=vectors)
# Query example: find top 2 similar vectors to a query
query_response = openai_client.embeddings.create(
model="text-embedding-3-small",
input=["Vector databases"]
)
query_vector = query_response.data[0].embedding
results = index.query(vector=query_vector, top_k=2, include_metadata=True)
print("Query results:")
for match in results.matches:
print(f"ID: {match.id}, Score: {match.score}, Text: {match.metadata['text']}") output
Query results: ID: vec1, Score: 0.92, Text: Pinecone vector database ID: vec2, Score: 0.85, Text: Store embeddings efficiently
Common variations
- Use async Pinecone client for high throughput applications.
- Store additional metadata with vectors for richer filtering.
- Use different embedding models like
text-embedding-3-largeor local sentence-transformers. - Batch upserts for large datasets to optimize performance.
Troubleshooting
- If you get dimension mismatch errors, verify the embedding vector size matches the Pinecone index dimension.
- Ensure your Pinecone API key has permissions and the environment is correctly set.
- Check network connectivity if
pinecone.init()fails. - Use
pinecone.list_indexes()to confirm your index exists before upserting.
Key Takeaways
- Always match embedding vector dimension with Pinecone index dimension to avoid errors.
- Use
index.upsert()to efficiently store embeddings with unique IDs and optional metadata. - Generate embeddings with OpenAI or other models before storing in Pinecone for similarity search.
- Batch operations improve performance when handling large embedding datasets.
- Verify API keys and environment settings to prevent connection issues.