Intermediate

Indexing Vectors

Learn to upsert vectors into Pinecone with metadata, organize data using namespaces, perform batch operations, and manage large-scale vector datasets efficiently.

Upserting Vectors

The upsert operation inserts new vectors or updates existing ones (if the ID already exists). Each vector needs an ID, the embedding values, and optional metadata:

Python 
from pinecone import Pinecone
import os

pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])
index = pc.Index("my-index")

# Upsert a single vector
index.upsert(vectors=[
    {
        "id": "doc-1",
        "values": [0.12, -0.45, 0.78, ...],  # 1536-dim embedding
        "metadata": {
            "title": "Introduction to ML",
            "category": "machine-learning",
            "source": "blog",
            "date": "2026-01-15"
        }
    }
])

# Upsert multiple vectors
index.upsert(vectors=[
    ("doc-2", [0.22, -0.33, ...], {"title": "Deep Learning"}),
    ("doc-3", [0.55, 0.12, ...], {"title": "Neural Networks"}),
])

Generating Embeddings

Before upserting, you need to convert your text into embeddings using a model like OpenAI's text-embedding-3-small:

Python 
from openai import OpenAI

client = OpenAI()

def get_embedding(text, model="text-embedding-3-small"):
    response = client.embeddings.create(
        input=text,
        model=model
    )
    return response.data[0].embedding

# Embed and upsert a document
text = "Machine learning is a subset of artificial intelligence."
embedding = get_embedding(text)

index.upsert(vectors=[{
    "id": "doc-4",
    "values": embedding,
    "metadata": {"text": text, "category": "ai"}
}])

Batch Upserting

For large datasets, upsert in batches to avoid timeouts and optimize throughput:

Python 
import itertools

def chunks(iterable, batch_size=100):
    """Yield successive batches from iterable."""
    it = iter(iterable)
    chunk = list(itertools.islice(it, batch_size))
    while chunk:
        yield chunk
        chunk = list(itertools.islice(it, batch_size))

# Prepare vectors
vectors = [
    {"id": f"doc-{i}", "values": get_embedding(doc), "metadata": {"text": doc}}
    for i, doc in enumerate(documents)
]

# Upsert in batches of 100
for batch in chunks(vectors, batch_size=100):
    index.upsert(vectors=batch)
    print(f"Upserted {len(batch)} vectors")

Namespaces

Namespaces partition your index into isolated groups. Useful for multi-tenant applications or organizing data by category:

Python 
# Upsert into different namespaces
index.upsert(
    vectors=[("doc-1", embedding, {"text": "..."})],
    namespace="user-alice"
)

index.upsert(
    vectors=[("doc-1", embedding, {"text": "..."})],
    namespace="user-bob"
)

# Each namespace is independent
# Same IDs can exist in different namespaces
# Queries only search within the specified namespace

# Check stats per namespace
stats = index.describe_index_stats()
print(stats.namespaces)
# {'user-alice': {'vector_count': 1}, 'user-bob': {'vector_count': 1}}

Metadata Best Practices

✅
Store the source text in metadata: Always store the original text chunk in metadata so you can retrieve it alongside search results. This avoids a second database lookup and simplifies your RAG pipeline.

Metadata supports these value types:

  • Strings: category, title, source URL
  • Numbers: price, rating, timestamp
  • Booleans: is_published, is_verified
  • Lists of strings: tags, categories

Deleting Vectors

Python 
# Delete by ID
index.delete(ids=["doc-1", "doc-2"])

# Delete by metadata filter
index.delete(filter={"category": "outdated"})

# Delete all vectors in a namespace
index.delete(delete_all=True, namespace="old-data")
⚠
Metadata size limit: Each vector's metadata is limited to 40KB. For long documents, store only a text chunk and reference ID in metadata, and keep the full document in a separate database.

What's Next?

In the next lesson, we will learn how to query Pinecone for similarity search, filter results by metadata, and optimize query performance.

← Previous Setup & Configuration

Ready to Go Deeper?

Live instructor-led courses from our partners. Affiliate disclosure.

🧠

AI & ML Courses - 30% Off

Live instructor-led AI, machine learning, data science, and cloud courses for working professionals. Use code Limited30 at checkout.

Edureka
📊

DataCamp - AI & Data Science

Hands-on Python, machine learning, and AI courses with interactive exercises and real projects.

DataCamp
🎓

edX - Top AI Courses

University-level AI courses from MIT, Harvard, Stanford. Earn certificates that employers recognize.

edX