Modeling anything beyond one vector per record has always meant bending your data to fit the index. Searchable text got flattened into a single field in a single sparse index, or dropped into metadata where it could be filtered but never ranked. Keyword search, if you needed it, usually meant standing up a second sparse index. Then reconciling two sets of results yourself, by hand.
The new Documents API changes that. It's an index defined by a schema that holds text fields, dense vectors, sparse vectors, and metadata together on the same document, and it's queried through a new set of APIs built around documents rather than vectors. The Documents API is what makes full-text search possible. Both are generally available today.
This post covers what the Documents API is, why we built it, what it means for an existing index, and what a schema-based index looks like in code. The examples below use engineering incident postmortems as a running example: a narrative field, a short summary field, a dense vector for semantic search over the narrative, and metadata like severity and team.
What's new
This ships in Python SDK 10.0.0. Creating an index now takes a schema. A schema declares the fields on an index up front: one or more text fields (each independently configurable for full-text search), a dense vector field, and a sparse vector field. Anything upserted that isn't declared in the schema is stored and automatically indexed as filterable metadata, the same as before.
from pinecone import Pinecone, SchemaBuilder
pc = Pinecone(api_key="YOUR_API_KEY")
schema = (
SchemaBuilder()
.add_string_field("narrative", full_text_search={"language": "en"})
.add_string_field("summary", full_text_search={"language": "en"})
.add_dense_vector_field("embedding", dimension=1536, metric="cosine")
.build()
)
pc.indexes.create(
name="incident-postmortems",
schema=schema,
)Working with a schema-based index uses a new set of methods, scoped to documents instead of vectors:
index = pc.Index("incident-postmortems")
index.documents.upsert(documents=[...])
index.documents.search(score_by=[...])Everything here is written against the Python SDK. The same capabilities are available in the TypeScript SDK.
Why we built it
The Documents API is a different way of modeling data on Pinecone, and the foundation for capabilities beyond what ships today.
Start from what an agent actually needs. An agent trying to complete a task rarely wants "the ten nearest vectors" on their own. It wants specific documents: matched by exact terms, filtered by metadata, ranked by relevance, often all at once, with text carrying as much weight in that mix as vectors do.
One schema, every field type. A schema set at index creation can declare multiple independent text fields alongside a dense vector field, a sparse vector field, and any amount of filterable metadata, all on the same document. narrative and summary can each be their own field, scored or filtered on its own terms, instead of getting flattened into one blob because that's all the index had room for.
Keyword search and vector search run against that same schema, in the same index, so there’s nothing extra to stand up or reconcile. That's one index to provision, monitor, and pay for instead of two, and one call to get a result back instead of a client-side merge step in between.
What stays the same
An existing vectors-API index keeps working. Upgrading the SDK doesn't move it to a new data plane, and its query, upsert, fetch, update, delete, and list methods keep working unchanged against it. Integrated embedding and the records API are unaffected entirely.
Creating a new vectors-API index also works the same way it always has:
pc.create_index(
name="incident-postmortems",
vector_type="dense",
dimension=1536,
metric="cosine",
spec=ServerlessSpec(cloud="aws", region="us-east-1"),
)It's queried and updated with the same query, upsert, fetch, update, delete, and list methods as any other legacy index.
Breaking changes to know about
Dense and sparse fields must be declared explicitly
If you're building a schema with both a dense and a sparse field, declare both explicitly. On the old vectors API, metric="dotproduct" alone implied a sparse-capable index. On a schema-based index, a sparse_vector field has to be added to the schema at creation time, or upserting sparse values later will fail.
Reading index metadata
Index-management calls like describe_index and list_indexes return updated shapes after an SDK upgrade, for indexes of either kind. Fields like .dimension and .metric move under .schema.fields[...]:
# Before
index_info.dimension
# After
index_info.schema.fields["embedding"].dimensiondimension, for example, only appears when an index has a dense vector field.
Refer to the guide for adopting the Documents API for more details.
Before and after
Creating the index
With the vectors API, creating an index meant specifying a vector_type, dimension, metric, and spec. Now, a schema-based index replaces all four with a single schema:
schema = (
SchemaBuilder()
.add_string_field("narrative", full_text_search={"language": "en"})
.add_string_field("summary", full_text_search={"language": "en"}) .add_dense_vector_field("embedding", dimension=1536, metric="cosine")
.build()
)
pc.indexes.create(
name="incident-postmortems",
schema=schema,
deployment={"deployment_type": "managed", "cloud": "aws", "region": "us-east-1"},
)The schema declares narrative and summary as two independent text fields alongside the dense vector field named embedding.
Upserting
With the vectors API, an upsert meant a vector plus a metadata dict, with anything filterable or human-readable (like the narrative text) stuffed into that dict. On a schema-based index, an upsert is a document: the schema fields and any extra fields sit side by side, with no separate metadata parameter to reason about.
index.documents.upsert(
documents=[
{
"_id": "inc-4471",
"narrative": "Checkout latency spiked after a config rollout...",
"summary": "Checkout latency regression",
"embedding": embedding,
"severity": "sev2",
"team": "payments",
}
],
)Fields declared in the schema (narrative, summary, embedding) are validated against it; anything else (severity, team) is stored and indexed for filtering automatically.
Querying
On the old vectors API, narrative could live in metadata, filterable but with no ranking, or get encoded into a sparse vector for lexical scoring, real ranking but no native BM25. On a schema-based index, it's a real text field with BM25 ranking behind it:
index.documents.search(
top_k=10,
score_by=[{"type": "text", "field": "narrative", "query": "checkout latency"}],
filter={"severity": {"$match_phrase":"sev2"}},
include_fields=["narrative", "severity"],
)This shows a single scoring method on one field alongside metadata filtering that is applied before scoring. This is meant to show the shape of the new query, not the full range of what's possible. The full-text search guide covers the rest: scoring by dense or sparse vectors, Lucene query syntax, and the filter operators available on text fields.
Migrating an existing index
Moving an index onto a schema is opt-in and doesn't happen on its own. Nothing about an existing index changes until this is done deliberately:
- Upgrade the python SDK to 10.0.0 or later.
- Create a new index with the schema you want.
- Reindex your data into it.
The guide for adopting the Documents API steps through each step in detail.
FAQ
Do I need to change anything about my existing index?
No. It keeps working as-is.
Does this affect integrated embedding or the records API?
No. Integrated embedding on schema-based (document) indexes is coming.
Can I still create vectors-API indexes going forward?
Yes, just make sure to use the vectors-API query, upsert, fetch, update, delete, and list methods.
Can I convert an existing index to a schema-based one?
No. There's no in-place conversion. A new schema means a new index and a reindex.
More questions, including hybrid search, integrated inference, and pinning API versions, are answered in the guide for adopting the Documents API.
Wrapping up
For a team building keyword and vector search together, the Documents API means no more running a separate sparse index to keep in sync with the dense one, and no more merging two sets of results by hand. It's one schema, one index, and it's what makes full-text search, generally available today, possible in the first place.
For more, see the guide for adopting the Documents API, the full-text search guide, and the data modeling guide.
Was this article helpful?
