Tracking: move multitenancy from sharded (one process per collection, per-tenant tables and shard keys) to non-sharded (shared structures served by any process)

[edwinyyyu]

Why this issue exists

The issues linked below were filed one at a time, most of them as bugs. Read individually they look like an accumulation of unrelated defects in the storage layer. Read together they are one change of model: the codebase was built for sharded horizontal scaling (each logical collection or partition owned by exactly one process, and separated from other tenants by a physical structure of its own), and the requirement is now non-sharded scaling (any process serves any tenant, and tenants are rows or values inside structures they share). This issue states that change once, records why the previous model was there, classifies each linked issue against it, and tracks what remains.

The model the codebase was built on, and why

Two decisions, both explicit and documented when they were made.

1. Ownership by process sharding. #1201 (merged 2026-04-02) set the VectorStore contract that is still in common/vector_store/vector_store.py today:

A given logical collection identified by a (namespace, name) pair must be managed by at most one process at a time. The consumer is responsible for sharding names across processes.

Its description states the guarantees plainly: "safe concurrent usage on a single process" and "safe concurrent sharded usage (one node per logical collection) on multiple processes". With ownership fixed per process, lifecycle operations (create, open-or-create, delete) could be read-check-write sequences guarded by per-process asyncio locks, and backends without transactions or unique constraints (Qdrant, Milvus) could keep their catalog of logical collections inside the backend itself. The multi-process guarantee was correct under its contract and was never broken: strict sharding always worked. The restriction itself is what became the limitation (#1524, #1525).

2. Per-tenant physical structures. #1205 (merged 2026-05-08) scoped the event memory "by partition key: should allow sharding and horizontal scaling", and on PostgreSQL gave every segment-store partition its own pair of child tables under LIST-partitioned parents, created and dropped by per-tenant DDL. #1292 (merged 2026-05-02) gave every SQLite vector collection its own tables, for a stated reason: "Partition keys are avoided in favor of per-collection tables, since sqlite-vec ANN indexes may not support them." Qdrant's is_distributed flag gives every logical collection its own shard key. The payoff was real: deleting a tenant was one DROP rather than a row-scanning DELETE, one tenant's rows never interleaved with another's in a heap or index, and backend index limitations were sidestepped. Note that #1201 already rejected one native collection per tenant ("most vector databases recommend against one collection per tenant, and empirical tests show that many collections induces significant overhead"); the per-tenant structure was the shard key or child table beneath a shared native collection, not the collection itself.

Both decisions fit the deployments they served: a single server process, embedded SQLite as a first-class backend, an opt-in pre-GA event backend, and tenant counts at which per-tenant DDL was cheap. Nothing below should be read as those decisions having been mistakes at the time.

What changed

Two requirements the sharded model does not meet:

• Multiple server processes must serve the same tenants. Any replica handles any session; routing by session affinity is not a requirement a hosted deployment can carry. This is the horizontal-scalability problem as restated in [Feat]: Cross-process collection registry for vector store metadata #1524 and [Feat]: Redesign QdrantVectorStore collection registry for multi-process collection management #1525: multiple processes working the same collections and partitions, not more processes each owning their own.
• Tenant counts of tens of thousands to millions, with individual tenants ranging from tens of thousands to tens of millions of rows (design/segment_store_shared_tables.md in Overhaul segment store: shared tables with incarnation-scoped tenant keys (fixes #1544, #1546, #1549) #1545).

Why the sharded model fails them

Measured, in the linked issues and in the design doc:

Cost of the model	Evidence
Per-tenant DDL on shared PostgreSQL parents	#1544's second half, and the design doc in #1545: create and delete DDL deadlocked with writers and with each other, on upstream and on every intermediate fix, in two cycle shapes; the class has nowhere to live once there is no lifecycle DDL. (#1544's first half, a CASCADE that removed the shared foreign key, is a plain bug; see the classification below.)
Per-tenant partitions and prepared statements	#1546: after five executions PostgreSQL switches to a generic plan that locks every child partition per read: 9 locks per read with a custom plan, 319 with the generic plan at 316 partitions, HTTP 500 out of shared memory at 128-way concurrency. Each per-tenant table also costs about 5 catalog relations and disk files, which does not survive the required tenant counts.
Per-tenant shard keys on Qdrant	#1564: 450 ms per tenant admission, 45 s for 100 tenants against 0.3 s with payload partitioning, 505 segments against 5, and cluster mode required.
Per-process ownership	#1524, #1525: the moment two processes manage the same name, lock-guarded read-check-write lifecycle has no arbiter. #1566 (folded into #1564): the in-process lock table was a leftover of that serialisation.

The comparison that settled the segment store (raw asyncpg, 40 tenants x 2000 row-pairs, pgvector:pg16, 2 CPUs; full table in the design doc):

layout	ingest	seed read	tenant create	catalog cost
PARTITION OF children	8.7-9.2k pairs/s	0.43-0.46 ms	~12 ms DDL	~5 relations/tenant
shared tables	15k pairs/s	0.21 ms	0.006 ms	none

Per-tenant tables won only small-tenant removal latency (9 ms drop against 58 ms row delete), which does not justify their catalog cost at the required tenant counts. This is the industry's pool model: O(1) logical deletion via a registry plus asynchronous batched reclamation; O(1) physical drops exist only in per-tenant-resource (silo) architectures, which the tenant-count requirement excludes.

The target model

The layer above the stores that ties these together (a tenant record with a step table and a reconciler, UUID-only store keys minted per tenant lifetime, a single tenant handle, and prompt purge as best-effort) is proposed in design/tenant_lifecycle.md under #1579, with the resource-level contracts the stores expose to it. The bullets below are the store-level targets that proposal builds on.

• A tenant is a row, not a relation. One registry row per live tenant, carrying a random incarnation minted at creation; data rows and records are keyed by the incarnation. Creating a tenant is an insert.
• One physical structure per backend, shared by every tenant: shared tables on SQL; payload-partitioned native collections on vector backends, with native collections shared by config as Improve vector store API #1201 already did.
• Lifecycle arbitrated by the database, so any process may perform it: unique constraints decide create races, row locks serialise delete against in-flight writers, FOR UPDATE SKIP LOCKED lets concurrent purgers split a backlog.
• Deletion is O(1) logical (registry delete plus a purge-queue entry in one transaction); physical reclamation is asynchronous, bounded per call, safe to run from any process, and driven by a background purger.
• Stale handles cannot reach a successor. A handle carries the incarnation it was opened on. Where the backend can pin a registry row (the segment store, with FOR SHARE), a stale handle raises; where it cannot (the vector backends), a stale handle's writes land in a dead incarnation that nothing reads and reclamation removes, with lease validation against the registry as the remaining option (Handles held across delete and recreate resurrect records in QdrantVectorStore #1563, Commit Qdrant to payload-partitioned multitenancy: drop per-collection shard keys, retire the in-process lock table, recover fencing and reclamation #1564). Either way a tenant recreated under the same key is unreachable from the old handle, on any process, without coordination.
• The contract says what each store supports. The blanket "at most one process" sentence goes; a store whose metadata moves to a cross-process authority says so, and a deployment can introspect what it may do instead of relying on out-of-band knowledge ([Feat]: Cross-process collection registry for vector store metadata #1524, [Feat]: Redesign QdrantVectorStore collection registry for multi-process collection management #1525).
• Native containers are provisioned, not created on the request path. Creating a logical collection is one registry insert arbitrated by a primary key; the container it lands in already exists, shaped once per deployment (one schema per container, because a shared container cannot carry per-tenant schemas) and ensured by the same provisioning step that owns SQL schema (Collection creation spans the registry and the vector backend, so it cannot be atomic; native containers belong in provisioning, not on the create path #1572, VectorStore.create_collection takes a per-collection config, promising per-tenant schemas that shared-container multitenancy cannot support #1573, Schema provisioning runs from every process's boot path: create_all races on cold boot and never evolves a table, Alembic runs destructive migrations on first use #1570).
• Per-tenant physical isolation stays available as a tier for an oversized tenant (Qdrant's tiered multitenancy, per Commit Qdrant to payload-partitioned multitenancy: drop per-collection shard keys, retire the in-process lock table, recover fencing and reclamation #1564), never as the default.

Status by component

Each row names the issues that carry the work; fixes and their state are tracked on those issues, not here.

Component	Sharded model	Target	Issues
Segment store, PostgreSQL	LIST-partitioned parents, per-tenant child tables and DDL	shared tables keyed by incarnation; registry and purge queue; writers pin the registry row	#1544, #1546, #1549
Segment store, SQLite	shared tables keyed by partition key (never sharded), no fencing	same architecture on every dialect	#1549
Vector store lifecycle, all backends	per-backend ad-hoc catalogs, per-process locks; contract forbids multi-process management	collection metadata arbitrated by a cross-process authority, so any process can create, open and delete; the contract states what each store supports	#1524, #1525
Vector store create path and interface	native container created inside create_collection (a dual write across registry and backend, safe only because names are content-addressed); per-collection VectorStoreCollectionConfig promising per-tenant schemas	containers provisioned before serving; create_collection is a registry insert; collection shape configured per deployment	#1572, #1573
Qdrant	is_distributed shard key per logical collection; native name re-derived from sha256(config) on every open; in-process lock table	payload partitioning; native name pinned at creation; lifecycle arbitrated by the cross-process authority; reclamation of dead partition keys	#1564, #1565, #1563, #1562 (#1566 folded into #1564)
SQLite vector stores	per-collection tables; stale handles reach the successor	incarnation-scoped native resources (per-collection tables stay; the store's stated reason is that sqlite-vec's index may not support a partition key)	#1536
Milvus	per-record partition_key field filtered on queries (already value-based); catalog in a dummy-vector __registry collection	catalog moved to the cross-process authority	#1524, #1535
Session data manager	read-then-write on session rows, safe only with one process per session	database-arbitrated create and update	#1543
Schema provisioning	every process runs create_all at boot; no migration path for a layout change	one provisioning owner; migrations carry layout changes such as the segment store's	#1570
Above the stores	process-local handle caches; no domain-error-to-status mapping	evict and re-resolve on a stale handle; explicit API status	#1571

Found along the way, not sharding-related, but blocking correctness on SQLite: #1568 (per-connection state registered per store on caller-supplied engines) and #1542 (engines built without WAL, busy timeout, or explicit write transactions). Contract issues that matter more once native collections are shared across tenants: #1534 (unbounded property-key cardinality, undeclared-property mandate) and #1535 (indexed_properties_schema is part of collection identity); #1573 narrows that to whether per-collection configuration belongs in the interface at all.

How to read the linked issues

• Plain defects, unrelated to the model: Deleting a segment store partition drops the derivative-link foreign key for all partitions, and can deadlock with a concurrent create #1544's first half (a CASCADE that removes a shared constraint), QdrantVectorStore re-derives the native collection name on every open, so any config-serialization change silently orphans existing data #1562 (config serialisation repoints data), SQLite per-connection state (foreign keys, sqlite-vec extension) is registered per-store on caller-supplied engines #1568 (per-connection SQLite state registered late).
• Handle bugs the sharded model could have fixed locally: Handles held across delete and recreate resurrect records in both SQLite vector stores #1536, Segment store partition handles are not fenced to a partition incarnation: stale handles silently operate on a recreated partition #1549, Handles held across delete and recreate resurrect records in QdrantVectorStore #1563. Each reproduces inside one process: an in-flight request holds a handle while that same process deletes and recreates the collection, and the handle reaches the successor. Real under any model, but not a failure of sharding: the owning process performs the delete and could invalidate every handle it issued at that moment, in memory. The target model removes that option, since the deleter may be another process, so invalidation moves into the data as the incarnation.
• Costs of the sharded model that only surface at scale: Episodic search reads lock every segment-store partition once prepared statements go generic (Postgres lock-table exhaustion, 500s under load) #1546, Commit Qdrant to payload-partitioned multitenancy: drop per-collection shard keys, retire the in-process lock table, recover fencing and reclamation #1564, Deleting a segment store partition drops the derivative-link foreign key for all partitions, and can deadlock with a concurrent create #1544's second half (lifecycle DDL deadlocking against shared parents), QdrantVectorStore accumulates one asyncio.Lock per logical collection for the client's lifetime #1566 (in-process lifecycle serialisation), and the catalog cost per tenant. Not coding errors; the model's own behaviour under tenant counts and process counts it was not designed for.
• Capability gaps the sharded model never had to fill: [Feat]: Cross-process collection registry for vector store metadata #1524, [Feat]: Redesign QdrantVectorStore collection registry for multi-process collection management #1525 (cross-process lifecycle arbitration), Qdrant integration never reclaims native collections or records under dead partition keys #1565 (reclamation, since a value cannot be dropped), Session data manager loses concurrent config updates and leaks IntegrityError on concurrent create #1543 (lost updates once two processes share a session), Schema provisioning runs from every process's boot path: create_all races on cold boot and never evolves a table, Alembic runs destructive migrations on first use #1570 (a schema owner, once tenant metadata is rows and only the fixed boot-time schema is left), Collection creation spans the registry and the vector backend, so it cannot be atomic; native containers belong in provisioning, not on the create path #1572 (a provisioning owner for native containers, once creating a collection is a registry insert).
• Consequences of the target model: Stale segment-store handles need cache eviction and an API status mapping #1571 (fencing makes staleness recognisable, so the layers above must handle it), VectorStore.create_collection takes a per-collection config, promising per-tenant schemas that shared-container multitenancy cannot support #1573 (a shared container carries one schema, so the interface must stop promising one per collection), the asynchronous-erasure semantics (a deleted tenant is unreachable immediately and erased in the background, so synchronous erasure is a deployment's explicit choice).

Done when

• The "at most one process" clause is gone from vector_store.py, replaced by a scope every store declares and honours.
• No backend creates a per-tenant physical structure by default; tenant creation is a row write everywhere, and the per-tenant tier is an explicit opt-in.
• Every lifecycle operation is safe from any process on the same backend: create races resolve by constraint, delete is idempotent and fences in-flight writers, purge is concurrent-safe and bounded.
• A stale handle cannot reach a successor tenant on any backend (raising where the backend can pin a registry row, landing in a dead incarnation where it cannot), and every backend has reclamation for dead tenants.
• Native containers are provisioned before serving and creating a logical collection is a single registry insert; collection shape is configured per deployment, one schema per container.
• Existing deployments have a stated path for each layout change: a migration through Schema provisioning runs from every process's boot path: create_all races on cold boot and never evolves a table, Alembic runs destructive migrations on first use #1570's infrastructure, or an explicit recreate-schema statement of the kind the segment store overhaul makes for the pre-GA event backend.

Out of scope

• The declarative (Neo4j) backend, which has not been audited for multi-process lifecycle.
• Cross-replica coherence of the process-local EpisodicMemoryManager cache beyond handle eviction ([Feat]: Server tech debt resolution wishlist #1297 rates it the main scaling bottleneck).

Investigated and written by Claude (Claude Code), filed from the account of the user who commissioned the investigation.