docs: center the core search library

docs/README.md

@@ -1,62 +1,85 @@
 # searchgres documentation
 
-searchgres is a Postgres-native search library for TypeScript. It gives you
-semantic (vector), keyword (BM25), and hybrid retrieval — with composable
-hierarchy, metadata, temporal, and regex filters — over a PostgreSQL database
-you own and run.
-
-New here? Start with **[Get started](getting-started.md)** for a working search
-in a few minutes.
-
-## Learn searchgres
-
-1. **[Get started](getting-started.md)** — from an empty database to your first
-   semantic and hybrid results.
-2. **[Install searchgres](installation.md)** — executables, packages,
-   prerequisites, and PostgreSQL setup.
-3. **[Evaluate with Docker Compose](guides/docker-compose.md)** — run PostgreSQL,
-   Ollama, provisioning, and the API server with one command and no API key.
-
-## Guides
-
-- **[Create and manage indexes](guides/indexes.md)** — choose dimensions and a
-  vector type, create and open an index, and rebuild safely.
-- **[Ingest records](guides/ingest.md)** — write one record or thousands,
-  idempotent upserts, named records, metadata, and temporal values.
-- **[Generate embeddings](guides/embeddings.md)** — how records become
-  semantically searchable, draining on demand, and running a worker.
-- **[Search and filter](guides/search.md)** — semantic, keyword, and hybrid
-  search, composable filters, and pagination.
+searchgres is a Postgres-native search library for TypeScript. It combines BM25,
+vector search, Reciprocal Rank Fusion, and structured filters over a PostgreSQL
+index you own.
+
+Bring a `postgres.js` connection and an AI SDK embedding model. searchgres
+manages the schema, native indexes, query routines, and asynchronous embedding
+workflow; your application retains control of its data model, provider, access
+policy, and retrieval pipeline.
+
+## Start here
+
+1. **[Get started](getting-started.md)** — create an index, ingest records,
+   generate embeddings, and run semantic, keyword, and hybrid searches.
+2. **[How search works](concepts/how-search-works.md)** — understand BM25,
+   vector retrieval, RRF, filters, scores, and candidate windows.
+3. **[Model records](concepts/record-model.md)** — decide how content, trees,
+   metadata, names, and temporal ranges represent your corpus.
+4. **[Architecture and responsibilities](concepts/architecture.md)** — see what
+   searchgres manages and what remains in your application.
+
+Want to evaluate it without writing an application? Use the
+**[Docker Compose stack](guides/docker-compose.md)** to run PostgreSQL, Ollama,
+provisioning, and the optional API server with no provider key.
+
+## Core library guides
+
+- **[Install searchgres](installation.md)** — package, runtime, PostgreSQL,
+  extensions, and privileges.
+- **[Create and manage indexes](guides/indexes.md)** — dimensions, vector type,
+  immutable index shape, multiple indexes, and cutovers.
+- **[Ingest records](guides/ingest.md)** — batches, idempotency, derived records,
+  and indexing existing data sources.
+- **[Generate embeddings](guides/embeddings.md)** — queue lifecycle, on-demand
+  draining, continuous workers, and monitoring.
+- **[Search and filter](guides/search.md)** — retrieval recipes, composable
+  filters, ranking controls, and pagination.
+- **[Build a RAG retriever](guides/rag.md)** — use the library as the retrieval
+  stage in an application-controlled RAG pipeline.
 - **[Manage records and trees](guides/records-and-trees.md)** — read, patch,
   delete, subtree operations, and transactions.
-- **[Configure and run the API server](guides/server.md)** — generate files
-  offline, initialize PostgreSQL, serve, and use strict idempotent provisioning.
-- **[Evaluate with Docker Compose](guides/docker-compose.md)** — start the
-  five-service local demo, use it, restart it, and reset its persistent state.
-- **[Run in production](guides/production.md)** — deployment, pooling, worker
-  operations, monitoring, and reindex cutovers.
-- **[Use the MCP server](mcp/index.md)** — run `searchgres-mcp` over stdio and understand
-  its read, write, projection, and safety boundaries.
+- **[Run in production](guides/production.md)** — pools, workers, observability,
+  access control, backups, and reindexing.
+
+## Evaluation and examples
+
+- **[Choosing searchgres](comparison.md)** — compare it with raw pgvector, vector
+  databases, hosted search, RAG frameworks, and memory systems.
+- **[Runnable examples](../examples/README.md)** — small core-library programs
+  for basic search, RAG, document modeling, temporal search, and workers.
+
+## Optional applications
+
+The core library is the primary product. These applications are built on top of
+it and can be used as reference implementations or as-is:
+
+- **[API server](guides/server.md)** — expose one configured index over HTTP.
+- **[Docker Compose evaluation](guides/docker-compose.md)** — try the server and
+  search engine locally without an API key.
+- **[MCP server](mcp/index.md)** — give MCP-compatible agents read and write
+  tools over the API server.
 
 ## Reference
 
-- **[API reference](reference/api.md)** — every public function, option, and
-  return type.
-- **[Errors and recovery](reference/errors.md)** — the typed error hierarchy and
-  how to handle each case.
-- **[Direct SQL](reference/sql.md)** — optional: call the index's SQL routines
-  without the TypeScript library.
+- **[API reference](reference/api.md)** — public functions, options, and return
+  types.
+- **[Errors and recovery](reference/errors.md)** — typed errors and responses.
+- **[Direct SQL](reference/sql.md)** — call schema-local routines without the
+  TypeScript API.
 
 ## Core ideas
 
-- **You own the database and the connection.** You pass searchgres a
-  [`postgres.js`](https://github.com/porsager/postgres) pool; it never opens or
-  closes connections for you.
-- **An index is a PostgreSQL schema.** You choose its name and track it; there is
-  no hidden registry.
-- **Bring your own embedding model.** searchgres calls any
-  [AI SDK](https://sdk.vercel.ai) embedding model you supply and never touches
-  your provider credentials.
-- **Embedding is asynchronous by default.** A new or changed record is searchable
-  by keyword and filters immediately, and by semantic search once its vector is
-  generated.
+- **You own the database and connection.** searchgres never creates or closes
+  your pool.
+- **An index is a PostgreSQL schema.** It contains ordinary records plus native
+  BM25, HNSW, GiST, and GIN indexes.
+- **Retrieval modes compose with filters.** Search by meaning, exact terms,
+  hierarchy, metadata, represented time, and regex in one query.
+- **Bring your own embedding model.** Any AI SDK embedding model works;
+  searchgres does not handle provider credentials.
+- **Embedding is asynchronous by default.** New records work with BM25 and
+  filters immediately and join semantic results after queue processing.
+- **Application policy stays outside core.** Chunking, derivation, reranking,
+  authentication, and authorization can be composed around the library.

docs/comparison.md

@@ -0,0 +1,130 @@
+# Choosing searchgres
+
+searchgres is a good fit when you want a TypeScript library to provide strong
+hybrid and structured retrieval over PostgreSQL you control. It is not a hosted
+vendor or a complete RAG framework; it is the search engine you compose into
+those systems.
+
+## Compared with raw pgvector
+
+`pgvector` provides vector types, operators, and indexes. It does not define a
+complete retrieval application.
+
+searchgres adds:
+
+- BM25 through `pg_textsearch`;
+- RRF fusion of lexical and semantic rankings;
+- tree, metadata, temporal, and regex filters;
+- an indexed record model and schema-local routines;
+- embedding generation, queueing, retries, and concurrency control;
+- validation, typed errors, and OpenTelemetry instrumentation.
+
+Use raw pgvector when a single vector query and your own schema are all you
+need. Use searchgres when you would otherwise build and maintain the surrounding
+retrieval engine yourself.
+
+## Compared with a vector database
+
+Vector databases are optimized for nearest-neighbor search and can out-scale
+searchgres when the central problem is searching an enormous vector collection.
+But vector scale is only one axis of a search system, and many applications do
+not need billions of vectors. They need more ways to express relevance.
+
+A vector database primarily answers “which embeddings are closest?” searchgres
+makes vectors one part of a broader retrieval model:
+
+- BM25 finds exact terms, identifiers, and phrases that semantic similarity can
+  miss;
+- RRF combines lexical and semantic rankings without mixing incompatible score
+  scales;
+- hierarchy, JSON metadata, represented time, and regex constrain both ranking
+  paths in the same query;
+- filter-only search supports browsing and synchronization without inventing a
+  vector query.
+
+For many application and RAG workloads, that flexibility can matter more to
+search quality than specialized vector scale. The benchmark architecture behind
+searchgres used this combination to produce strong retrieval with a deliberately
+simple data model.
+
+The operational substrate is also a major difference. PostgreSQL is a mature,
+popular relational database with real SQL and a widely understood ecosystem for
+transactions, backups, replication, monitoring, access control, and incident
+response. With searchgres you get:
+
+- one transactional database and backup system;
+- no synchronization between relational and vector stores;
+- direct SQL access to ordinary records and schema-local routines;
+- PostgreSQL's hierarchy, JSON, and temporal types and indexes;
+- deployment, provider, and data ownership.
+
+Choose a specialized vector database when extreme vector scale is the dominant
+requirement. Choose searchgres when you want excellent hybrid and structured
+retrieval, SQL, and familiar PostgreSQL operations—and your workload fits on
+Postgres.
+
+searchgres requires PostgreSQL 18 with `pgvector`, `pg_textsearch`, and `ltree`
+available in `public`.
+
+## Compared with a hosted search service
+
+searchgres is a library, but it can power a hosted or internal search API. The
+included server demonstrates one arrangement, and your application can expose
+another.
+
+Choose a turnkey hosted service when you want a vendor to own all database and
+search operations. Choose searchgres when owning PostgreSQL, model selection,
+and application policy is a benefit rather than a burden.
+
+## Compared with a RAG framework
+
+A RAG framework may orchestrate loaders, chunkers, retrievers, prompts, models,
+and generation chains. searchgres focuses on retrieval.
+
+It does not require a particular:
+
+- chunking strategy,
+- generation model,
+- agent framework,
+- prompt format,
+- fact-extraction pipeline,
+- reranker.
+
+Use it as the retriever inside your framework, or call it directly from a small
+application. The [RAG guide](guides/rag.md) shows the latter.
+
+## Compared with an agent-memory system
+
+Memory products often decide what to remember, extract facts from conversations,
+build profiles, or maintain an application-specific memory lifecycle.
+searchgres does none of that automatically.
+
+It can store conversation turns, facts, summaries, decisions, or any other
+textual record, but your application decides what those records mean. The
+hierarchy, metadata, temporal model, and retrieval modes provide primitives for
+building memory or context systems without limiting the library to that use
+case.
+
+## Compared with a search server
+
+The core runs in the same process as your application and accepts a caller-owned
+`postgres.js` pool. This gives you direct types, transactions, and no network
+boundary.
+
+When processes or languages need remote access, put an API around it. You can
+use the included server/client or build a domain-specific service with enforced
+filters and response shaping.
+
+## Current core boundaries
+
+- PostgreSQL 18 and the three required extensions are mandatory.
+- Data must be represented as records in a searchgres index, although SQL,
+  triggers, CDC, or application jobs can populate it from existing tables.
+- Chunking, fact extraction, summarization, and reranking are application stages,
+  not core v1 features.
+- Authentication and authorization are enforced by the surrounding application
+  or database policy.
+- One index has one immutable vector shape and should use one embedding space.
+
+These boundaries keep the core focused while leaving higher-level workflows
+composable.