diff --git a/README.md b/README.md index f263d04..e0ce27d 100644 --- a/README.md +++ b/README.md @@ -4,204 +4,232 @@ [![CI](https://github.com/timescale/searchgres/actions/workflows/ci.yml/badge.svg)](https://github.com/timescale/searchgres/actions/workflows/ci.yml) [![Apache 2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](https://github.com/timescale/searchgres/blob/main/LICENSE) -Postgres-native search for TypeScript. Semantic (vector), keyword (BM25), and -hybrid retrieval — with composable hierarchy, metadata, temporal, and regex -filters — over a PostgreSQL database you own and run. +**Excellent hybrid search in the Postgres you own.** -## Why searchgres - -Most RAG stacks bolt a separate vector database onto the side of the database -that already holds your data, then reconcile two systems forever. searchgres -keeps retrieval where your data lives: real BM25 keyword search, HNSW vector -search, and rank fusion — all executed in Postgres, filtered by the same query. - -- **Semantic, keyword, and hybrid** in one call, with the mode inferred from the - arms you pass. -- **Filters that compose** — scope by tree path, JSONB metadata (containment or - JSONPath), a time range, or a regex, combined with `and`/`or`/`not`. -- **Bring your own embedding model** — any [AI SDK](https://ai-sdk.dev) - provider or a custom implementation. searchgres never touches your credentials. -- **Your database, your connection** — you pass a - [`postgres.js`](https://github.com/porsager/postgres) pool; the library never - opens or closes it. -- **Async embedding built in** — writes are fast; records become semantically - searchable as a queue is drained, by an in-process worker or on demand. -- **Cross-runtime core** — the same published package runs on Node, Bun, and - Deno. - -## Requirements - -- **PostgreSQL 18** with three extensions in the `public` schema: - [`pgvector`](https://github.com/pgvector/pgvector), - [`pg_textsearch`](https://github.com/timescale/pg_textsearch), and `ltree`. -- **Node ≥ 22**, **Bun ≥ 1.4**, or **Deno ≥ 2.0**. - -A Dockerfile that builds PostgreSQL 18 with all three extensions is included for -local development. See -[Install searchgres](https://github.com/timescale/searchgres/blob/main/docs/installation.md). +searchgres is an open-source TypeScript library that combines BM25 keyword +search, vector search, Reciprocal Rank Fusion, and structured filters in +PostgreSQL. Bring a [`postgres.js`](https://github.com/porsager/postgres) +connection and an [AI SDK](https://ai-sdk.dev) embedding model; searchgres +manages the search schema, indexes, and embedding workflow. -## Evaluate locally with no API key +- Find both **exact terms and related meaning**. +- Scope retrieval by **hierarchy, JSON metadata, time, and regex**. +- Keep your search index in **PostgreSQL you control**, not a separate vector + service. +- Use the library directly without adopting a RAG framework, server, or content + model. -The repository includes an evaluation-only stack with PostgreSQL, Ollama, -automatic `nomic-embed-text` download, strict index provisioning, and the API -server: - -```bash -git clone https://github.com/timescale/searchgres.git -cd searchgres -docker compose up --build -``` - -When `server` is healthy, Searchgres is available at -`http://127.0.0.1:3000`. The first run downloads several gigabytes of images and -may take a few minutes. No generated config, provider account, or API key is -needed. See the -[Docker Compose evaluation guide](https://github.com/timescale/searchgres/blob/main/docs/guides/docker-compose.md) -for sample commands, restart/reset behavior, and the evaluation-only security -boundary. - -### Evaluation performance - -This stack optimizes for a free, zero-configuration evaluation, not maximum -embedding throughput. Its Ollama service runs `nomic-embed-text` on CPU in the -container runtime's Linux VM; the stack does not configure GPU access. That is a -property of this evaluation environment, not an inherent Searchgres limit. -Searchgres can use any caller-supplied AI SDK embedding model, including a remote -provider or a GPU-backed local service. - -For orientation, one Apple Silicon arm64 run using a Podman VM with about 3.8 GB -of memory imported 500 short records in 0.10 seconds. The asynchronous worker, -configured in batches of 100, made all 500 semantically searchable in 14.7 -seconds—about 34 embeddings per second. Once embedded, BM25, semantic, and hybrid -queries each completed in roughly 46–60 ms. Records are available to filters and -BM25 immediately while their embeddings are generated in the background. - -These numbers are illustrative rather than a benchmark guarantee: CPU model, -VM resources, record length, cold starts, and host load all matter. Hundreds of -short records should be comfortable for evaluation; use a production embedding -provider or a GPU-backed service when ingestion throughput is important. - -Ollama may log that the requested 8,192-token context exceeds -`n_ctx_train=2048` for this model. Nomic v1.5 has a 2,048-token base context in -its GGUF metadata and extends to 8,192 tokens with RoPE. Ollama's packaged -Modelfile sets `num_ctx 8192` and applies that model-level override, while the -runner warning still reports the base GGUF/training metadata. The warning is -therefore Ollama-specific and does not indicate that Searchgres has a 2,048-token -context limit. - -## Install - -### TypeScript library - -Install the runtime-agnostic core, PostgreSQL driver, and the AI SDK provider of -your choice: +## Install and search ```bash npm install searchgres postgres @ai-sdk/openai ``` -`searchgres` is compiled ESM with type declarations and supports Node, Bun, and -Deno. It includes no native addon, postinstall script, provider credentials, or -Bun-only runtime dependency. - -### Compiled tools - -Install the latest release of all three compiled executables: - -```bash -curl -fsSL https://raw.githubusercontent.com/timescale/searchgres/main/install.sh | sh -``` - -The installer downloads `searchgres`, `searchgres-server`, and -`searchgres-mcp` for the current OS and architecture, then verifies each release -checksum. It installs into `~/.local/bin` when that directory or its parent -exists, otherwise `~/bin`. Set `SEARCHGRES_INSTALL_DIR` to choose another -location. - -## Quick start - ```ts +import { openai } from "@ai-sdk/openai"; import postgres from "postgres"; import { createIndex, openIndex } from "searchgres"; -import { openai } from "@ai-sdk/openai"; const sql = postgres(process.env.DATABASE_URL); -// Create an index (its own Postgres schema). dimensions must match your model. +// Run once. An index is a Postgres schema managed by searchgres. await createIndex(sql, "docs_index", { dimensions: 1536 }); -// Open it, supplying the embedding model. const index = await openIndex(sql, "docs_index", { embedding: openai.embedding("text-embedding-3-small"), }); -// Write records. await index.upsertMany([ - { content: "Auth tokens rotate every 24 hours.", tree: "docs.auth" }, - { content: "Rate limits are 100 req/min per API key.", tree: "docs.api" }, + { + content: "Auth tokens rotate every 24 hours.", + tree: "docs.auth", + meta: { audience: "operators" }, + }, + { + content: "Rate limits are 100 requests per minute for each API key.", + tree: "docs.api", + meta: { audience: "developers" }, + }, ]); -// Generate their embeddings (in production, run a worker instead). +// New records already work with BM25 and filters. Drain before semantic search. await index.processEmbeddings(); -// Search: semantic, keyword, or — passing both arms — hybrid. const hits = await index.search({ semantic: "how are request limits enforced?", fulltext: "rate limit", + filter: { + and: [ + { tree: "docs.api" }, + { meta: { audience: "developers" } }, + ], + }, limit: 5, }); for (const hit of hits) console.log(hit.score, hit.tree, hit.content); - -await sql.end(); +await sql.end(); // you own the pool ``` -Full walkthrough: -**[Get started](https://github.com/timescale/searchgres/blob/main/docs/getting-started.md)**. +See **[Get started](https://github.com/timescale/searchgres/blob/main/docs/getting-started.md)** +for the complete walkthrough. + +## Why searchgres + +### Search quality beyond vector similarity + +Pure semantic search struggles with identifiers, exact phrases, dates, and +scope. searchgres gives each query the retrieval strategy it needs: + +- **BM25** for exact and lexical relevance. +- **Vector search** for related meaning when wording differs. +- **RRF hybrid search** to combine both rankings without mixing incompatible raw + score scales. +- **Composable filters** over tree paths, JSONB metadata and JSONPath, temporal + ranges, and regex. +- **Filter-only listing** when ranking is unnecessary. + +### Postgres-native, not another retrieval stack + +A searchgres index is an ordinary PostgreSQL schema containing records, SQL +routines, and native indexes: HNSW through `pgvector`, BM25 through +`pg_textsearch`, GiST for hierarchy and time, and GIN for metadata. + +That means one database to operate, one transaction system, normal backups and +replication, and direct SQL access when you need it. Your search index lives in +PostgreSQL you control rather than in a separate proprietary service. + +### Search mechanics without application policy -## Product surfaces +searchgres owns the mechanics of retrieval, not your application architecture: -Compiled binaries layer remote workflows over the same core: +- You own the connection pool, embedding provider, credentials, and source data. +- You decide how to chunk, summarize, extract, or otherwise derive records. +- You can organize raw and derived records in one index or several indexes. +- You can call the TypeScript API or the schema-local SQL routines. +- The core contains no user, account, or authorization model. -- `searchgres-server` provisions and serves one configured index. -- `searchgres` provides records, trees, import/export, and search over HTTP. -- `searchgres-mcp` exposes twelve MCP tools over stdio. It talks only to - `searchgres-server`, registers all tools by default, and accepts `--read-only` - to omit mutations. +No fact-extraction pipeline, opaque summarization, or RAG framework is imposed. -Generate server files offline, review them, and then initialize PostgreSQL: +## How search works + +`index.search()` infers the retrieval mode from the arguments you pass: + +| Input | Behavior | +| --- | --- | +| `semantic` or a precomputed `vector` | HNSW cosine search | +| `fulltext` | BM25 keyword search | +| `semantic` and `fulltext` | RRF hybrid search | +| filters only | UUIDv7-ordered listing with keyset pagination | +| either ranked arg plus `filter` | Filtered and ranked retrieval | +| both ranked args plus `filter` | Filtered and RRF hybrid search retrieval | + +Every hit is the full record plus its score. Newly written records are available +to BM25 and filters immediately; they join semantic and hybrid results after the +built-in embedding queue is drained. Run a bounded `processEmbeddings()` pass or +start a concurrency-safe background `EmbeddingWorker`. + +Read **[How search works](https://github.com/timescale/searchgres/blob/main/docs/concepts/how-search-works.md)** +or jump to **[Search and filter](https://github.com/timescale/searchgres/blob/main/docs/guides/search.md)**. + +## Compose it into your application + +The core is deliberately a library. You can use it to build: + +- application search or a hosted search API; +- an indexing pipeline fed from existing tables through code, SQL, or triggers; +- RAG retrieval with your own chunking and generation stages; +- raw, summarized, or fact-extracted representations organized by tree; +- application-enforced access scopes by injecting mandatory tree or metadata + filters; +- a post-retrieval reranking stage. + +Authentication and authorization remain an application or database concern. +Filters become an access-control boundary only when callers cannot bypass the +layer that injects them or issue unrestricted database queries. + +See **[Architecture and responsibilities](https://github.com/timescale/searchgres/blob/main/docs/concepts/architecture.md)** +and **[Choosing searchgres](https://github.com/timescale/searchgres/blob/main/docs/comparison.md)**. + +## Evidence behind the design + +The architecture behind searchgres was evaluated on conversational-memory and +multi-hop retrieval benchmarks using a simplified, prototype based on the same +core approach: one Postgres record table, BM25, HNSW vectors, RRF, and structured +filters—without knowledge graphs or fact-extraction pipelines. + +- **LoCoMo:** `F1=0.666` after search-tool refinement, compared with `F1=0.493` + for the fixed retrieval baseline in the same experiments. +- **MuSiQue:** `86.5%` retrieval recall on a seeded 100-question sample spanning + two-, three-, and four-hop questions. + +These are architecture experiments, not a current leaderboard claim; answering +models, agent behavior, samples, and metrics also affect end-to-end scores. + +## Requirements + +- **PostgreSQL 18** with these extensions installed in `public`: + [`pgvector`](https://github.com/pgvector/pgvector), + [`pg_textsearch`](https://github.com/timescale/pg_textsearch), and `ltree`. + `pg_textsearch` must be included in `shared_preload_libraries`. +- **Node ≥ 22**, **Bun ≥ 1.4**, or **Deno ≥ 2.0**. + +`createIndex()` installs missing extensions when its database role has the +necessary privileges. The repository includes a PostgreSQL Dockerfile with the +extensions configured. See **[Install searchgres](https://github.com/timescale/searchgres/blob/main/docs/installation.md)**. + +## Want a ready-made application? + +The core library is the primary product. This repository also includes optional +applications built on it: + +- `searchgres-server` exposes one configured index over HTTP. +- `searchgres` is a remote CLI for records, trees, import/export, and search. +- `searchgres-mcp` exposes searchgres tools to MCP-compatible agents. +- The Docker Compose stack runs PostgreSQL, Ollama, provisioning, and the server for a + no-API-key local evaluation. + +Use them as reference implementations, an evaluation environment, or as-is for +remote and agentic search. Start the local evaluation with: ```bash -searchgres-server config -searchgres-server init --config searchgres.yaml -searchgres-server serve --config searchgres.yaml +git clone https://github.com/timescale/searchgres.git +cd searchgres +docker compose up --build ``` -Use `init --if-not-exists` for strict idempotent provisioning: an existing index -is accepted only when it is a valid, shape-compatible Searchgres index. See the -[API server guide](https://github.com/timescale/searchgres/blob/main/docs/guides/server.md). - -The MCP binary requires only `--server ` or `SEARCHGRES_URL`; it does not -read server config, dotenv, database credentials, or local import/export files. -See the -[MCP server guide](https://github.com/timescale/searchgres/blob/main/docs/mcp/index.md). +See the **[Docker Compose evaluation guide](https://github.com/timescale/searchgres/blob/main/docs/guides/docker-compose.md)** +or **[API server guide](https://github.com/timescale/searchgres/blob/main/docs/guides/server.md)**. ## Documentation +### Learn the core library + - [Get started](https://github.com/timescale/searchgres/blob/main/docs/getting-started.md) -- [Install searchgres](https://github.com/timescale/searchgres/blob/main/docs/installation.md) +- [How search works](https://github.com/timescale/searchgres/blob/main/docs/concepts/how-search-works.md) +- [Model records](https://github.com/timescale/searchgres/blob/main/docs/concepts/record-model.md) +- [Architecture and responsibilities](https://github.com/timescale/searchgres/blob/main/docs/concepts/architecture.md) - [Create and manage indexes](https://github.com/timescale/searchgres/blob/main/docs/guides/indexes.md) - [Ingest records](https://github.com/timescale/searchgres/blob/main/docs/guides/ingest.md) - [Generate embeddings](https://github.com/timescale/searchgres/blob/main/docs/guides/embeddings.md) - [Search and filter](https://github.com/timescale/searchgres/blob/main/docs/guides/search.md) +- [Build a RAG retriever](https://github.com/timescale/searchgres/blob/main/docs/guides/rag.md) - [Manage records and trees](https://github.com/timescale/searchgres/blob/main/docs/guides/records-and-trees.md) -- [Configure and run the API server](https://github.com/timescale/searchgres/blob/main/docs/guides/server.md) -- [Evaluate with Docker Compose](https://github.com/timescale/searchgres/blob/main/docs/guides/docker-compose.md) - [Run in production](https://github.com/timescale/searchgres/blob/main/docs/guides/production.md) + +### Evaluate and integrate + +- [Runnable examples](https://github.com/timescale/searchgres/tree/main/examples) +- [Choosing searchgres](https://github.com/timescale/searchgres/blob/main/docs/comparison.md) +- [Configure the API server](https://github.com/timescale/searchgres/blob/main/docs/guides/server.md) +- [Evaluate with Docker Compose](https://github.com/timescale/searchgres/blob/main/docs/guides/docker-compose.md) - [Use the MCP server](https://github.com/timescale/searchgres/blob/main/docs/mcp/index.md) -- [API reference](https://github.com/timescale/searchgres/blob/main/docs/reference/api.md) · - [Errors](https://github.com/timescale/searchgres/blob/main/docs/reference/errors.md) · - [Direct SQL](https://github.com/timescale/searchgres/blob/main/docs/reference/sql.md) + +### Reference + +- [API reference](https://github.com/timescale/searchgres/blob/main/docs/reference/api.md) +- [Errors and recovery](https://github.com/timescale/searchgres/blob/main/docs/reference/errors.md) +- [Direct SQL](https://github.com/timescale/searchgres/blob/main/docs/reference/sql.md) ## License diff --git a/docs/README.md b/docs/README.md index fd58cf0..d4941a6 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,62 +1,85 @@ # searchgres documentation -searchgres is a Postgres-native search library for TypeScript. It gives you -semantic (vector), keyword (BM25), and hybrid retrieval — with composable -hierarchy, metadata, temporal, and regex filters — over a PostgreSQL database -you own and run. - -New here? Start with **[Get started](getting-started.md)** for a working search -in a few minutes. - -## Learn searchgres - -1. **[Get started](getting-started.md)** — from an empty database to your first - semantic and hybrid results. -2. **[Install searchgres](installation.md)** — executables, packages, - prerequisites, and PostgreSQL setup. -3. **[Evaluate with Docker Compose](guides/docker-compose.md)** — run PostgreSQL, - Ollama, provisioning, and the API server with one command and no API key. - -## Guides - -- **[Create and manage indexes](guides/indexes.md)** — choose dimensions and a - vector type, create and open an index, and rebuild safely. -- **[Ingest records](guides/ingest.md)** — write one record or thousands, - idempotent upserts, named records, metadata, and temporal values. -- **[Generate embeddings](guides/embeddings.md)** — how records become - semantically searchable, draining on demand, and running a worker. -- **[Search and filter](guides/search.md)** — semantic, keyword, and hybrid - search, composable filters, and pagination. +searchgres is a Postgres-native search library for TypeScript. It combines BM25, +vector search, Reciprocal Rank Fusion, and structured filters over a PostgreSQL +index you own. + +Bring a `postgres.js` connection and an AI SDK embedding model. searchgres +manages the schema, native indexes, query routines, and asynchronous embedding +workflow; your application retains control of its data model, provider, access +policy, and retrieval pipeline. + +## Start here + +1. **[Get started](getting-started.md)** — create an index, ingest records, + generate embeddings, and run semantic, keyword, and hybrid searches. +2. **[How search works](concepts/how-search-works.md)** — understand BM25, + vector retrieval, RRF, filters, scores, and candidate windows. +3. **[Model records](concepts/record-model.md)** — decide how content, trees, + metadata, names, and temporal ranges represent your corpus. +4. **[Architecture and responsibilities](concepts/architecture.md)** — see what + searchgres manages and what remains in your application. + +Want to evaluate it without writing an application? Use the +**[Docker Compose stack](guides/docker-compose.md)** to run PostgreSQL, Ollama, +provisioning, and the optional API server with no provider key. + +## Core library guides + +- **[Install searchgres](installation.md)** — package, runtime, PostgreSQL, + extensions, and privileges. +- **[Create and manage indexes](guides/indexes.md)** — dimensions, vector type, + immutable index shape, multiple indexes, and cutovers. +- **[Ingest records](guides/ingest.md)** — batches, idempotency, derived records, + and indexing existing data sources. +- **[Generate embeddings](guides/embeddings.md)** — queue lifecycle, on-demand + draining, continuous workers, and monitoring. +- **[Search and filter](guides/search.md)** — retrieval recipes, composable + filters, ranking controls, and pagination. +- **[Build a RAG retriever](guides/rag.md)** — use the library as the retrieval + stage in an application-controlled RAG pipeline. - **[Manage records and trees](guides/records-and-trees.md)** — read, patch, delete, subtree operations, and transactions. -- **[Configure and run the API server](guides/server.md)** — generate files - offline, initialize PostgreSQL, serve, and use strict idempotent provisioning. -- **[Evaluate with Docker Compose](guides/docker-compose.md)** — start the - five-service local demo, use it, restart it, and reset its persistent state. -- **[Run in production](guides/production.md)** — deployment, pooling, worker - operations, monitoring, and reindex cutovers. -- **[Use the MCP server](mcp/index.md)** — run `searchgres-mcp` over stdio and understand - its read, write, projection, and safety boundaries. +- **[Run in production](guides/production.md)** — pools, workers, observability, + access control, backups, and reindexing. + +## Evaluation and examples + +- **[Choosing searchgres](comparison.md)** — compare it with raw pgvector, vector + databases, hosted search, RAG frameworks, and memory systems. +- **[Runnable examples](../examples/README.md)** — small core-library programs + for basic search, RAG, document modeling, temporal search, and workers. + +## Optional applications + +The core library is the primary product. These applications are built on top of +it and can be used as reference implementations or as-is: + +- **[API server](guides/server.md)** — expose one configured index over HTTP. +- **[Docker Compose evaluation](guides/docker-compose.md)** — try the server and + search engine locally without an API key. +- **[MCP server](mcp/index.md)** — give MCP-compatible agents read and write + tools over the API server. ## Reference -- **[API reference](reference/api.md)** — every public function, option, and - return type. -- **[Errors and recovery](reference/errors.md)** — the typed error hierarchy and - how to handle each case. -- **[Direct SQL](reference/sql.md)** — optional: call the index's SQL routines - without the TypeScript library. +- **[API reference](reference/api.md)** — public functions, options, and return + types. +- **[Errors and recovery](reference/errors.md)** — typed errors and responses. +- **[Direct SQL](reference/sql.md)** — call schema-local routines without the + TypeScript API. ## Core ideas -- **You own the database and the connection.** You pass searchgres a - [`postgres.js`](https://github.com/porsager/postgres) pool; it never opens or - closes connections for you. -- **An index is a PostgreSQL schema.** You choose its name and track it; there is - no hidden registry. -- **Bring your own embedding model.** searchgres calls any - [AI SDK](https://sdk.vercel.ai) embedding model you supply and never touches - your provider credentials. -- **Embedding is asynchronous by default.** A new or changed record is searchable - by keyword and filters immediately, and by semantic search once its vector is - generated. +- **You own the database and connection.** searchgres never creates or closes + your pool. +- **An index is a PostgreSQL schema.** It contains ordinary records plus native + BM25, HNSW, GiST, and GIN indexes. +- **Retrieval modes compose with filters.** Search by meaning, exact terms, + hierarchy, metadata, represented time, and regex in one query. +- **Bring your own embedding model.** Any AI SDK embedding model works; + searchgres does not handle provider credentials. +- **Embedding is asynchronous by default.** New records work with BM25 and + filters immediately and join semantic results after queue processing. +- **Application policy stays outside core.** Chunking, derivation, reranking, + authentication, and authorization can be composed around the library. diff --git a/docs/comparison.md b/docs/comparison.md new file mode 100644 index 0000000..0370c2e --- /dev/null +++ b/docs/comparison.md @@ -0,0 +1,130 @@ +# Choosing searchgres + +searchgres is a good fit when you want a TypeScript library to provide strong +hybrid and structured retrieval over PostgreSQL you control. It is not a hosted +vendor or a complete RAG framework; it is the search engine you compose into +those systems. + +## Compared with raw pgvector + +`pgvector` provides vector types, operators, and indexes. It does not define a +complete retrieval application. + +searchgres adds: + +- BM25 through `pg_textsearch`; +- RRF fusion of lexical and semantic rankings; +- tree, metadata, temporal, and regex filters; +- an indexed record model and schema-local routines; +- embedding generation, queueing, retries, and concurrency control; +- validation, typed errors, and OpenTelemetry instrumentation. + +Use raw pgvector when a single vector query and your own schema are all you +need. Use searchgres when you would otherwise build and maintain the surrounding +retrieval engine yourself. + +## Compared with a vector database + +Vector databases are optimized for nearest-neighbor search and can out-scale +searchgres when the central problem is searching an enormous vector collection. +But vector scale is only one axis of a search system, and many applications do +not need billions of vectors. They need more ways to express relevance. + +A vector database primarily answers “which embeddings are closest?” searchgres +makes vectors one part of a broader retrieval model: + +- BM25 finds exact terms, identifiers, and phrases that semantic similarity can + miss; +- RRF combines lexical and semantic rankings without mixing incompatible score + scales; +- hierarchy, JSON metadata, represented time, and regex constrain both ranking + paths in the same query; +- filter-only search supports browsing and synchronization without inventing a + vector query. + +For many application and RAG workloads, that flexibility can matter more to +search quality than specialized vector scale. The benchmark architecture behind +searchgres used this combination to produce strong retrieval with a deliberately +simple data model. + +The operational substrate is also a major difference. PostgreSQL is a mature, +popular relational database with real SQL and a widely understood ecosystem for +transactions, backups, replication, monitoring, access control, and incident +response. With searchgres you get: + +- one transactional database and backup system; +- no synchronization between relational and vector stores; +- direct SQL access to ordinary records and schema-local routines; +- PostgreSQL's hierarchy, JSON, and temporal types and indexes; +- deployment, provider, and data ownership. + +Choose a specialized vector database when extreme vector scale is the dominant +requirement. Choose searchgres when you want excellent hybrid and structured +retrieval, SQL, and familiar PostgreSQL operations—and your workload fits on +Postgres. + +searchgres requires PostgreSQL 18 with `pgvector`, `pg_textsearch`, and `ltree` +available in `public`. + +## Compared with a hosted search service + +searchgres is a library, but it can power a hosted or internal search API. The +included server demonstrates one arrangement, and your application can expose +another. + +Choose a turnkey hosted service when you want a vendor to own all database and +search operations. Choose searchgres when owning PostgreSQL, model selection, +and application policy is a benefit rather than a burden. + +## Compared with a RAG framework + +A RAG framework may orchestrate loaders, chunkers, retrievers, prompts, models, +and generation chains. searchgres focuses on retrieval. + +It does not require a particular: + +- chunking strategy, +- generation model, +- agent framework, +- prompt format, +- fact-extraction pipeline, +- reranker. + +Use it as the retriever inside your framework, or call it directly from a small +application. The [RAG guide](guides/rag.md) shows the latter. + +## Compared with an agent-memory system + +Memory products often decide what to remember, extract facts from conversations, +build profiles, or maintain an application-specific memory lifecycle. +searchgres does none of that automatically. + +It can store conversation turns, facts, summaries, decisions, or any other +textual record, but your application decides what those records mean. The +hierarchy, metadata, temporal model, and retrieval modes provide primitives for +building memory or context systems without limiting the library to that use +case. + +## Compared with a search server + +The core runs in the same process as your application and accepts a caller-owned +`postgres.js` pool. This gives you direct types, transactions, and no network +boundary. + +When processes or languages need remote access, put an API around it. You can +use the included server/client or build a domain-specific service with enforced +filters and response shaping. + +## Current core boundaries + +- PostgreSQL 18 and the three required extensions are mandatory. +- Data must be represented as records in a searchgres index, although SQL, + triggers, CDC, or application jobs can populate it from existing tables. +- Chunking, fact extraction, summarization, and reranking are application stages, + not core v1 features. +- Authentication and authorization are enforced by the surrounding application + or database policy. +- One index has one immutable vector shape and should use one embedding space. + +These boundaries keep the core focused while leaving higher-level workflows +composable. diff --git a/docs/concepts/architecture.md b/docs/concepts/architecture.md new file mode 100644 index 0000000..801672e --- /dev/null +++ b/docs/concepts/architecture.md @@ -0,0 +1,139 @@ +# Architecture and responsibilities + +searchgres is a library and a managed PostgreSQL index format. It deliberately +solves retrieval mechanics while leaving application policy in the application. + +## Responsibility boundary + +| searchgres owns | Your application owns | +| --- | --- | +| Index schema and schema-local SQL routines | Source documents and chunking | +| BM25, HNSW, GiST, and GIN indexes | Embedding model and provider credentials | +| Hybrid RRF and structured filter execution | PostgreSQL connection lifecycle | +| Query embedding orchestration | Authentication, authorization, and tenancy policy | +| Database-backed embedding queue and worker | Retrieval prompts and answer generation | +| Record validation, versions, and typed errors | Optional extraction, summarization, and reranking | + +The boundary is compositional rather than restrictive. A hosted API, source-table +indexer, agent tool, or RAG service can all be built around the same core. The +repository's server, CLI, MCP server, and Compose stack demonstrate some of +those arrangements. + +## One schema per index + +Each searchgres index is a literal caller-named PostgreSQL schema containing: + +- a `record` table; +- an `embedding_queue` table; +- native indexes for vectors, BM25, trees, metadata, and time; +- integrity and queue triggers; +- schema-local CRUD, tree, and search routines; +- an immutable schema-format marker. + +There is no global searchgres catalog. Your application tracks its index names. +Several schemas can share one `postgres.js` pool, and separate pools can point at +different databases. + +Every runtime query is schema-qualified. searchgres never persistently mutates +the pool's `search_path` and never closes a pool it did not create. + +## Why logic lives in PostgreSQL + +Core reads and writes call schema-local, `security invoker` SQL routines. This +provides consistent behavior for the TypeScript API and direct SQL callers while +keeping integrity triggers effective for every writer. + +Database-native primitives do the specialized work: + +- `pgvector` and HNSW for cosine retrieval; +- `pg_textsearch` for BM25; +- `ltree` and GiST for hierarchical scope; +- `tstzrange` and GiST for represented time; +- JSONB and GIN for metadata. + +The TypeScript layer validates public inputs, embeds query text, invokes the +routines, maps errors, and emits OpenTelemetry instrumentation. + +## Embedding is an asynchronous dataflow + +When a writer inserts content without an embedding, a database trigger creates +queue work. A drainer later: + +1. claims current work with `FOR UPDATE SKIP LOCKED`; +2. commits the claim; +3. calls the caller-supplied embedding model outside a transaction; +4. writes the vector only if the record version is still current. + +This design lets source writers operate without AI credentials and allows +several worker processes to drain one index safely. A caller can also provide a +vector directly and skip the queue. + +BM25 and structured filters do not wait for embedding generation. + +## Building a service around the library + +A hosted or internal search API typically owns: + +- the database pool and index handle; +- provider configuration and the embedding worker; +- request authentication; +- mandatory access filters; +- rate limits and network policy; +- response projection and optional reranking. + +The included `searchgres-server` is one implementation, not a requirement. An +application can expose its own REST, GraphQL, RPC, job, or in-process interface. + +## Access control with composable filters + +The core has no identity model. Applications can translate authenticated +identity into an enforced filter: + +```ts +const scope = { tree: `tenants.${tenantLabel}` } as const; + +return index.search({ + semantic: request.query, + fulltext: request.query, + filter: request.filter + ? { and: [scope, request.filter] } + : scope, +}); +``` + +Metadata filters can represent ACL or ownership facets when hierarchy is not the +right model. These are valid authorization mechanisms only when the trusted +application constructs the final query and untrusted callers cannot reach an +unscoped handle or database role. + +For defense in depth, combine application enforcement with separate indexes, +database roles and grants, or database-level policy appropriate to your threat +model. `tree` and `meta` are data dimensions; searchgres does not claim that a +caller-supplied filter is itself authentication. + +## Derived content and pipelines + +searchgres begins at the record boundary. Before that boundary, your pipeline +may: + +- parse and chunk files; +- project rows from existing tables; +- extract facts or entities; +- generate summaries; +- attach hierarchy, metadata, and represented time. + +After retrieval, it may rerank, deduplicate, expand neighboring chunks, format +context, and call a generation model. Because search results contain the full +record, these stages do not require a second fetch. + +## Immutable index shape + +Vector type, dimensions, and index settings become PostgreSQL objects at +creation. A schema-format marker covers persisted behavior as well as storage. +To change an incompatible shape or embedding model, create a new schema, +backfill it, validate it, and switch application traffic. + +This makes database state explicit and avoids hidden migration or configuration +drift. + +Next: [Create and manage indexes](../guides/indexes.md). diff --git a/docs/concepts/how-search-works.md b/docs/concepts/how-search-works.md new file mode 100644 index 0000000..596c255 --- /dev/null +++ b/docs/concepts/how-search-works.md @@ -0,0 +1,168 @@ +# How search works + +searchgres combines multiple retrieval strategies because no single score +captures every kind of relevance. Semantic similarity finds related meaning; +BM25 finds exact language and identifiers; structured filters establish the +scope in which either ranking should operate. + +`index.search()` provides all of these paths. The mode is inferred from the +query arms rather than selected with a separate `mode` field. + +| Input | Retrieval path | +| --- | --- | +| `semantic` | Embed the query, then run HNSW cosine search | +| `vector` | Run HNSW cosine search with a precomputed vector | +| `fulltext` | Run BM25 keyword search | +| semantic/vector plus `fulltext` | Run both arms and fuse them with RRF | +| no ranking arm | List matching records in UUIDv7 order | + +A `filter` can be added to every path. + +## Semantic retrieval + +A semantic query finds records whose embeddings point in a similar direction to +the query embedding. This is useful when wording differs: + +```ts +await index.search({ + semantic: "how often are credentials renewed?", + limit: 10, +}); +``` + +searchgres applies the index handle's truncator, calls its embedding model, +checks the returned dimensions, and searches the HNSW index with cosine +distance. The public score is cosine similarity: higher is better, with a +possible range of `[-1, 1]`. + +Pass `vector` instead of `semantic` when your application already computed the +query embedding. The two fields are mutually exclusive. + +Use `semanticThreshold` to reject low-similarity candidates. It applies before +results are returned and accepts values from `0` through `1`. + +## BM25 keyword retrieval + +BM25 rewards terms that occur in a record but are uncommon in the corpus, while +accounting for term frequency and record length. It is usually better than +vector search for product names, error codes, identifiers, and exact phrases: + +```ts +await index.search({ fulltext: "HTTP 429 rate limit" }); +``` + +The score is a positive, query-dependent BM25 value. Its scale is not comparable +to cosine similarity or to BM25 scores from a different query. searchgres +returns only genuine lexical matches; it does not pad the result set to `limit`. + +## Hybrid retrieval and RRF + +Hybrid search runs a semantic arm and a BM25 arm independently: + +```ts +await index.search({ + semantic: "how are requests throttled?", + fulltext: "HTTP 429 rate limit", + limit: 10, +}); +``` + +Raw BM25 and cosine scores have different meanings and scales, so adding them +would make one scoring system dominate arbitrarily. searchgres instead uses +Reciprocal Rank Fusion (RRF): + +```text +score = fulltextWeight / (k + fulltextRank) + + semanticWeight / (k + semanticRank) +``` + +A record that ranks well in both arms rises above one that ranks well in only +one. A missing arm contributes zero. The defaults are `k=60`, equal weights, and +30 candidates from each arm. + +The hybrid score is meaningful only as an ordering within that result set. It is +not an absolute confidence value and should not be compared across queries. + +## Candidate windows and top-k results + +Each hybrid arm retrieves a candidate window before fusion. `candidateLimit` +controls its size and `limit` controls the final result count. A larger candidate +window can improve recall, but it also increases work and can introduce noisy +matches. Tune it with evaluation data rather than assuming more is always +better. + +Ranked retrieval is a fused top-k operation. It has no cursor because ranks and +RRF scores depend on the complete candidate window. Raise `limit` when you need +more ranked results. + +## Structured filters + +Filters are boolean predicates over the same record being ranked: + +- `tree`, `lquery`, and `ltxtquery` scope hierarchy; +- `meta` uses JSONB containment; +- `metaPredicate` evaluates JSONPath; +- temporal leaves query represented instants and ranges; +- `regexp` provides a precision content filter. + +```ts +await index.search({ + semantic: "credential policy", + fulltext: "token rotation", + filter: { + and: [ + { tree: "docs.security" }, + { meta: { status: "current" } }, + { not: { regexp: "deprecated" } }, + ], + }, +}); +``` + +The filter applies to both hybrid arms. This matters: fusing globally ranked +results and filtering afterward could discard the useful candidates before the +correct scope is considered. + +A regex cannot be the only filter in an unranked search because that would allow +an unbounded scan. Combine it with a ranking arm or an indexable tree, metadata, +or temporal filter. + +## Filter-only listing + +With no semantic/vector or fulltext arm, search becomes an ordered record +listing. Its score is the sentinel `-1` and its UUIDv7 order supports keyset +pagination: + +```ts +const page = await index.search({ + filter: { tree: "docs.api" }, + order: "asc", + after: previousLastId, + limit: 100, +}); +``` + +`order`, `after`, and `before` are valid only on this unranked path. + +## Embedding visibility + +A record is available to BM25 and filters immediately after it is written. It +participates in semantic and hybrid retrieval only after it has an embedding. +Provide one during ingest or process the database-backed embedding queue with +`processEmbeddings()` or `startEmbeddingWorker()`. + +## Choosing a mode + +- Start with **BM25** for exact lookup and highly specific terminology. +- Start with **semantic** when users express the same idea with varied wording. +- Use **hybrid** as a strong general retrieval path when both exact and semantic + signals matter. +- Add **filters** whenever application structure can remove irrelevant regions + of the corpus. +- Use **filter-only** search for browsing, synchronization, and batch workflows. + +The right settings depend on your corpus and task. Evaluate retrieval separately +from answer generation so you can tell whether failures came from finding the +context or reasoning over it. + +Next: [Search and filter](../guides/search.md). diff --git a/docs/concepts/record-model.md b/docs/concepts/record-model.md new file mode 100644 index 0000000..d651058 --- /dev/null +++ b/docs/concepts/record-model.md @@ -0,0 +1,163 @@ +# Model records + +A searchgres index stores **records**. One record is one independently +searchable unit of text plus orthogonal annotations for hierarchy, metadata, and +represented time. + +searchgres does not prescribe what a record means. It can be a document chunk, +a support answer, an event, a source-code explanation, an extracted fact, a +conversation turn, or a generated summary. + +## Record fields + +| Field | Purpose | +| --- | --- | +| `content` | Text searched by BM25 and represented by the embedding | +| `tree` | Dotted hierarchy used for organization and subtree filtering | +| `meta` | JSON object used for facets and JSONPath predicates | +| `temporal` | Optional represented instant or time range | +| `name` | Optional stable name unique within a tree | +| `id` | UUIDv7 identity, generated when omitted | +| `embedding` | Optional caller-supplied vector | + +Search results also include version fields, timestamps, and whether an embedding +is present. + +## One record is one chunk + +The core does not split documents. Choose boundaries that make each returned +record useful on its own: + +- preserve headings or paragraphs rather than cutting solely by character count; +- include enough local context to interpret the passage; +- avoid records so large that every match returns unrelated material; +- retain source identity and position in `meta` or `name`; +- evaluate chunking against the questions your application actually asks. + +A stable record address makes repeat ingestion idempotent: + +```ts +await index.upsertMany( + chunks.map((chunk, position) => ({ + tree: `docs.${document.slug}`, + name: `chunk-${position}`, + content: chunk.text, + meta: { + sourceId: document.id, + heading: chunk.heading, + position, + revision: document.revision, + }, + })), + { onConflict: "replace" }, +); +``` + +## Trees organize retrieval scope + +`tree` is a raw dotted PostgreSQL `ltree` path such as: + +```text +docs.api.authentication +tickets.customer_acme.open +research.models.embeddings +``` + +A `{ tree: "docs.api" }` filter matches that node and its descendants. Use trees +for stable, hierarchical scope: tenant, corpus, product area, document, speaker, +session, or lifecycle state when those concepts naturally form a hierarchy. + +Trees are not automatically permissions. An application can enforce an access +scope by injecting an unavoidable tree filter, but callers with unrestricted +library or database access can bypass it. + +## Metadata provides orthogonal facets + +Use `meta` for dimensions that do not form a single hierarchy: + +```ts +{ + source: "runbook", + language: "en", + audience: ["operators", "support"], + revision: 7, + current: true +} +``` + +JSONB containment handles exact subsets, while JSONPath handles predicates such +as numeric comparisons. Prefer fields with consistent names and value types; +metadata flexibility is most useful when producers share conventions. + +Do not duplicate large source documents in metadata. Keep searchable text in +`content` and source blobs in their source system unless the metadata itself is +needed in results. + +## Temporal means represented time + +`createdAt` and `updatedAt` describe the database record. `temporal` describes +what its content represents. + +- `[time]` stores a point event. +- `[start, end]` stores a half-open period `[start, end)`. + +Examples include an incident window, policy validity, event occurrence, or the +period covered by a report. This distinction lets you ask whether records fall +within, overlap, occur before or after, or contain a query time. + +## Raw and derived representations + +Chunking, summarization, and fact extraction can happen before ingest. Derived +content is still just a record and can live: + +- beside raw records under distinct tree branches; +- in the same document subtree with a `meta.kind` facet; +- or in a separate index when it needs a different model, dimensions, lifecycle, + or retrieval policy. + +For example: + +```text +knowledge.raw.handbook.security +knowledge.summary.handbook.security +knowledge.facts.handbook.security +``` + +Preserving raw records alongside derived ones keeps the original evidence +available while allowing specialized representations. + +## Indexing existing application tables + +searchgres does not search arbitrary table layouts in place; its routines and +indexes operate on the index's `record` table. Existing data can feed that table +through: + +- application code that reads source rows and calls `upsertMany()`; +- scheduled SQL that calls the index's `batch_upsert` routine; +- `AFTER INSERT OR UPDATE` triggers on source tables; +- change-data-capture consumers or job queues. + +The searchgres record trigger populates the embedding queue whenever an inserted +or changed record needs a vector, including records written through direct SQL. +This allows a database-native indexing pipeline without putting provider +credentials in the source writer. + +When using a source-table trigger, keep the trigger small and deterministic. A +common design is to project the source row into the searchgres record and let a +separate worker perform remote embedding calls asynchronously. + +## Multiple indexes + +Use one index when records share an embedding model and benefit from searching +across the same corpus. Use separate indexes when you need: + +- different embedding models or dimensions; +- independent lifecycle or ownership; +- physical isolation; +- different BM25 language configuration; +- no cross-corpus retrieval. + +An index is a caller-named PostgreSQL schema, so several indexes can share a +pool without sharing records. + +Next: [Ingest records](../guides/ingest.md). diff --git a/docs/getting-started.md b/docs/getting-started.md index 7d02c49..5342ce3 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,7 +1,9 @@ # Get started -This guide takes you from an empty PostgreSQL database to working semantic and -hybrid search. It should take a few minutes. +This guide takes you from an empty PostgreSQL database to working semantic, +keyword, and hybrid search with the core TypeScript library. It should take a +few minutes. To evaluate the optional server without writing code instead, use +the [Docker Compose guide](guides/docker-compose.md). You will: @@ -161,10 +163,14 @@ await sql.end(); ## What next? +- [How search works](concepts/how-search-works.md) — BM25, vectors, RRF, + filters, scores, and candidate windows. +- [Model records](concepts/record-model.md) — chunking, hierarchy, metadata, + temporal values, and derived records. - [Search and filter](guides/search.md) — scope searches by tree, metadata, time, and regex. -- [Ingest records](guides/ingest.md) — idempotent upserts, named records, and - batches. +- [Build a RAG retriever](guides/rag.md) — add trusted scope and format records + as model context. - [Generate embeddings](guides/embeddings.md) — run a continuous worker and monitor the queue. - [Run in production](guides/production.md) — deployment and operations. diff --git a/docs/guides/indexes.md b/docs/guides/indexes.md index 0c161ac..be3168b 100644 --- a/docs/guides/indexes.md +++ b/docs/guides/indexes.md @@ -1,8 +1,10 @@ # Create and manage indexes -An index is a single PostgreSQL schema holding your records and their search -indexes. You choose its name and your application tracks it — searchgres keeps no -registry and offers no discovery. +An index is a single PostgreSQL schema holding your records, search indexes, +queue, triggers, and SQL routines. You choose its name and your application +tracks it—searchgres keeps no registry and offers no discovery. See +[Architecture and responsibilities](../concepts/architecture.md) for the full +boundary. ## Choose your index shape diff --git a/docs/guides/ingest.md b/docs/guides/ingest.md index 091104e..d1c27f3 100644 --- a/docs/guides/ingest.md +++ b/docs/guides/ingest.md @@ -1,7 +1,9 @@ # Ingest records -A record is one unit of searchable content — one chunk. searchgres does not -split documents for you; the caller decides how to chunk. +A record is one unit of searchable content—one caller-defined chunk, fact, +summary, event, or other textual unit. searchgres does not split or transform +source material for you; your application decides what representations to +index. See [Model records](../concepts/record-model.md) for design guidance. ## Write one record @@ -119,6 +121,38 @@ Duplicate ids within a batch, duplicate `(tree, name)` keys, or a batch where an id and a name resolve to the same existing record are rejected with [`InvalidConfigError`](../reference/errors.md) before anything is written. +## Index existing application data + +searchgres operates on records in its managed index schema, but those records +can be projections of arbitrary existing tables. Populate them through: + +- application jobs that read source rows and call `upsertMany()`; +- scheduled SQL that calls the schema-local `batch_upsert` routine; +- triggers on source tables; +- a change-data-capture consumer. + +Writes performed through direct SQL still run the index's integrity and queue +triggers. If projected content has no vector, it enters the embedding queue just +like a library write. Keep remote embedding calls out of source-table triggers; +let a separate searchgres worker process the queue asynchronously. + +See [Direct SQL](../reference/sql.md) for routine signatures. + +## Store raw and derived records + +Chunking, fact extraction, and summarization can happen before ingest. Preserve +raw evidence and organize representations explicitly: + +```text +knowledge.raw.handbook.security +knowledge.summary.handbook.security +knowledge.facts.handbook.security +``` + +Use tree or metadata filters to select a representation. A second index is often +better when derived records require a different embedding model, dimensions, or +lifecycle. + ## When is a record searchable? Immediately for keyword and filter queries. For **semantic** search a record diff --git a/docs/guides/production.md b/docs/guides/production.md index 6d086da..97db6f9 100644 --- a/docs/guides/production.md +++ b/docs/guides/production.md @@ -80,14 +80,35 @@ too chatty. Parameter values (including vectors) are never attached to spans. ## Access control -searchgres has no user, account, or authorization model — `tree` is a data -dimension, not a permission boundary. Access control is the responsibility of: +searchgres has no user, account, or authorization model. The surrounding +application authenticates callers and can translate identity into mandatory +`tree` or `meta` filters: -- your **application**, which decides who may call which operations, and -- your **database roles/grants**, which decide what the connecting role can do. +```ts +const tenantScope = { tree: `tenants.${trustedTenantLabel}` } as const; +const filter = requestFilter + ? { and: [tenantScope, requestFilter] as const } + : tenantScope; + +const hits = await index.search({ + semantic: query, + fulltext: query, + filter, +}); +``` + +This is an authorization boundary only when the application constructs the +final filter and the caller cannot access an unscoped index handle or issue +unrestricted SQL. Never trust a caller merely to include its own tenant filter. +Validate or map external tenant identifiers to legal `ltree` labels rather than +interpolating them directly. + +Database roles and grants provide another boundary. The index's SQL routines run +as `security invoker`, so they act with the calling role's privileges. Grant each +role only what it needs, and use separate indexes or database-level policy when +your threat model requires stronger physical or database enforcement. -The index's SQL routines run as `security invoker`, so they act with the calling -role's privileges. Grant that role only what it needs. +See [Architecture and responsibilities](../concepts/architecture.md#access-control-with-composable-filters). ## Reindexing and cutover diff --git a/docs/guides/rag.md b/docs/guides/rag.md new file mode 100644 index 0000000..0527ab2 --- /dev/null +++ b/docs/guides/rag.md @@ -0,0 +1,170 @@ +# Build a RAG retriever + +searchgres can provide the retrieval stage of a RAG application without +prescribing chunking, prompting, or generation. This guide builds a small +retriever that combines semantic meaning, exact terms, and application scope. + +## 1. Model and ingest chunks + +Split source documents before the searchgres boundary. Preserve stable source +identity and chunk position so repeated ingestion updates in place: + +```ts +await index.upsertMany( + documents.flatMap((document) => + chunkDocument(document).map((chunk, position) => ({ + tree: `knowledge.${document.collection}.${document.slug}`, + name: `chunk-${position}`, + content: chunk.text, + meta: { + sourceId: document.id, + title: document.title, + position, + visibility: document.visibility, + }, + })), + ), + { onConflict: "replace" }, +); +``` + +`chunkDocument` belongs to your application. It can preserve headings, attach +neighbor information, or use a tokenizer appropriate to the generation model. +searchgres treats each output as one searchable record. + +## 2. Generate embeddings + +Records work with BM25 and filters as soon as they are committed. Drain the +queue before expecting them in semantic or hybrid results: + +```ts +await index.processEmbeddings({ batchSize: 50 }); +``` + +For continuous ingestion, run `startEmbeddingWorker()` in a long-lived process. + +## 3. Write a scoped retriever + +Hybrid retrieval is a strong default for user questions because it preserves +both semantic and exact-term signals. Add mandatory application scope to keep +irrelevant or inaccessible records out of both candidate arms: + +```ts +import type { Filter, Index, SearchResult } from "searchgres"; + +interface RetrievalScope { + tenant: string; + collection?: string; + visibility: "public" | "internal"; +} + +export async function retrieve( + index: Index, + question: string, + scope: RetrievalScope, +): Promise { + const required: Filter[] = [ + { + tree: scope.collection + ? `tenants.${scope.tenant}.${scope.collection}` + : `tenants.${scope.tenant}`, + }, + { meta: { visibility: scope.visibility } }, + ]; + + return index.search({ + semantic: question, + fulltext: question, + filter: { and: required }, + candidateLimit: 50, + limit: 8, + }); +} +``` + +If the scope is an authorization boundary, construct it from trusted identity +rather than accepting it directly from an untrusted request. Do not expose an +unscoped index handle to that caller. + +## 4. Format context + +Every hit contains the full record, so context formatting can happen locally: + +```ts +export function formatContext(hits: readonly SearchResult[]): string { + return hits + .map( + (hit, index) => + `\n` + + `${hit.content}\n` + + ``, + ) + .join("\n\n"); +} +``` + +Keep stable source IDs in the formatted context so the answer can cite evidence +and your application can resolve it later. + +## 5. Generate separately + +Pass `formatContext(hits)` and the question to the model and prompt format your +application chooses. Keeping this stage separate has practical benefits: + +- retrieval can be evaluated without generation variability; +- the same index can serve several models or prompts; +- access filters remain explicit; +- reranking or context budgeting can be inserted between retrieval and + generation. + +## Optional reranking + +Core v1 does not include a reranker. Since search returns full records, rerank a +larger candidate set before formatting: + +```ts +const candidates = await index.search({ + semantic: question, + fulltext: question, + filter: trustedScope, + candidateLimit: 100, + limit: 30, +}); + +const hits = await rerank(question, candidates, { limit: 8 }); +``` + +The first-stage `candidateLimit` controls candidates inside RRF; the final +`limit: 30` controls how many fused records reach your reranker. + +## Derived records + +Summaries and extracted facts can be useful for some corpora, but they need not +replace source evidence. Store them under separate tree branches or in another +index: + +```text +tenants.acme.raw.handbook.security +tenants.acme.summary.handbook.security +tenants.acme.facts.handbook.security +``` + +Use a tree or metadata filter to select one representation or search several. +A separate index is appropriate when derived records use a different embedding +model or retrieval policy. + +## Evaluate retrieval + +Create a set of representative questions with known relevant record IDs. Track +at least: + +- recall at the context limit; +- precision or irrelevant-context rate; +- results by question category; +- latency, including query embedding; +- failures caused by indexing or filters versus failures caused by generation. + +Tune chunking, `candidateLimit`, `semanticThreshold`, and hybrid weights from +those results. More context is not automatically better. + +Next: [Run in production](production.md). diff --git a/docs/guides/search.md b/docs/guides/search.md index 4170e5f..3e11aa1 100644 --- a/docs/guides/search.md +++ b/docs/guides/search.md @@ -1,7 +1,9 @@ # Search and filter `index.search(options)` runs every kind of query. The retrieval mode is inferred -from which arms you supply — there is no `mode` parameter. +from which arms you supply—there is no `mode` parameter. For the reasoning behind +BM25, vector search, RRF, candidate windows, and score semantics, read +[How search works](../concepts/how-search-works.md). | You supply | You get | | --- | --- | @@ -151,7 +153,7 @@ await index.search({ | `{ temporalBefore: t }` | Record's time is strictly before `t`. | | `{ temporalAfter: t }` | Record's time is strictly after `t`. | | `{ temporalContains: t }` | Record's time range contains `t`. | -| `{ regexp: "429|throttl" }` | Case-insensitive POSIX match on `content`. | +| { regexp: "429|throttl" } | Case-insensitive POSIX match on `content`. | Rules: @@ -197,4 +199,5 @@ const next = await index.search({ `order`, `after`, and `before` apply only to filter-only listing. Supplying them with a ranking arm is rejected — ranked results are top-k, not a paginated feed. -Next: [Manage records and trees](records-and-trees.md). +Next: [Build a RAG retriever](rag.md) or +[Manage records and trees](records-and-trees.md). diff --git a/docs/installation.md b/docs/installation.md index 7518478..bc8b804 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,64 +1,32 @@ # Install searchgres -For a no-API-key evaluation without installing a package or executable first, -clone the repository and start the complete local stack: +The primary distribution is the runtime-agnostic TypeScript library. Install it +with `postgres.js` and the AI SDK provider of your choice: ```bash -git clone https://github.com/timescale/searchgres.git -cd searchgres -docker compose up --build +npm install searchgres postgres @ai-sdk/openai ``` -This evaluation-only path includes PostgreSQL, Ollama, model download, -provisioning, and the API server. It is separate from installing Searchgres for -a database and embedding provider you manage. See -[Evaluate with Docker Compose](guides/docker-compose.md). +`searchgres` is compiled ESM with type declarations. It has no native addon, +postinstall script, provider credentials, or Bun-only runtime dependency. -## Compiled executables +The [`postgres`](https://github.com/porsager/postgres) package is the database +driver. searchgres uses the provider-agnostic [`ai`](https://ai-sdk.dev) package +but no provider package; replace `@ai-sdk/openai` with Mistral, Google, another +AI SDK provider, or your own compatible embedding model. -Install the latest release of `searchgres`, `searchgres-server`, and -`searchgres-mcp`: - -```bash -curl -fsSL https://raw.githubusercontent.com/timescale/searchgres/main/install.sh | sh -``` - -The installer detects Linux or macOS on amd64/arm64 (and Windows under a POSIX -shell), downloads all three matching GitHub release assets, and verifies their -individual SHA-256 files before installing anything. The default destination is -`~/.local/bin` when `~/.local` exists, otherwise `~/bin`. - -Override the destination or install a specific release tag with environment -variables on the receiving shell: - -```bash -curl -fsSL https://raw.githubusercontent.com/timescale/searchgres/main/install.sh | \ - SEARCHGRES_INSTALL_DIR="$HOME/.local/bin" SEARCHGRES_VERSION=v0.1.0 sh -``` - -## Packages - -```bash -npm install searchgres postgres -``` - -`searchgres` uses the [`postgres.js`](https://github.com/porsager/postgres) -driver (a peer you install alongside it) and the provider-agnostic -[`ai`](https://sdk.vercel.ai) package. It does **not** depend on any specific -provider package — you choose and install one: - -```bash -npm install @ai-sdk/openai # or @ai-sdk/mistral, @ai-sdk/google, ... -``` +Next, ensure your PostgreSQL environment meets the requirements below, then +follow [Get started](getting-started.md). ## Runtime support -searchgres is runtime-agnostic and ships as ESM with type declarations: - - Node 22 or newer - Bun 1.4 or newer - Deno 2 or newer +Published consumers use compiled JavaScript. Running the repository's `.ts` +examples directly with Node requires Node 22.18 or newer. + ## PostgreSQL searchgres targets **PostgreSQL 18** and requires three extensions: @@ -67,52 +35,76 @@ searchgres targets **PostgreSQL 18** and requires three extensions: | --- | --- | --- | | [`pgvector`](https://github.com/pgvector/pgvector) | `vector` / `halfvec` columns and HNSW cosine indexes | 0.8.0 | | [`pg_textsearch`](https://github.com/timescale/pg_textsearch) | BM25 index and ranking | 1.4.0 | -| `ltree` | hierarchical tree paths (ships with PostgreSQL) | 1.3.0 | +| `ltree` | Hierarchical tree paths (ships with PostgreSQL) | 1.3.0 | Two requirements are worth calling out up front: -- **`pg_textsearch` must be preloaded.** It uses a shared library that has to be - configured before the server starts: +- **`pg_textsearch` must be preloaded.** It uses a shared library configured + before the server starts: ```conf shared_preload_libraries = 'pg_textsearch' ``` -- **The extensions must live in the `public` schema.** searchgres is opinionated - here to keep every reference unambiguous. `createIndex()` installs any missing - extension into `public`; if one already exists in another schema it fails with - an [`ExtensionError`](reference/errors.md) rather than moving it. +- **The extensions must live in `public`.** `createIndex()` installs a missing + extension into `public`; if one already exists in another schema it throws an + [`ExtensionError`](reference/errors.md) rather than moving it. ### Privileges -The role you use with `createIndex()` needs to: +The role used with `createIndex()` needs to: -- run `CREATE EXTENSION` the first time an extension is missing, and +- run `CREATE EXTENSION` the first time an extension is missing; - create schemas and objects. -If your database restricts extension creation, pre-install the three extensions -in `public` as a superuser; `createIndex()` then only needs schema/object -creation rights. +If extension creation is restricted, install all three in `public` with a +privileged role first. The application role then only needs the privileges +required to create and use its index schema. + +## Production PostgreSQL with Tiger Cloud + +[Tiger Cloud](https://www.tigerdata.com/cloud) is a turnkey managed PostgreSQL +option for production searchgres workloads. PostgreSQL 18 and all three required +extensions—`pgvector`, `pg_textsearch`, and `ltree`—are available on the +platform, so you do not need to build or operate a custom database image. + +Create a Tiger Cloud service, copy its PostgreSQL connection string into your +application's `DATABASE_URL`, and use it with `postgres.js` normally: + +```ts +const sql = postgres(process.env.DATABASE_URL, { ssl: "require" }); +``` + +Then call `createIndex()` with a role allowed to install the available +extensions and create schemas, or pre-install the extensions in `public` with an +administrative role before provisioning the index. Your application continues +to own its pool, index names, embedding provider, and worker deployment. + +For production concerns beyond database provisioning, see +[Run in production](guides/production.md). ## Database-only setup with Docker The repository includes a Dockerfile that builds PostgreSQL 18 with all three -extensions and the required preload configuration. Use this path when you want -to manage the Searchgres library/server and embedding provider yourself rather -than running the evaluation stack above. +extensions and the required preload configuration. Use it when you want to run +the core library against a local database while managing the application and +embedding provider yourself. ```bash -# Build the image +git clone https://github.com/timescale/searchgres.git +cd searchgres + docker build -t searchgres-postgres -f docker/Dockerfile.postgres docker/ -# Run it (trust auth for local development only) docker run -d --name searchgres-postgres \ -e POSTGRES_HOST_AUTH_METHOD=trust \ -p 127.0.0.1:5432:5432 \ searchgres-postgres ``` -Verify the extensions are available: +Trust authentication is for local development only. + +Verify extension availability: ```bash psql postgres://postgres@127.0.0.1:5432/postgres -c \ @@ -120,22 +112,11 @@ psql postgres://postgres@127.0.0.1:5432/postgres -c \ where name in ('vector','pg_textsearch','ltree') order by name;" ``` -If you are using the API server, configuration generation does not need this -database to be running. Generate and review files first, then provision: - -```bash -searchgres-server config -searchgres-server init --config searchgres.yaml -searchgres-server serve --config searchgres.yaml -``` - -See [Configure and run the API server](guides/server.md) for noninteractive -options, dotenv precedence, and strict `--if-not-exists` behavior. - ## You own the connection pool searchgres never creates, closes, or persistently reconfigures a connection. -Create the pool, pass it in, and close it yourself: +Create the pool, pass it to `createIndex()` and `openIndex()`, and close it when +your application shuts down: ```ts import postgres from "postgres"; @@ -143,13 +124,54 @@ import postgres from "postgres"; const sql = postgres(process.env.DATABASE_URL, { max: 10 }); try { - // pass `sql` to createIndex/openIndex and index methods + // Use sql with searchgres. } finally { await sql.end(); } ``` -You can point separate pools at different databases and run independent indexes -across them — see [Create and manage indexes](guides/indexes.md). +Separate pools can point at different databases, and several index handles can +share one pool. See [Create and manage indexes](guides/indexes.md). + +## Try without writing an application + +For a no-API-key evaluation, run the optional Compose stack: + +```bash +git clone https://github.com/timescale/searchgres.git +cd searchgres +docker compose up --build +``` + +It includes PostgreSQL, Ollama, automatic model download, strict index +provisioning, and the API server. This is an alternate evaluation path, not a +requirement for the core library. See +[Evaluate with Docker Compose](guides/docker-compose.md) for sample commands, +performance expectations, persistence, and its evaluation-only security +boundary. + +## Optional compiled applications + +Install the latest `searchgres` CLI, `searchgres-server`, and `searchgres-mcp` +executables: + +```bash +curl -fsSL https://raw.githubusercontent.com/timescale/searchgres/main/install.sh | sh +``` + +The installer detects Linux or macOS on amd64/arm64 (and Windows under a POSIX +shell), downloads matching GitHub release assets, and verifies individual +SHA-256 files before installing. The default destination is `~/.local/bin` when +`~/.local` exists, otherwise `~/bin`. + +Override the destination or release tag on the receiving shell: + +```bash +curl -fsSL https://raw.githubusercontent.com/timescale/searchgres/main/install.sh | \ + SEARCHGRES_INSTALL_DIR="$HOME/.local/bin" SEARCHGRES_VERSION=v0.1.0 sh +``` + +These applications are optional layers over the same core. Use them as reference +implementations or as-is for remote and agentic search. Next: [Get started](getting-started.md). diff --git a/examples/README.md b/examples/README.md new file mode 100644 index 0000000..a2eaa67 --- /dev/null +++ b/examples/README.md @@ -0,0 +1,33 @@ +# searchgres examples + +These examples use the published core library directly. They deliberately avoid +the optional API server so the application/library boundary is visible. + +| Example | Demonstrates | +| --- | --- | +| [basic-search](basic-search/) | Create, ingest, embed, and run filtered hybrid search | +| [rag-retriever](rag-retriever/) | A scoped retrieval function and local context formatting | +| [document-search](document-search/) | Stable chunks, tree organization, metadata, and derived records | +| [temporal-search](temporal-search/) | Events, periods, and temporal filters | +| [worker](worker/) | A separate continuous embedding worker process | + +All examples expect: + +- PostgreSQL 18 with `vector`, `pg_textsearch`, and `ltree` available in + `public`; +- `pg_textsearch` in `shared_preload_libraries`; +- `DATABASE_URL` and `OPENAI_API_KEY` in the environment; +- an index whose dimensions match `text-embedding-3-small`. + +From an example directory: + +```bash +npm install searchgres postgres @ai-sdk/openai +node index.ts +``` + +Running TypeScript directly requires Node 22.18 or newer. Alternatively use Bun, +Deno, or your application's TypeScript build. + +The programs create fixed example schemas and are intended for local databases. +Drop those schemas when finished or change the names before use. diff --git a/examples/basic-search/README.md b/examples/basic-search/README.md new file mode 100644 index 0000000..c794159 --- /dev/null +++ b/examples/basic-search/README.md @@ -0,0 +1,14 @@ +# Basic search + +Creates `example_basic`, writes two records, drains embeddings, and performs a +hybrid query scoped by tree and metadata. + +```bash +export DATABASE_URL=postgres://postgres@127.0.0.1:5432/postgres +export OPENAI_API_KEY=... +npm install searchgres postgres @ai-sdk/openai +node index.ts +``` + +The example drops its index at the end. Remove that line if you want to inspect +the schema afterward. diff --git a/examples/basic-search/index.ts b/examples/basic-search/index.ts new file mode 100644 index 0000000..a6ecb6b --- /dev/null +++ b/examples/basic-search/index.ts @@ -0,0 +1,48 @@ +import { openai } from "@ai-sdk/openai"; +import postgres from "postgres"; +import { createIndex, openIndex } from "searchgres"; + +const databaseUrl = process.env.DATABASE_URL; +if (!databaseUrl) throw new Error("DATABASE_URL is required"); +const sql = postgres(databaseUrl); + +try { + await createIndex(sql, "example_basic", { dimensions: 1536 }); + const index = await openIndex(sql, "example_basic", { + embedding: openai.embedding("text-embedding-3-small"), + }); + + await index.upsertMany([ + { + tree: "docs.auth", + name: "rotation", + content: "Authentication tokens rotate every 24 hours.", + meta: { audience: "operators" }, + }, + { + tree: "docs.api", + name: "limits", + content: "Each API key is limited to 100 requests per minute.", + meta: { audience: "developers" }, + }, + ]); + + console.log(await index.processEmbeddings()); + + const hits = await index.search({ + semantic: "how are requests throttled?", + fulltext: "API rate limit", + filter: { + and: [{ tree: "docs.api" }, { meta: { audience: "developers" } }], + }, + limit: 5, + }); + + for (const hit of hits) { + console.log(hit.score.toFixed(4), hit.tree, hit.content); + } + + await index.drop(); +} finally { + await sql.end(); +} diff --git a/examples/document-search/README.md b/examples/document-search/README.md new file mode 100644 index 0000000..0bd573f --- /dev/null +++ b/examples/document-search/README.md @@ -0,0 +1,12 @@ +# Document search + +Demonstrates caller-controlled records for source chunks and a derived summary. +Both representations live in `example_documents` and are separated by tree and +metadata rather than by a hidden extraction pipeline. + +```bash +export DATABASE_URL=postgres://postgres@127.0.0.1:5432/postgres +export OPENAI_API_KEY=... +npm install searchgres postgres @ai-sdk/openai +node index.ts +``` diff --git a/examples/document-search/index.ts b/examples/document-search/index.ts new file mode 100644 index 0000000..4033ce3 --- /dev/null +++ b/examples/document-search/index.ts @@ -0,0 +1,54 @@ +import { openai } from "@ai-sdk/openai"; +import postgres from "postgres"; +import { createIndex, openIndex } from "searchgres"; + +const databaseUrl = process.env.DATABASE_URL; +if (!databaseUrl) throw new Error("DATABASE_URL is required"); +const sql = postgres(databaseUrl); + +try { + await createIndex(sql, "example_documents", { dimensions: 1536 }); + const index = await openIndex(sql, "example_documents", { + embedding: openai.embedding("text-embedding-3-small"), + }); + + await index.upsertMany( + [ + { + tree: "knowledge.raw.api_limits", + name: "chunk-0", + content: "API keys allow 100 requests per minute.", + meta: { sourceId: "api-limits", kind: "raw", position: 0 }, + }, + { + tree: "knowledge.raw.api_limits", + name: "chunk-1", + content: "The service returns HTTP 429 after the quota is exhausted.", + meta: { sourceId: "api-limits", kind: "raw", position: 1 }, + }, + { + tree: "knowledge.summary.api_limits", + name: "current", + content: + "API usage is capped per minute and excess requests return 429.", + meta: { sourceId: "api-limits", kind: "summary" }, + }, + ], + { onConflict: "replace" }, + ); + await index.processEmbeddings(); + + const rawEvidence = await index.search({ + semantic: "what happens after exceeding the quota?", + fulltext: "quota HTTP 429", + filter: { + and: [{ tree: "knowledge.raw" }, { meta: { sourceId: "api-limits" } }], + }, + }); + + console.log( + rawEvidence.map(({ tree, name, content }) => ({ tree, name, content })), + ); +} finally { + await sql.end(); +} diff --git a/examples/rag-retriever/README.md b/examples/rag-retriever/README.md new file mode 100644 index 0000000..42dea2c --- /dev/null +++ b/examples/rag-retriever/README.md @@ -0,0 +1,15 @@ +# RAG retriever + +Shows a small application-owned RAG retrieval stage. It creates `example_rag`, +ingests stable chunks, applies a trusted tenant and visibility scope, and formats +full records as model context. + +The example prints context rather than calling a generation model so retrieval +can be inspected independently. + +```bash +export DATABASE_URL=postgres://postgres@127.0.0.1:5432/postgres +export OPENAI_API_KEY=... +npm install searchgres postgres @ai-sdk/openai +node index.ts +``` diff --git a/examples/rag-retriever/index.ts b/examples/rag-retriever/index.ts new file mode 100644 index 0000000..a4ccf73 --- /dev/null +++ b/examples/rag-retriever/index.ts @@ -0,0 +1,73 @@ +import { openai } from "@ai-sdk/openai"; +import postgres from "postgres"; +import { + createIndex, + type Filter, + type Index, + openIndex, + type SearchResult, +} from "searchgres"; + +async function retrieve( + index: Index, + question: string, + trustedScope: Filter, +): Promise { + return index.search({ + semantic: question, + fulltext: question, + filter: trustedScope, + candidateLimit: 50, + limit: 8, + }); +} + +function formatContext(hits: readonly SearchResult[]): string { + return hits + .map( + (hit, i) => + `\n` + + `${hit.content}\n`, + ) + .join("\n\n"); +} + +const databaseUrl = process.env.DATABASE_URL; +if (!databaseUrl) throw new Error("DATABASE_URL is required"); +const sql = postgres(databaseUrl); + +try { + await createIndex(sql, "example_rag", { dimensions: 1536 }); + const index = await openIndex(sql, "example_rag", { + embedding: openai.embedding("text-embedding-3-small"), + }); + + await index.upsertMany([ + { + tree: "tenants.acme.handbook.security", + name: "chunk-0", + content: "Production credentials rotate every 24 hours.", + meta: { sourceId: "handbook", position: 0, visibility: "internal" }, + }, + { + tree: "tenants.acme.handbook.api", + name: "chunk-0", + content: "API clients retry HTTP 429 responses with exponential backoff.", + meta: { sourceId: "handbook", position: 1, visibility: "internal" }, + }, + ]); + await index.processEmbeddings(); + + // Construct this filter from authenticated identity, not request input. + const trustedScope: Filter = { + and: [{ tree: "tenants.acme" }, { meta: { visibility: "internal" } }], + }; + const hits = await retrieve( + index, + "what should a client do when requests are throttled?", + trustedScope, + ); + console.log(formatContext(hits)); +} finally { + await sql.end(); +} diff --git a/examples/temporal-search/README.md b/examples/temporal-search/README.md new file mode 100644 index 0000000..cb43862 --- /dev/null +++ b/examples/temporal-search/README.md @@ -0,0 +1,11 @@ +# Temporal search + +Stores a point event and a validity period in `example_temporal`, then searches +by represented time. `temporal` is distinct from record creation/update time. + +```bash +export DATABASE_URL=postgres://postgres@127.0.0.1:5432/postgres +export OPENAI_API_KEY=... +npm install searchgres postgres @ai-sdk/openai +node index.ts +``` diff --git a/examples/temporal-search/index.ts b/examples/temporal-search/index.ts new file mode 100644 index 0000000..74f0ac2 --- /dev/null +++ b/examples/temporal-search/index.ts @@ -0,0 +1,51 @@ +import { openai } from "@ai-sdk/openai"; +import postgres from "postgres"; +import { createIndex, openIndex } from "searchgres"; + +const databaseUrl = process.env.DATABASE_URL; +if (!databaseUrl) throw new Error("DATABASE_URL is required"); +const sql = postgres(databaseUrl); + +try { + await createIndex(sql, "example_temporal", { dimensions: 1536 }); + const index = await openIndex(sql, "example_temporal", { + embedding: openai.embedding("text-embedding-3-small"), + }); + + await index.upsertMany([ + { + tree: "operations.incidents", + name: "incident-42", + content: "Elevated API latency during a database failover.", + temporal: ["2026-03-10T14:00:00Z", "2026-03-10T14:37:00Z"], + meta: { severity: 2 }, + }, + { + tree: "operations.releases", + name: "release-7", + content: "Version 7 was released.", + temporal: ["2026-03-10T15:00:00Z"], + meta: { version: 7 }, + }, + ]); + + const duringIncident = await index.search({ + filter: { + and: [ + { tree: "operations" }, + { temporalContains: "2026-03-10T14:15:00Z" }, + ], + }, + }); + + const marchEvents = await index.search({ + fulltext: "database release latency", + filter: { + temporalOverlaps: ["2026-03-01T00:00:00Z", "2026-04-01T00:00:00Z"], + }, + }); + + console.log({ duringIncident, marchEvents }); +} finally { + await sql.end(); +} diff --git a/examples/tsconfig.json b/examples/tsconfig.json new file mode 100644 index 0000000..a492646 --- /dev/null +++ b/examples/tsconfig.json @@ -0,0 +1,8 @@ +{ + "extends": "../tsconfig.base.json", + "compilerOptions": { + "noEmit": true, + "types": ["node"] + }, + "include": ["*/index.ts"] +} diff --git a/examples/worker/README.md b/examples/worker/README.md new file mode 100644 index 0000000..4206c26 --- /dev/null +++ b/examples/worker/README.md @@ -0,0 +1,17 @@ +# Separate embedding worker + +A process that opens `application_search` and continuously drains its embedding +queue. The application that writes records can run separately without embedding +provider credentials. + +Create the index from the writer or provisioning process before starting this +worker. + +```bash +export DATABASE_URL=postgres://postgres@127.0.0.1:5432/postgres +export OPENAI_API_KEY=... +npm install searchgres postgres @ai-sdk/openai +node index.ts +``` + +Send `SIGINT` or `SIGTERM` for graceful shutdown. diff --git a/examples/worker/index.ts b/examples/worker/index.ts new file mode 100644 index 0000000..8f0008b --- /dev/null +++ b/examples/worker/index.ts @@ -0,0 +1,30 @@ +import { openai } from "@ai-sdk/openai"; +import postgres from "postgres"; +import { openIndex } from "searchgres"; + +const databaseUrl = process.env.DATABASE_URL; +if (!databaseUrl) throw new Error("DATABASE_URL is required"); +const sql = postgres(databaseUrl); +const index = await openIndex(sql, "application_search", { + embedding: openai.embedding("text-embedding-3-small"), +}); +const worker = index.startEmbeddingWorker({ + batchSize: 100, + intervalMs: 1_000, + pruneRetentionMs: 7 * 24 * 60 * 60 * 1_000, +}); + +console.log(`draining ${index.schema}; press Ctrl-C to stop`); + +await new Promise((resolve) => { + let stopping = false; + const stop = async () => { + if (stopping) return; + stopping = true; + await worker.stop(); + await sql.end(); + resolve(); + }; + process.once("SIGINT", stop); + process.once("SIGTERM", stop); +}); diff --git a/package.json b/package.json index b9af1ee..6d1208e 100644 --- a/package.json +++ b/package.json @@ -15,7 +15,7 @@ "compile": "rm -rf dist && mkdir -p dist && ./bun run build && ./bun run --filter @searchgres/cli compile && ./bun run --filter @searchgres/server compile && ./bun run --filter @searchgres/mcp compile", "compile:all": "rm -rf dist && mkdir -p dist && ./bun run build && ./bun run --filter @searchgres/cli compile:all && ./bun run --filter @searchgres/server compile:all && ./bun run --filter @searchgres/mcp compile:all", "clean": "rm -rf dist && ./bun run --filter '*' clean", - "typecheck": "./bun run --filter searchgres build && ./bun run --filter @searchgres/protocol build && ./bun run --filter @searchgres/filter build && ./bun run --filter @searchgres/presentation build && ./bun run --filter @searchgres/client build && ./bun run --filter searchgres typecheck && ./bun run --filter @searchgres/protocol typecheck && ./bun run --filter @searchgres/filter typecheck && ./bun run --filter @searchgres/presentation typecheck && ./bun run --filter @searchgres/client typecheck && ./bun run --filter @searchgres/server typecheck && ./bun run --filter @searchgres/cli typecheck && ./bun run --filter @searchgres/mcp typecheck", + "typecheck": "./bun run --filter searchgres build && ./bun run --filter @searchgres/protocol build && ./bun run --filter @searchgres/filter build && ./bun run --filter @searchgres/presentation build && ./bun run --filter @searchgres/client build && ./bun run --filter searchgres typecheck && ./bun run --filter @searchgres/protocol typecheck && ./bun run --filter @searchgres/filter typecheck && ./bun run --filter @searchgres/presentation typecheck && ./bun run --filter @searchgres/client typecheck && ./bun run --filter @searchgres/server typecheck && ./bun run --filter @searchgres/cli typecheck && ./bun run --filter @searchgres/mcp typecheck && ./bun x tsc -p examples/tsconfig.json", "lint": "./bun x biome check", "format": "./bun x biome check --write", "test:unit": "./bun run --filter searchgres build && ./bun run --filter @searchgres/protocol build && ./bun run --filter @searchgres/filter build && ./bun run --filter @searchgres/presentation build && ./bun run --filter @searchgres/client build && ./bun run --filter searchgres test:unit && ./bun run --filter @searchgres/protocol test:unit && ./bun run --filter @searchgres/filter test:unit && ./bun run --filter @searchgres/presentation test:unit && ./bun run --filter @searchgres/client test:unit && ./bun run --filter @searchgres/server test:unit && ./bun run --filter @searchgres/cli test:unit && ./bun run --filter @searchgres/mcp test:unit && ./bun test ./scripts/*.test.ts", diff --git a/scripts/docs-links.test.ts b/scripts/docs-links.test.ts new file mode 100644 index 0000000..911d174 --- /dev/null +++ b/scripts/docs-links.test.ts @@ -0,0 +1,48 @@ +import assert from "node:assert/strict"; +import { existsSync, readdirSync, readFileSync } from "node:fs"; +import { dirname, extname, join, resolve } from "node:path"; +import { test } from "node:test"; + +function markdownFiles(path: string): readonly string[] { + if (extname(path) === ".md") return [path]; + return readdirSync(path, { withFileTypes: true }).flatMap((entry) => { + const child = join(path, entry.name); + if (entry.isDirectory()) return markdownFiles(child); + return entry.isFile() && extname(child) === ".md" ? [child] : []; + }); +} + +const files = [ + "README.md", + ...markdownFiles("docs"), + ...markdownFiles("examples"), +]; + +const markdownLink = /(? { + assert.ok(files.includes("docs/concepts/how-search-works.md")); + assert.ok(files.includes("examples/basic-search/README.md")); +}); + +test("relative links in public Markdown resolve", () => { + const broken: string[] = []; + for (const file of files) { + const content = readFileSync(file, "utf8"); + for (const match of content.matchAll(markdownLink)) { + const destination = match[1]; + if ( + !destination || + destination.startsWith("#") || + /^[a-z][a-z+.-]*:/i.test(destination) + ) { + continue; + } + const path = destination.split("#", 1)[0]; + if (path && !existsSync(resolve(dirname(file), path))) { + broken.push(`${file} -> ${destination}`); + } + } + } + assert.deepEqual(broken, []); +});