From fd938e5727a25a40555914ed7e99f77d9d6046c9 Mon Sep 17 00:00:00 2001 From: phernandez Date: Tue, 11 Aug 2026 23:10:24 -0500 Subject: [PATCH 1/5] docs: prepare v0.23 release documentation --- content/0.welcome.md | 2 +- content/01.start-here/3.quickstart-local.md | 2 +- content/02.whats-new/2.v0-23-upgrade.md | 81 +++++++ content/02.whats-new/3.changelog.md | 14 ++ content/03.cloud/05.cloud-sync.md | 33 ++- content/03.cloud/12.cloud-cli.md | 54 +++++ content/06.local/5.user-guide.md | 2 +- content/06.local/6.canvas.md | 128 ------------ content/06.local/6.postgres-and-milvus.md | 150 +++++++++++++ content/07.concepts/1.knowledge-format.md | 13 +- .../3.observations-and-relations.md | 16 ++ content/07.concepts/7.semantic-search.md | 84 +++++++- content/08.integrations/01.claude-desktop.md | 2 +- content/08.integrations/02.claude-code.md | 11 +- content/08.integrations/03.codex.md | 11 +- content/08.integrations/09.vscode.md | 2 +- content/08.integrations/10.obsidian.md | 3 - content/08.integrations/12.harness-capture.md | 103 +++++++++ content/10.reference/1.cli-reference.md | 114 +++++++++- content/10.reference/2.mcp-tools-reference.md | 43 ++-- content/10.reference/5.troubleshooting.md | 25 +-- content/10.reference/6.configuration.md | 197 ++++++++++++++++-- content/10.reference/7.docker.md | 8 +- nuxt.config.ts | 3 +- 24 files changed, 905 insertions(+), 196 deletions(-) create mode 100644 content/02.whats-new/2.v0-23-upgrade.md delete mode 100644 content/06.local/6.canvas.md create mode 100644 content/06.local/6.postgres-and-milvus.md create mode 100644 content/08.integrations/12.harness-capture.md diff --git a/content/0.welcome.md b/content/0.welcome.md index ba40a11..328ecd7 100644 --- a/content/0.welcome.md +++ b/content/0.welcome.md @@ -11,7 +11,7 @@ seo: **Basic Memory** is real memory for your AI — a knowledge base you own. Built-in AI memory is small, vendor-owned, and usually invisible. Basic Memory is large, structured, and inspectable: a knowledge graph written in plain Markdown. Open it in any editor, hand it to any AI tool, and keep it forever. ::note{icon="i-lucide-megaphone"} -**New in Basic Memory Cloud:** review notes change by change with [Comments and Suggestions](/whats-new/comments-and-suggestions), and browse, edit, or graph your knowledge directly inside ChatGPT and Claude with the [Interactive MCP App](/whats-new/interactive-mcp-app). +**New in v0.23:** the search release — reranking, Milvus, and a deadlock-free indexing core. See the [Changelog](/whats-new/changelog). Also new in Basic Memory Cloud: [Comments and Suggestions](/whats-new/comments-and-suggestions) and the [Interactive MCP App](/whats-new/interactive-mcp-app) for ChatGPT and Claude. :: ## Pick your path diff --git a/content/01.start-here/3.quickstart-local.md b/content/01.start-here/3.quickstart-local.md index 00bb2ff..b3b0d30 100644 --- a/content/01.start-here/3.quickstart-local.md +++ b/content/01.start-here/3.quickstart-local.md @@ -218,7 +218,7 @@ Restart Claude Desktop for changes to take effect. For more on projects and defa ## Useful CLI commands ```bash -# Check sync status +# Check project index status basic-memory status # List all projects diff --git a/content/02.whats-new/2.v0-23-upgrade.md b/content/02.whats-new/2.v0-23-upgrade.md new file mode 100644 index 0000000..dd7c4d9 --- /dev/null +++ b/content/02.whats-new/2.v0-23-upgrade.md @@ -0,0 +1,81 @@ +--- +title: Upgrade to v0.23 +description: Breaking changes, migrations, and required actions when upgrading Basic Memory from v0.22.1 to v0.23. +--- + +v0.23 applies database migrations automatically, but integrations and operator scripts need review. Complete the actions below before relying on existing automation. + +## Rebuild vectors after changing indexes + +If you change `semantic_vector_index` between `pgvector` and `milvus`, rebuild the configured vector store: + +```bash +bm reindex --embeddings +``` + +The default incremental run is enough. Before it finishes, vector search returns no results and hybrid search quietly serves full-text results only. See [Run Postgres with Milvus](/local/postgres-and-milvus) for the full switch and recovery procedure. + +Milvus requires the `basic-memory[milvus]` extra. Standalone Redis read caching requires `basic-memory[redis]`. Reranking remains off by default and downloads its local model on the first reranked search. + +## Update CLI automation + +- `bm status` now reports project index status instead of the old sync-report tree. Scripts that parse its human output must switch to `--json` and the new shape. +- `bm reindex --embeddings` exits nonzero when embedding fails. Treat that exit as a real failed rebuild instead of assuming success. +- Interactive `bm tool` commands render Rich output on a TTY. Piped output remains machine-readable; pass `--json` when a script requires a stable contract. +- The new top-level groups are `bm config` and `bm hook`. The top-level command set otherwise remains intact; `bm sync` was already absent in v0.22.1. + +## Update MCP clients + +- The `canvas`, `cloud_info`, and `release_notes` MCP tools are removed. Obsidian Canvas generation and the API resource-write endpoints behind it are no longer available. +- The ChatGPT compatibility tools `search` and `fetch` now accept OpenAI MCP clients only. Other clients must use `search_notes` and `read_note`. +- `list_directory` is bounded and paginated. Consumers must use `page` and `page_size` instead of assuming one exhaustive response. +- `replace_section` is heading-level-aware. By default it replaces the selected section through the next heading of the same or higher level, including nested subsections. Set `replace_subsections=false` to preserve nested subsections. +- Ambiguous identifiers now raise an error instead of returning a best guess, and `write_note` rejects filename-convention twins instead of creating a duplicate. +- FastMCP moves to `4.0.0b1` with MCP SDK v2. Downstream embedders pinned to FastMCP 3.x must upgrade with Basic Memory. + +## Update HTTP clients + +Resource write endpoints and pre-v0.18.0 legacy routes are removed. Old clients that still call those routes receive `404` responses. The current v2 API remains the supported surface. + +## Update configuration + +Basic Memory automatically migrates these config-file keys and legacy environment variables: + +| Old | New | +|---|---| +| `sync_delay` | `index_delay` | +| `sync_changes` | `index_changes` | + +When old and new names are both set, the new name wins. + +`sync_thread_pool_size` and `sync_max_concurrent_files` are removed without aliases and are silently ignored in `config.json`. Remove them. `materialization_workers` is the nearest replacement for bounding local write materialization, but it is not a one-for-one thread-pool setting. + +## Review behavior changes + +::warning +`bm cloud sync` is a one-way mirror and now deletes a previously synced cloud file when that file becomes ignored by `.bmignore`. Preview the exact deletion set with `bm cloud sync --name --dry-run`, or use the additive `bm cloud push` and `bm cloud pull` workflow. +:: + +- Deleting the default project now chooses a new default instead of refusing the deletion. +- `created` and `modified` frontmatter timestamps are authoritative. Historical values can change recency ordering after the first reindex. +- Timestamp-shaped labels and Markdown checkbox markers are no longer indexed as observation categories. +- Note-type filters are case-canonicalized, so values such as `Person` and `person` now match the same population. + +## Let migrations finish + +Five revisions run automatically on first start: + +1. A durable note-file vacate record distinguishes moves from byte-identical copies. +2. The semantic manifest gains vector-index identity and readiness state. +3. Relation-derived search refreshes become durable, retryable work items. +4. Relations and their search refreshes gain generation ownership. +5. A one-time repair removes duplicate observations and orphaned observation rows from full-text search. + +The final repair is not reversible through a downgrade. Markdown files remain the source of truth and the index is rebuildable, but back up `memory.db` before the upgrade if you want a pre-migration database snapshot. + +## Continue setup + +- [Semantic Search](/concepts/semantic-search) — enable and tune reranking. +- [Run Postgres with Milvus](/local/postgres-and-milvus) — start local Postgres, configure Milvus, and switch indexes safely. +- [Configuration](/reference/configuration) — review every new v0.23 setting. +- [CLI Reference](/reference/cli-reference) — update commands and output contracts. diff --git a/content/02.whats-new/3.changelog.md b/content/02.whats-new/3.changelog.md index bb7ce98..5a9e33a 100644 --- a/content/02.whats-new/3.changelog.md +++ b/content/02.whats-new/3.changelog.md @@ -7,6 +7,20 @@ Notable changes in each Basic Memory release, most recent first. For complete pa --- +## v0.23 — Search that ranks and scales + +*[v0.23.0](https://github.com/basicmachines-co/basic-memory/releases/tag/v0.23.0)* + +Semantic search takes a major step forward in v0.23. Opt-in cross-encoder reranking can rescore vector and hybrid candidates locally with FastEmbed or through LiteLLM, while Postgres deployments can move vector storage from pgvector to Milvus, Milvus Lite, or Zilliz Cloud. File-watcher writes now receive embeddings, non-BGE FastEmbed models are normalized correctly, SQLite full-text search covers complete long notes and relaxed CJK queries, and reindex failures are reported honestly. + +The operator surface expands with **`bm config`** for validated settings, **`bm hook`** for harness lifecycle capture, Rich interactive output for **`bm tool`**, a diagnostics MCP tool, a usable local Postgres path, Cloud share management, and targeted **`bm cloud prune`** cleanup. + +Underneath those features, the indexing and persistence core moves to generation-versioned relations and observations, compare-and-swap materialization, consistent lock ordering, and batched resolution. Observations and relations now land with accepted writes, eliminating the deadlock and stale-projection failures that appeared under concurrent agent workloads. + +This release also removes MCP tools and legacy API routes and changes several CLI, config, sync, edit, and pagination contracts. Read [Upgrade to v0.23](/whats-new/v0-23-upgrade) before upgrading an integration or automation. + +--- + ## August 2026 — Cloud review and interactive MCP Apps *August 2026 · Basic Memory Cloud* diff --git a/content/03.cloud/05.cloud-sync.md b/content/03.cloud/05.cloud-sync.md index 7d148f9..49ae46a 100644 --- a/content/03.cloud/05.cloud-sync.md +++ b/content/03.cloud/05.cloud-sync.md @@ -24,10 +24,11 @@ The cloud CLI enables you to: | `bm cloud pull` | cloud → local | **additive** — never deletes local | ✅ | ✅ | | `bm cloud push` | local → cloud | **additive** — never deletes cloud | ✅ | ✅ | | `bm cloud sync` | local → cloud | **mirror** — deletes cloud files missing locally | ✅ | ❌ | +| `bm cloud prune` | cloud cleanup | Deletes cloud files matching this machine's `.bmignore` | ✅ | ❌ | **`push` and `pull` are the standard workflow.** They are additive — they never delete files on the destination — and conflict-aware, so they are safe everywhere, including shared Team workspaces where a mirror operation could delete a teammate's files. -`sync` is a mirror operation: your local tree becomes authoritative and cloud files missing locally get deleted. That can be useful on a Personal workspace when local is the single source of truth, but it is blocked on Team workspaces, where it exits early with a clear error pointing you at `push`/`pull`. +`sync` is a mirror operation: your local tree becomes authoritative and cloud files missing locally get deleted. That can be useful on a Personal workspace when local is the single source of truth, but it is blocked on Team workspaces, where it exits early with a clear error pointing you at `push`/`pull`. In v0.23, this deletion set also includes previously synced files that now match `.bmignore`. --- @@ -203,6 +204,7 @@ You should see: | `bm cloud pull` | Cloud → Local | Fetch cloud changes additively (Personal + Team) | | `bm cloud push` | Local → Cloud | Upload local changes additively (Personal + Team) | | `bm cloud sync` | Local → Cloud | One-way mirror, make cloud match local (Personal only) | +| `bm cloud prune` | Cloud cleanup | Delete files that match this machine's `.bmignore` (Personal only) | | `bm cloud check` | Verify | Check if files match (Personal only) | ### Push and Pull (Additive, Git-Style) @@ -255,6 +257,27 @@ bm cloud sync --name research Makes cloud identical to local — including deleting cloud files that are missing locally. Use when local is the source of truth. Personal workspaces only; on Team workspaces use `bm cloud push` (additive). +::warning +Starting in v0.23, adding a pattern to `.bmignore` makes a previously synced matching file eligible for deletion from cloud on the next `bm cloud sync`. Preview with `bm cloud sync --name research --dry-run` before applying a new ignore rule. Use additive `push`/`pull` if you do not want local deletions or ignore changes propagated. +:: + +### Prune newly ignored cloud files + +Use `prune` when you want only the `.bmignore` cleanup without a full mirror sync: + +```bash +# Show the exact remote paths that match this machine's ignore file +bm cloud prune --name research --dry-run + +# Show the paths again, then ask for confirmation before deletion +bm cloud prune --name research + +# Skip the confirmation prompt after reviewing a dry run +bm cloud prune --name research --yes +``` + +`prune` can run without a configured local sync directory because it inspects remote paths against `~/.basic-memory/.bmignore`. It is Personal-workspace-only and permanently deletes only the exact paths shown in its preview. + ### Preview Changes (Dry Run) ```bash @@ -354,6 +377,10 @@ config.json Basic Memory also respects `.gitignore` files in your projects. Use `.bmignore` for global patterns across all projects. :: +::warning +Ignore rules are part of one-way mirror retention behavior. A file that was uploaded before you added a matching `.bmignore` rule is deleted by `bm cloud sync`, or on demand by `bm cloud prune`. Always preview either operation before changing retention-sensitive patterns. +:: + --- ## Troubleshooting @@ -451,6 +478,10 @@ bm cloud pull --name --on-conflict keep-both bm cloud sync --name bm cloud sync --name --dry-run +# Targeted deletion of cloud files matching ~/.basic-memory/.bmignore +bm cloud prune --name --dry-run +bm cloud prune --name + # Integrity check (Personal only) bm cloud check --name diff --git a/content/03.cloud/12.cloud-cli.md b/content/03.cloud/12.cloud-cli.md index df64833..49e223a 100644 --- a/content/03.cloud/12.cloud-cli.md +++ b/content/03.cloud/12.cloud-cli.md @@ -94,6 +94,42 @@ bm project list --- +## Manage public share links + +Create a public link for one note, optionally with an expiration: + +```bash +bm cloud share create research notes/my-idea +bm cloud share create research notes/my-idea --expires-at 2099-12-31 +``` + +List all links or filter them by project: + +```bash +bm cloud share list +bm cloud share list --project research +``` + +Disable a link without deleting it, enable it again, or update its expiration: + +```bash +bm cloud share update SHARE_TOKEN --disable +bm cloud share update SHARE_TOKEN --enable +bm cloud share update SHARE_TOKEN --expires-at 2099-12-31 +bm cloud share update SHARE_TOKEN --expires-at none +``` + +Revoke a link permanently: + +```bash +bm cloud share revoke SHARE_TOKEN +bm cloud share revoke SHARE_TOKEN --force +``` + +Use `--workspace ` on any share command when the link belongs to a non-default workspace or when a project name is ambiguous. Create resolves the workspace from the explicit option, the project's configured workspace, or your default workspace; the other commands can route by the explicit workspace. + +--- + ## Route selected local projects through Cloud Routing is for hybrid setups where you have the CLI or a local MCP server installed, but want specific projects to use Cloud. @@ -131,6 +167,24 @@ bm cloud push --name research See [Cloud Sync](/cloud/cloud-sync) for the full workflow reference, including conflict handling with `--on-conflict`. +The Personal-workspace mirror command can delete cloud files, including files that become newly ignored: + +```bash +bm cloud sync --name research --dry-run +bm cloud sync --name research +``` + +Delete only remote files that match this machine's `.bmignore`: + +```bash +bm cloud prune --name research --dry-run +bm cloud prune --name research +``` + +::warning +Starting in v0.23, `bm cloud sync` deletes a previously synced cloud file when it becomes ignored by `.bmignore`. Preview the mirror or prune operation before confirming deletion. Additive `bm cloud push` and `bm cloud pull` do not delete destination files. +:: + --- ## Recover content from the terminal diff --git a/content/06.local/5.user-guide.md b/content/06.local/5.user-guide.md index 25023dd..9a56483 100644 --- a/content/06.local/5.user-guide.md +++ b/content/06.local/5.user-guide.md @@ -449,7 +449,7 @@ No manual sync needed for normal workflows. ### Checking Status ```bash -# View sync status +# View project index status basic-memory status # View project information diff --git a/content/06.local/6.canvas.md b/content/06.local/6.canvas.md deleted file mode 100644 index 6f3395f..0000000 --- a/content/06.local/6.canvas.md +++ /dev/null @@ -1,128 +0,0 @@ ---- -title: Canvas Visualizations -description: Create visual knowledge maps using Obsidian's Canvas feature to understand relationships between concepts ---- - -Basic Memory can create visual knowledge maps using Obsidian's Canvas feature. These visualizations help you understand relationships between concepts, map out processes, and visualize your knowledge structure. - -::note -Canvas is a **local-only** tool — it writes `.canvas` files for [Obsidian](/integrations/obsidian) alongside your notes. In cloud workspaces, use the [Explore graph view](/cloud/web-app#explore-the-graph) instead. -:: - -## Creating Canvas Visualizations - -Ask Claude to create a visualization by describing what you want to map: - -**Example requests:** - -- "Create a canvas visualization of my project components and their relationships." -- "Make a concept map showing the main themes from our discussion about climate change." -- "Can you make a canvas diagram of the perfect pour over method?" - -Canvas Visualization Example - -## Types of Visualizations - -Basic Memory can create several types of visual maps: - -- **Document Maps** — Visualize connections between your notes and documents -- **Concept Maps** — Create visual representations of ideas and their relationships -- **Process Diagrams** — Map workflows, sequences, and procedures -- **Thematic Analysis** — Organize ideas around central themes -- **Relationship Networks** — Show how different entities relate to each other in your knowledge base - -## Visualization Sources - -Claude can create visualizations based on: - -### Documents in Your Knowledge Base - -```example -You: "Create a canvas showing the connections between my project planning documents" -``` - -### Conversation Content - -```example -You: "Make a canvas visualization of the main points we just discussed" -``` - -### Search Results - -```example -You: "Find all my notes about psychology and create a visual map of the concepts" -``` - -### Themes and Relationships - -```example -You: "Create a visual map showing how different philosophical schools relate to each other" -``` - -## Visualization Workflow - -::steps -### Request a visualization -Describe what you want to see by asking Claude to create a canvas - -### Claude creates the canvas file -The canvas file is automatically created in your Basic Memory directory - -### Open the file in Obsidian -View the visualization in Obsidian's Canvas feature - -### Refine the visualization -Ask Claude for adjustments: -- "Could you reorganize the canvas to group related components together?" -- "Please add more detail about the connection between these two concepts." - -:: - -## Technical Details - -Behind the scenes, the AI: - -::note -1. Creates a `.canvas` file in JSON format -2. Adds nodes for each concept or document -3. Creates edges to represent relationships -4. Sets positions for visual clarity -5. Includes any relevant metadata -:: - -The resulting file is fully compatible with Obsidian's Canvas feature and can be edited directly in Obsidian. - -## Tips for Effective Visualizations - -- **Be Specific** — Clearly describe what you want to visualize -- **Specify Detail Level** — Mention how much detail you need -- **Mention Type** — Specify the visualization type (concept map, process flow, etc.) -- **Start Simple** — Begin with basic visualizations and ask for refinements - -::note -**Provide context** about what documents or concepts to include for the best results. -:: - ---- - -## Next Steps - -:::card-group -::card ---- -title: Obsidian Integration -icon: i-lucide-hexagon -to: /integrations/obsidian ---- -Open your canvases and vault in Obsidian. -:: - -::card ---- -title: Knowledge Format -icon: i-lucide-file-text -to: /concepts/knowledge-format ---- -The note structure canvases visualize. -:: -::: diff --git a/content/06.local/6.postgres-and-milvus.md b/content/06.local/6.postgres-and-milvus.md new file mode 100644 index 0000000..d7846ad --- /dev/null +++ b/content/06.local/6.postgres-and-milvus.md @@ -0,0 +1,150 @@ +--- +title: Postgres and Milvus +description: Run Basic Memory on local Postgres and use pgvector, Milvus Lite, Milvus, or Zilliz Cloud for semantic vectors. +--- + +SQLite remains the simplest local backend. Use this guide when you need to exercise the Postgres runtime or move a Postgres deployment's vector storage from pgvector to Milvus. + +## Start local Postgres + +Basic Memory's Postgres backend requires the pgvector extension even when Milvus will own semantic vectors. Create `docker-compose-postgres.yml` with the same local service used by the Basic Memory repository: + +```yaml [docker-compose-postgres.yml] +services: + postgres: + image: pgvector/pgvector:pg17 + container_name: basic-memory-postgres + environment: + POSTGRES_DB: basic_memory + POSTGRES_USER: basic_memory_user + POSTGRES_PASSWORD: dev_password + ports: + - "5433:5432" + volumes: + - postgres_data:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U basic_memory_user -d basic_memory"] + interval: 10s + timeout: 5s + retries: 5 + +volumes: + postgres_data: +``` + +Start the database: + +```bash +docker compose -f docker-compose-postgres.yml up -d +docker compose -f docker-compose-postgres.yml ps +``` + +Point Basic Memory at it before starting the MCP server or running a CLI command: + +```bash +export BASIC_MEMORY_DATABASE_BACKEND=postgres +export BASIC_MEMORY_DATABASE_URL='postgresql+asyncpg://basic_memory_user:dev_password@localhost:5433/basic_memory' + +bm status +``` + +The first startup applies migrations and seeds the default project automatically. The credentials above are for a private local development database only; choose real credentials and network controls for any shared deployment. + +::note +Plain `postgres:17` does not include pgvector. Use the `pgvector/pgvector:pg17` image or run `CREATE EXTENSION IF NOT EXISTS vector;` on an existing Postgres server before Basic Memory migrates it. +:: + +## Use the default pgvector index + +Postgres uses pgvector unless you change `semantic_vector_index`. Build embeddings for already indexed notes with: + +```bash +bm reindex --embeddings +``` + +SQLite ignores `semantic_vector_index` and continues to use sqlite-vec. + +## Install Milvus support + +Install Basic Memory with the optional Milvus dependency: + +::code-group +```bash [uv] +uv tool install 'basic-memory[milvus]' +``` + +```bash [pip] +pip install 'basic-memory[milvus]' +``` +:: + +The extra supports Milvus Lite on macOS and Linux and remote Milvus or Zilliz Cloud on every supported platform. Windows cannot run Milvus Lite but can connect to a remote deployment. + +## Configure Milvus Lite + +Create the parent directory first, then use an absolute URI ending in `.db`: + +```bash +mkdir -p "$PWD/.basic-memory-milvus" +bm config set semantic_vector_index milvus +bm config set milvus_uri "$PWD/.basic-memory-milvus/basic-memory.db" +``` + +Despite the suffix, Milvus Lite creates a **directory tree** at that path, including `collections/`, `databases/`, and `LOCK`; it does not create one database file. Pymilvus rejects a URI without the `.db` suffix or a URI whose parent directory does not already exist. + +`milvus_database` is ignored by Milvus Lite because Lite does not support named databases. Leave its default value in place. + +## Connect remote Milvus or Zilliz + +```bash +bm config set semantic_vector_index milvus +bm config set milvus_uri http://localhost:19530 +bm config set milvus_token root:Milvus +``` + +For Zilliz Cloud, set its endpoint and token instead. `milvus_timeout_seconds` applies to every client operation and defaults to 30 seconds. Collections reload across processes, so a restart reuses an existing compatible collection. + +## Reindex after every index switch + +Run an incremental embedding rebuild immediately after changing `semantic_vector_index`: + +```bash +bm reindex --embeddings +``` + +The incremental rebuild is sufficient; `--full` is not required. Until the configured index has ready rows: + +- vector search returns no results; +- hybrid search quietly serves full-text results only; and +- `bm reindex --embeddings` reports which index it populated and exits nonzero on failure. + +Hands-on verification found identical vector rankings and scores to four decimal places across sqlite-vec, pgvector, and Milvus Lite on the same corpus. Basic Memory also refuses dimension mismatches, reports a missing Milvus extra directly, honors the configured timeout, and reloads collections after restart. + +## Switch indexes safely + +Vector-index ownership is fail-closed. If a switch or switch-back finds manifest rows owned by another index, Basic Memory refuses to mutate them instead of guessing which vector store is authoritative. + +1. Restore the previously configured index if a premature switch raises `SemanticVectorIndexExtensionError`. +2. Remove the current index's project-scoped vectors and the corresponding vector-manifest rows while that index is still configured. +3. Change `semantic_vector_index`. +4. Run `bm reindex --embeddings` and confirm it exits successfully. + +There is no general cleanup command yet. Treat manual vector-store and manifest cleanup as an operator procedure and back up the database first. + +::warning +Switching from pgvector to Milvus leaves the old rows in Postgres's `search_vector_embeddings` projection. They are not used by the Milvus search path, but they continue to consume disk until you remove them manually. Do not delete them until the Milvus rebuild is successful and you have decided not to switch back using that projection. +:: + +## Stop local Postgres + +```bash +docker compose -f docker-compose-postgres.yml down +``` + +Add `--volumes` only when you intentionally want to delete the local database volume. + +## Related pages + +- [Semantic Search](/concepts/semantic-search) — retrieval modes, reranking, and reindex behavior. +- [Configuration](/reference/configuration) — all Postgres, Milvus, and embedding settings. +- [Upgrade to v0.23](/whats-new/v0-23-upgrade) — breaking changes and migration notes. diff --git a/content/07.concepts/1.knowledge-format.md b/content/07.concepts/1.knowledge-format.md index e0201b5..a9402b0 100644 --- a/content/07.concepts/1.knowledge-format.md +++ b/content/07.concepts/1.knowledge-format.md @@ -23,6 +23,8 @@ title: Authentication Design type: note tags: [auth, security, backend] permalink: authentication-design +created: 2026-08-01T14:30:00Z +modified: 2026-08-10T09:15:00Z --- # Authentication Design @@ -53,6 +55,8 @@ title: Authentication Design type: note tags: [auth, security, backend] permalink: authentication-design +created: 2026-08-01T14:30:00Z +modified: 2026-08-10T09:15:00Z --- ``` @@ -60,6 +64,9 @@ permalink: authentication-design - **type** — What kind of note this is (e.g. `note`, `meeting`, `decision`). You can use any type you want. - **tags** — For organization and filtering. - **permalink** — A stable identifier for this note. Generated automatically from the file's path (and prefixed with the project name by default) if you don't set one. Stays the same even if you later move the file. +- **created** and **modified** — Optional authoritative timestamps for the note. When present, Basic Memory uses them for recency ordering instead of filesystem timestamps. + +Historical `created` or `modified` values can move a note earlier in `recent_activity` and date-sorted search results after the note is reindexed. Use valid date or datetime values and change them only when you intend to change the note's recorded chronology. ::tip Frontmatter is just standard YAML. You can add any fields you want — `status`, `priority`, `author`, `due_date`, whatever is useful for your workflow. The AI can set these automatically when creating notes, and you can search by them later with metadata search. If you want to formalize which fields a note type should have, that's what [schemas](/concepts/schema-system) are for. @@ -85,7 +92,7 @@ Categories can be anything that makes sense for what you're capturing — `[deci The power of observations is that each one is indexed individually. When you search your knowledge base, Basic Memory can surface the specific fact you need rather than just pointing you at a whole document. See [Semantic Search](/concepts/semantic-search) for how this works. ::note -Basic Memory recognizes `[category]` syntax as observations but won't confuse them with markdown checkboxes (`[ ]` or `[x]`). +Basic Memory recognizes `[category]` syntax as observations but does not treat Markdown checkbox markers such as `[ ]`, `[x]`, `[X]`, `[-]`, `[/]`, `[>]`, and `[?]`, or timestamp-shaped transcript prefixes, as categories. :: For category conventions, tags, and worked examples, see the deep dive: [Observations and Relations](/concepts/observations-and-relations). @@ -142,6 +149,10 @@ Permalinks stay the same even if you rename or move the file. For more on how me Organize your files however you want — the knowledge graph is built from the content of your notes, not from where they sit on disk. See [Projects and Folders](/concepts/projects-and-folders) for organizing with projects and folder structure. +::note +`write_note` rejects filename-convention twins instead of creating a duplicate. For example, if a folder already contains the conventional equivalent of `Project Plan.md`, writing `project-plan.md` returns an error. Read or edit the existing note, or choose a genuinely different name. +:: + --- ## Schemas diff --git a/content/07.concepts/3.observations-and-relations.md b/content/07.concepts/3.observations-and-relations.md index 9141a3a..ef69321 100644 --- a/content/07.concepts/3.observations-and-relations.md +++ b/content/07.concepts/3.observations-and-relations.md @@ -29,6 +29,22 @@ The category (the word in brackets) classifies the kind of fact. The rest of the Categories help your AI assistant understand what kind of information it's looking at. When it builds context from your knowledge base, it can distinguish a decision from a risk, a preference from a requirement. That distinction matters when it's helping you reason through a problem. +### Avoid checkbox and timestamp markers + +Observation categories are semantic labels, not task state or transcript time. Basic Memory does not index these Markdown checkbox markers as categories: + +```markdown +- [ ] Not started +- [x] Complete +- [X] Complete +- [-] Canceled +- [/] In progress +- [>] Deferred +- [?] Needs clarification +``` + +Timestamp-shaped prefixes on transcript lines are also ignored as categories. Use a descriptive label such as `[decision]`, `[status]`, or `[follow-up]` when you want the line indexed as an observation. + ### Tags within observations The `#hashtags` at the end of an observation line add extra searchability: diff --git a/content/07.concepts/7.semantic-search.md b/content/07.concepts/7.semantic-search.md index 450bc6a..15e971c 100644 --- a/content/07.concepts/7.semantic-search.md +++ b/content/07.concepts/7.semantic-search.md @@ -17,7 +17,10 @@ code: | Q --> V[Vector Search] T --> SF[Score Fusion] V --> SF - SF --> R[Results] + SF --> RR{Reranking enabled?} + RR -->|No| R[Results] + RR -->|Yes| X[Cross-encoder rerank] + X --> R --- :: @@ -147,16 +150,87 @@ Semantic search works automatically — there's nothing to set up. It's included For a few hundred notes, expect 1–3 minutes for the initial index build. After that, new and edited notes are indexed incrementally during normal sync. -To manually rebuild the search index (e.g., after switching providers): +To manually rebuild the search index (for example, after switching embedding providers): ```bash bm reindex --embeddings ``` +Changing the Postgres vector index between pgvector and Milvus also requires this command. Until it completes, vector search returns nothing and hybrid search serves full-text results only. Follow [Run Postgres with Milvus](/local/postgres-and-milvus) for setup, switching, and recovery details. + Running on an Intel Mac? See [Platform compatibility](#platform-compatibility) below for a workaround. --- +## Enable reranking + +Reranking adds a cross-encoder after vector or hybrid retrieval. Retrieval finds a candidate set; the reranker reads the query and each candidate together, then replaces the leading candidates' scores with a more precise relevance order. Text-only search is unchanged. + +Reranking is opt-in: + +```bash +bm config set reranker_enabled true +``` + +The default `fastembed` provider runs locally with `jinaai/jina-reranker-v1-tiny-en`. The first reranked search downloads the model, so expect that first request to take longer. Later requests reuse the cached process-wide model. + +To use a hosted or self-hosted reranker through LiteLLM: + +```bash +bm config set reranker_enabled true +bm config set reranker_provider litellm +bm config set reranker_model cohere/rerank-v3.5 +export COHERE_API_KEY=... +``` + +LiteLLM models use explicit `provider/model` names. You can set `reranker_api_base` for a custom endpoint and `reranker_api_key` to pass the credential directly; otherwise LiteLLM uses its normal provider environment variables. + +### Choose a provider + +| Provider | Runs | Default model | Tradeoff | +|---|---|---|---| +| `fastembed` | Local ONNX | `jinaai/jina-reranker-v1-tiny-en` | No API key or per-query cost; downloads on first use and adds local inference latency. | +| `litellm` | Hosted or self-hosted API | Set an explicit `provider/model` | Supports Cohere, Jina, Voyage, and other LiteLLM rerank providers; adds network latency, provider cost, and credential requirements. | + +### Configure reranking + +| Config field | Default | Description | +|---|---|---| +| `reranker_enabled` | `false` | Enable cross-encoder reranking for vector and hybrid search; semantic search must also be enabled. | +| `reranker_provider` | `fastembed` | Use local `fastembed` or API-backed `litellm`. | +| `reranker_model` | `jinaai/jina-reranker-v1-tiny-en` | Model identifier; LiteLLM requires `provider/model` form. | +| `reranker_candidates` | `20` | Number of leading retrieval candidates to rescore. Larger windows can improve recall but cost more time or provider usage. | +| `reranker_max_document_chars` | `0` | Maximum candidate characters sent to the cross-encoder. `0` sends the full matched text. | +| `reranker_timeout` | `30.0` | Maximum seconds per LiteLLM request. FastEmbed ignores this setting. | +| `reranker_api_base` | unset | Custom API base for a self-hosted LiteLLM reranker. | +| `reranker_api_key` | unset | Credential passed directly to LiteLLM; normal provider environment variables work when unset. | + +Every field is also available as `BASIC_MEMORY_`. See [Configuration](/reference/configuration#reranker-settings) for types and environment-variable names. + +### Tune latency + +Reranking cost depends heavily on candidate length and count: + +- Short-note benchmarks add about **90 ms at p50** with the default local model and settings. +- In a long transcript-chunk benchmark, p95 latency rose from **420 ms to about 2.1 seconds** when reranking was enabled — about **5×** — while retrieval quality improved. +- Lower `reranker_candidates` when a smaller recall window still contains the right results. +- Set `reranker_max_document_chars` to a positive cap for long notes or transcript chunks. The matched chunk leads the reranker document, so the strongest retrieval signal is kept first. Measure quality on your corpus before standardizing a cap; the v0.23 default remains unbounded. + +Changing reranker settings does not change stored embeddings, so it does not require `bm reindex --embeddings`. + +### Handle reranker failures + +An enabled reranker is part of the requested ranking contract. Basic Memory deliberately does not fall back to the original retrieval order when reranking fails: + +- Temporary provider, rate-limit, connection, timeout, or first-download failures return HTTP `503` so callers can retry. +- Malformed or incomplete provider responses return HTTP `502`. +- Authentication, model, dependency, and permanent configuration failures surface directly. +- A retryable failure from any project aborts a multi-project result page instead of returning partially reranked results. + +Disable reranking if retrieval-only results are preferable to a failed request in your application. + +--- + ## Under the Hood You don't need to read this section to use search — it's here for the curious. @@ -232,6 +306,8 @@ The settings you're most likely to tune: | `semantic_search_enabled` | `true` | Enable semantic search | | `semantic_embedding_provider` | `"fastembed"` | Common values: `fastembed` (local), `openai` (API) | | `semantic_min_similarity` | `0.55` | Minimum similarity threshold | +| `semantic_vector_index` | `"pgvector"` | Postgres vector index: `pgvector` or `milvus`; SQLite always uses sqlite-vec | +| `reranker_enabled` | `false` | Opt in to cross-encoder reranking of vector and hybrid candidates | The full list — models, dimensions, batch size, vector candidates, env vars — is in the [Configuration reference](/reference/configuration#semantic-search-settings). @@ -330,9 +406,12 @@ bm reindex -p my-project - **First time** — Happens automatically on first startup - **Switching provider or model** — Embeddings from different models aren't compatible +- **Switching the Postgres vector index** — Run `bm reindex --embeddings` after changing between pgvector and Milvus; the incremental rebuild is enough - **After a database reset** — `bm reset` clears everything - **Troubleshooting** — a full rebuild (`bm reindex --full`) can fix index issues; the default run is incremental and skips unchanged content +`bm reindex --embeddings` exits nonzero if any embedding fails. Treat that as an incomplete vector index and fix the reported provider or index error before retrying. + --- ## FAQ @@ -359,6 +438,7 @@ A few things to try: - **Be more specific in your query.** "authentication" is vague; "how we handle JWT token refresh" gives the vector search more meaning to work with. - **Lower the similarity threshold.** The default `0.55` filters out loosely related results. Try `0.3` if you want broader recall. See [Configuration](#configuration). +- **Enable reranking.** If retrieval finds the right material but orders it poorly, opt in with `bm config set reranker_enabled true`, then measure the added latency on your notes. - **Check that embeddings are built.** Run `bm reindex --embeddings` to make sure the index is up to date. - **Try a different search mode.** If you're looking for an exact term, use text search. If you're exploring a concept, use vector search directly instead of hybrid. diff --git a/content/08.integrations/01.claude-desktop.md b/content/08.integrations/01.claude-desktop.md index 7460647..52516be 100644 --- a/content/08.integrations/01.claude-desktop.md +++ b/content/08.integrations/01.claude-desktop.md @@ -172,7 +172,7 @@ You can also reference specific notes with `memory://` URLs. Learn more in [Memo ### Tools appear but return errors - **Cloud**: check the [Activity view](/cloud/web-app#activity) for failed operations, or disconnect and reconnect the connector. -- **Local**: check Basic Memory is installed (`basic-memory --version`), verify sync status (`basic-memory status`), and check file permissions in your `~/basic-memory` directory. +- **Local**: check Basic Memory is installed (`basic-memory --version`), verify project index status (`basic-memory status`), and check file permissions in your `~/basic-memory` directory. ### Changes not showing up (local) - Real-time sync is on by default. Run `basic-memory status` to confirm. diff --git a/content/08.integrations/02.claude-code.md b/content/08.integrations/02.claude-code.md index f594367..2ff3c20 100644 --- a/content/08.integrations/02.claude-code.md +++ b/content/08.integrations/02.claude-code.md @@ -87,7 +87,7 @@ Install [agent skills](/integrations/skills) to teach Claude Code best practices - For cloud: re-run the `claude mcp add` command and complete the OAuth flow again **Notes not syncing?** -- Check sync status: `basic-memory status` +- Check project index status: `basic-memory status` - Look for errors in `~/.basic-memory/basic-memory.log` --- @@ -133,6 +133,15 @@ to: /integrations/skills Pre-built skills that teach Claude Code best practices for Basic Memory. :: +::card +--- +title: Harness Capture +icon: i-lucide-activity +to: /integrations/harness-capture +--- +Understand SessionStart, PreCompact, checkpoint, capture, and hook health behavior. +:: + ::card --- title: Knowledge Format diff --git a/content/08.integrations/03.codex.md b/content/08.integrations/03.codex.md index ca3f05b..04d5fac 100644 --- a/content/08.integrations/03.codex.md +++ b/content/08.integrations/03.codex.md @@ -198,7 +198,7 @@ codex mcp add basic-memory bash -c "uvx basic-memory mcp" **Cloud authentication errors** — For the CLI, make sure `BASIC_MEMORY_API_KEY` is exported in the shell Codex was launched from. For the app, remove and re-add the connector to redo the OAuth flow. -**Context not loading from memory:// URLs** — Confirm the note exists in your current project, check the URL syntax (`memory://folder/note`), and verify sync status with `basic-memory status`. +**Context not loading from memory:// URLs** — Confirm the note exists in your current project, check the URL syntax (`memory://folder/note`), and verify project index status with `basic-memory status`. --- @@ -244,6 +244,15 @@ to: /reference/mcp-tools-reference See every tool available to Codex through MCP. :: +::card +--- +title: Harness Capture +icon: i-lucide-activity +to: /integrations/harness-capture +--- +Understand SessionStart, PreCompact, post-compaction checkpoints, and hook health. +:: + ::card --- title: Knowledge Format diff --git a/content/08.integrations/09.vscode.md b/content/08.integrations/09.vscode.md index 56a0cc6..9e7addc 100644 --- a/content/08.integrations/09.vscode.md +++ b/content/08.integrations/09.vscode.md @@ -137,7 +137,7 @@ VS Code's Copilot Chat now supports MCP servers. Add Basic Memory in your VS Cod ## Troubleshooting **Sync not picking up changes?** -Check sync status with `bm status` and verify file permissions in your project directory. +Check project index status with `bm status` and verify file permissions in your project directory. **Search missing results?** VS Code's built-in search works for text matches. For knowledge graph queries and semantic search, use the CLI: `bm tool search-notes "your query"`. diff --git a/content/08.integrations/10.obsidian.md b/content/08.integrations/10.obsidian.md index 783df48..dee120e 100644 --- a/content/08.integrations/10.obsidian.md +++ b/content/08.integrations/10.obsidian.md @@ -130,7 +130,6 @@ Because Basic Memory uses standard markdown with wiki links, Obsidian understand - **Backlinks** — Every `[[wiki link]]` your AI creates is a clickable backlink in Obsidian. See all documents that reference the current note and navigate connections easily. - **Tag Explorer** — Tags from observations and frontmatter show up in Obsidian's tag pane. Filter, browse, and combine tags to find what you need. - **Live Updates** — Notes created by your AI (in Claude Desktop, Claude Code, or any MCP client) automatically appear in Obsidian since they share the same markdown files. No import or export needed. -- **Canvas** — Basic Memory's `canvas` tool generates Obsidian Canvas files. Ask your AI to visualize project structure or concept maps, then open the `.canvas` file in Obsidian to view and edit it. Use [schemas](/concepts/schema-system) to keep your notes consistent whether you're writing in Obsidian or through your AI. @@ -140,8 +139,6 @@ Use [schemas](/concepts/schema-system) to keep your notes consistent whether you **Tags not showing up**: Use `#tag` syntax (no spaces — use hyphens instead). Ensure the tag panel is enabled in Obsidian settings. -**Canvas files not opening**: Update to the latest Obsidian version and enable the Canvas core plugin. - **Notes not syncing to cloud**: Run `bm cloud push --name ` after editing. See the [Cloud Sync Guide](/cloud/cloud-sync) for details. ## Next Steps diff --git a/content/08.integrations/12.harness-capture.md b/content/08.integrations/12.harness-capture.md new file mode 100644 index 0000000..8f36da5 --- /dev/null +++ b/content/08.integrations/12.harness-capture.md @@ -0,0 +1,103 @@ +--- +title: Harness Capture +description: Understand and operate Basic Memory lifecycle hooks for Claude Code and Codex. +--- + +Basic Memory plugins for Claude Code and Codex use `bm hook` as one shared lifecycle front door. Plugin hook scripts only pass the harness event into Basic Memory; context recall, checkpoint coordination, capture, retention, and health reporting live in the installed Basic Memory package. + +## Understand the lifecycle events + +| Command | Harness event | What Basic Memory does | +|---|---|---| +| `bm hook session-start` | SessionStart | Reads configured projects, prints a bounded context brief, and captures a `session_started` envelope when capture is enabled. After Codex compaction, it can also request an agent-authored checkpoint. | +| `bm hook pre-compact` | PreCompact | Captures a `compaction_imminent` envelope. Claude can write a durable checkpoint from its transcript; Codex defers the checkpoint request to the post-compaction SessionStart because Codex ignores PreCompact output. | +| `bm hook stop` | Retired Stop hook | Returns a continue response so stale pre-upgrade Codex hook entries do not block a session. New installs do not register it. | + +Active lifecycle commands are fail-open: a capture or recall error is reported without interrupting the agent session. + +## Use plugin-managed hooks + +The Basic Memory Claude Code and Codex plugins register SessionStart and PreCompact automatically. Their status workflows also surface the shared local inbox, pending and archived counts, last flush, resolved project mapping, capture state, checkpoint state, Basic Memory version, and uv version. + +Run the core status command directly when you need the underlying evidence: + +::code-group +```bash [Codex] +bm hook status --harness codex --project-dir /path/to/repository +``` + +```bash [Claude Code] +bm hook status --harness claude --project-dir /path/to/repository +``` +:: + +The plugin's `bm-status` workflow includes this hook health in its broader Basic Memory configuration and reachability report. + +## Install hooks without a plugin + +Standalone users can add the same user-level hooks directly: + +```bash +bm hook install --harness codex +bm hook install --harness claude +``` + +Install is idempotent. It adds ownership-tagged entries to `~/.codex/hooks.json` or `~/.claude/settings.json` and preserves unrelated user hooks and settings. + +Remove only the entries Basic Memory installed: + +```bash +bm hook remove --harness codex +bm hook remove --harness claude +``` + +The plugin and standalone flows use the same runtime behavior. Do not manually register both for one harness. + +::note +For development against a nonstandard Basic Memory executable, `BM_BIN` overrides the uv-managed command used by plugin hook shims. +:: + +## Control local event capture + +Lifecycle capture is enabled by default. Each event appends one bounded JSON envelope to the owner-private local WAL at the Basic Memory data directory's `inbox/` path, normally `~/.basic-memory/inbox/`. The envelope contains lifecycle metadata such as the event, session, working directory, project hint, and model; it is not a transcript or a knowledge-graph note. + +Set `captureEvents` to the JSON boolean `false` to opt out: + +::code-group +```json [.codex/basic-memory.json] +{ + "basicMemory": { + "captureEvents": false + } +} +``` + +```json [.claude/settings.json] +{ + "basicMemory": { + "captureEvents": false + } +} +``` +:: + +Malformed capture settings fail closed: Basic Memory disables capture rather than guessing that an invalid value means `true`. + +## Flush and retain envelopes + +Archive pending envelopes locally: + +```bash +bm hook flush +bm hook flush --older-than-days 30 +``` + +Flush moves valid envelopes into `inbox/processed/`, deduplicates replays, leaves unresolved pending work visible, and prunes processed or unresolved-pending envelopes older than the retention window. It never writes graph notes or sends the lifecycle trace to a team project. + +Use `bm hook status` after flushing to confirm the pending count, archived count, and last-flush timestamp. + +## Related pages + +- [Claude Code](/integrations/claude-code) — connect Claude Code to Basic Memory. +- [OpenAI Codex](/integrations/codex) — connect the Codex app or CLI. +- [CLI Reference](/reference/cli-reference) — exact `bm hook` command reference. diff --git a/content/10.reference/1.cli-reference.md b/content/10.reference/1.cli-reference.md index 980ac7e..9408142 100644 --- a/content/10.reference/1.cli-reference.md +++ b/content/10.reference/1.cli-reference.md @@ -11,7 +11,7 @@ Use `bm` (short) or `basic-memory` (full). Examples below use `bm`. ### `bm status` -Show sync status between files and database. +Show the current project-index observation: how many project files were seen, with file details under `--verbose`. The v0.23 display replaces the old sync-report tree; use `--json` for automation. ```bash bm status @@ -103,6 +103,8 @@ bm reindex -p main When neither `--search` nor `--embeddings` is specified, both are rebuilt. +`bm reindex --embeddings` exits nonzero if any embedding fails and reports the configured vector index. Changing `semantic_vector_index` requires this command; until it succeeds, vector search returns no results and hybrid search serves full-text results only. + ### `bm reset` Reset database — drops all tables and recreates them. Files are never deleted. @@ -175,6 +177,83 @@ bm format notes/ # Format all files in a directory --- +## Configuration commands + +### `bm config list` + +List every scalar setting with its effective value and source. Environment overrides are marked with the exact `BASIC_MEMORY_*` variable. + +```bash +bm config list +bm config list --json +``` + +### `bm config get` + +Show one effective value and report any environment override: + +```bash +bm config get reranker_enabled +bm config get milvus_uri +``` + +### `bm config set` + +Validate a value through the same config model used at startup, then write it to `config.json`: + +```bash +bm config set reranker_enabled true +bm config set cli_output_style plain +``` + +Invalid keys or values exit nonzero without changing the config file. If an environment variable overrides the file value, the command writes the value but warns that the environment still wins. + +### `bm config unset` + +Remove a setting from `config.json`: + +```bash +bm config unset reranker_enabled +``` + +The effective value then falls back to an environment override when one exists, otherwise to the built-in default. + +`list`, `get`, `set`, and `unset` mask secret values and redact credentials embedded in URLs. Structured fields such as projects are managed by their dedicated command groups and do not appear as writable scalar settings. + +--- + +## Harness hook commands + +`bm hook` is the lifecycle front door used by the Basic Memory Claude Code and Codex plugins. See [Harness Capture](/integrations/harness-capture) for event behavior, settings, and retention. + +```bash +# Inspect shared inbox and harness settings health +bm hook status --harness codex --project-dir /path/to/repository +bm hook status --harness claude --project-dir /path/to/repository + +# Archive local lifecycle envelopes +bm hook flush +bm hook flush --older-than-days 30 + +# Wire or remove user-level hooks without a plugin +bm hook install --harness codex +bm hook remove --harness codex +bm hook install --harness claude +bm hook remove --harness claude +``` + +| Command | Description | +|---|---| +| `session-start` | Print the configured context brief and capture a session-start envelope when enabled | +| `pre-compact` | Capture the compaction event and coordinate a durable checkpoint | +| `stop` | Compatibility no-op for stale pre-upgrade Stop hooks; not registered by new installs | +| `status` | Show inbox depth, last flush, settings, capture/checkpoint state, and tool versions | +| `flush` | Archive pending envelopes locally; never writes graph notes | +| `install` | Add ownership-tagged user-level harness hooks idempotently | +| `remove` | Remove only the entries written by `bm hook install` | + +--- + ## Project management ### `bm project list` @@ -358,6 +437,10 @@ bm cloud pull --name research --on-conflict keep-cloud # One-way mirror: local → cloud (Personal workspaces only) bm cloud sync --name research +# Preview and delete cloud files matching ~/.basic-memory/.bmignore (Personal only) +bm cloud prune --name research --dry-run +bm cloud prune --name research + # Verify file integrity between local and cloud (Personal only) bm cloud check --name research @@ -365,7 +448,23 @@ bm cloud check --name research bm cloud sync-setup research ~/Documents/research ``` -`push`/`pull` are additive (never delete on the destination) and abort on conflicts by default, git-style. `sync` is a destructive mirror and is blocked on Team workspaces. See the [Cloud Sync guide](/cloud/cloud-sync) for details. +`push`/`pull` are additive (never delete on the destination) and abort on conflicts by default, git-style. `sync` is a destructive mirror and is blocked on Team workspaces. In v0.23 it also deletes previously synced files that become ignored by `.bmignore`; preview with `--dry-run`. `prune` performs only that ignore-based remote cleanup and asks for confirmation unless you pass `--yes`. See the [Cloud Sync guide](/cloud/cloud-sync) for details. + +### Public shares + +```bash +bm cloud share create research notes/my-idea +bm cloud share create research notes/my-idea --expires-at 2099-12-31 +bm cloud share list +bm cloud share list --project research +bm cloud share update SHARE_TOKEN --disable +bm cloud share update SHARE_TOKEN --enable +bm cloud share update SHARE_TOKEN --expires-at none +bm cloud share revoke SHARE_TOKEN +bm cloud share revoke SHARE_TOKEN --force +``` + +All share commands accept `--workspace` to route by workspace slug, display name, or tenant ID. `revoke` confirms before deletion unless `--force` is set. ### Snapshots @@ -495,6 +594,15 @@ bm schema diff person --project research Command group: `bm tool` (singular). This provides CLI access to the same MCP tools that AI assistants use, useful for scripting, debugging, and quick operations from the terminal. +Search, read, context, and activity commands use Rich tables, panels, trees, or Markdown when stdout is a TTY. Output precedence is: + +1. `--json` for raw machine-readable JSON. +2. `--plain` for undecorated human-readable text. +3. Non-TTY output defaults to JSON, preserving piped and redirected scripts. +4. TTY output follows `cli_output_style` (`rich` by default, or `plain`). + +`--json` and `--plain` are mutually exclusive. Set the interactive default with `bm config set cli_output_style plain`. + ### `bm tool write-note` ```bash @@ -543,6 +651,7 @@ bm tool read-note my-note --include-frontmatter bm tool edit-note my-note --operation append --content "new content" bm tool edit-note my-note --operation find_replace --find-text "old" --content "new" bm tool edit-note my-note --operation replace_section --section "## Notes" --content "updated" +bm tool edit-note my-note --operation replace_section --section "## Notes" --content "updated" --no-replace-subsections ``` | Flag | Description | @@ -552,6 +661,7 @@ bm tool edit-note my-note --operation replace_section --section "## Notes" --con | `--find-text` | Text to find (required for `find_replace`) | | `--section` | Section heading (required for `replace_section`) | | `--expected-replacements` | Expected replacement count for `find_replace` (default `1`) | +| `--replace-subsections`, `--no-replace-subsections` | For `replace_section`, replace nested subsections by default or preserve them with `--no-replace-subsections` | | `--project` | Target project | | `--local` | Force local routing | | `--cloud` | Force cloud routing | diff --git a/content/10.reference/2.mcp-tools-reference.md b/content/10.reference/2.mcp-tools-reference.md index 359dfdb..1c56ad9 100644 --- a/content/10.reference/2.mcp-tools-reference.md +++ b/content/10.reference/2.mcp-tools-reference.md @@ -71,8 +71,14 @@ Edit an existing note incrementally. | `section` | string | Conditional | Required for `replace_section`, `insert_before_section`, `insert_after_section` | | `find_text` | string | Conditional | Required for `find_replace` | | `expected_replacements` | integer | No | Default `1` | +| `replace_subsections` | boolean | No | For `replace_section`; default `true` replaces nested subsections. Set `false` to preserve them | +| `metadata` | object | No | Merge frontmatter fields independently of the body operation; provided keys overwrite or add values | | `workspace` | string | No | Cloud workspace containing the note (name or tenant ID) | +`replace_section` now follows Markdown heading levels. By default, a selected section extends through the next heading of the same or higher level, so replacing an `##` section also replaces its nested `###` subsections. Set `replace_subsections=false` to stop at the next heading of any level and keep nested subsections. + +`metadata` preserves unrelated frontmatter and the note body. It ignores `title`, `type`, and `permalink`, which have dedicated handling, and does not support deleting keys. + ### `move_note` Move a note or directory. @@ -165,6 +171,8 @@ Main search tool with text, vector, and hybrid modes plus structured filters. The `search_type` parameter controls the search strategy. `hybrid` is the default — it combines keyword and semantic search. `text` is keyword-only. `vector` and `semantic` are equivalent — pure meaning-based similarity. See [Semantic Search](/concepts/semantic-search) for details on each mode. +Search results expose each note's stable `external_id` in JSON and Markdown output. Use it for web-app deep links or exact follow-up operations instead of parsing a title or permalink. + Use `categories` for observation categories such as `[decision]`, `[rule]`, or `[follow-up]`. `metadata_filters` only checks note frontmatter, so `metadata_filters={"category": "decision"}` matches a frontmatter field named `category`, not observation categories. The singular `category` is accepted as an alias, and comma-separated strings work (`categories="decision,rule"`). Matching is exact, and categories only exist on observations — explicitly passing `entity_types=["entity"]` alongside `categories` returns nothing. ::note @@ -199,6 +207,8 @@ Build context graph from a memory URL. Traverses the knowledge graph from a star Recent activity in one project or cross-project discovery mode. When no project can be resolved at all (no parameter, no default project), it returns activity across all projects (discovery mode); otherwise it uses the resolved project. +Entity rows include `external_id` so clients can address the exact note returned. + | Parameter | Type | Required | Notes | |---|---|---|---| | `type` | string or list[string] | No | Filter by result type: `"entity"`, `"relation"`, `"observation"` | @@ -251,17 +261,11 @@ List directory contents with optional depth and glob filter. | `dir_name` | string | No | Directory path to list (root if omitted) | | `depth` | integer | No | How many levels deep to list | | `file_name_glob` | string | No | Glob pattern to filter files (e.g., `*.md`, `schemas/*`) | +| `page` | integer | No | One-indexed page; default `1` | +| `page_size` | integer | No | Nodes per page; default `10`, maximum `200`. Aliases: `limit`, `per_page` | +| `output_format` | `"text"` or `"json"` | No | JSON includes structured pagination metadata | -### `canvas` - -Create Obsidian canvas files for visual knowledge graph exploration. Local-only — see [Canvas Visualizations](/local/canvas). - -| Parameter | Type | Required | Notes | -|---|---|---|---| -| `nodes` | list | Yes | List of node definitions for the canvas | -| `edges` | list | Yes | List of edge definitions connecting nodes | -| `title` | string | Yes | Canvas file title | -| `directory` | string | Yes | Relative folder path for the canvas file | +Large listings are bounded. Continue with the next `page` instead of assuming one response contains every file. File nodes include `external_id` in both text and JSON output. ### `list_workspaces` @@ -304,20 +308,31 @@ Compare a schema definition against actual note usage to detect drift. --- -## Discovery tools +## Diagnostics + +### `basic_memory_diagnostics` -- **`cloud_info()`** — Returns an overview of Basic Memory Cloud: features and how to get set up (`bm cloud login`). No parameters. -- **`release_notes()`** — Returns bundled release-notes highlights for the installed version. No parameters. +Return a read-only Markdown report for support and installation troubleshooting. It takes no parameters and includes: + +- Basic Memory package and API versions; +- Python, operating-system, and architecture details; and +- the config path and current config contents with secrets and URL credentials redacted. + +::note +v0.23 removes the `canvas`, `cloud_info`, and `release_notes` MCP tools. Use the Cloud documentation, GitHub release notes, and your editor's native visualization features instead. See [Upgrade to v0.23](/whats-new/v0-23-upgrade). +:: --- ## ChatGPT compatibility tools -These are compatibility wrappers for ChatGPT's MCP implementation, which uses a simplified two-tool interface by default. +These are compatibility wrappers for ChatGPT's MCP implementation, which uses a simplified two-tool interface by default. v0.23 gates them by MCP client identity: only OpenAI clients can call them. - **`search(query)`** — Search across the knowledge base. Equivalent to `search_notes` with default parameters. - **`fetch(id)`** — Retrieve full document content by permalink. Equivalent to `read_note`. +Non-OpenAI MCP clients receive a clear rejection and must call `search_notes` and `read_note` directly. + See the [ChatGPT integration guide](/integrations/chatgpt) for usage details and limitations. --- diff --git a/content/10.reference/5.troubleshooting.md b/content/10.reference/5.troubleshooting.md index f93b4d6..abe5356 100644 --- a/content/10.reference/5.troubleshooting.md +++ b/content/10.reference/5.troubleshooting.md @@ -35,7 +35,7 @@ This guide covers common issues and their solutions. For live help, visit our [D 3. **Restart applications** - Close and reopen both Terminal and Claude Desktop -4. **Check sync status**: +4. **Check index status**: ```bash basic-memory status ``` @@ -107,15 +107,15 @@ bm cloud login --- -## Sync Issues +## Indexing Issues -### Changes Not Syncing +### Changes Not Indexing **Problem:** File changes not appearing in knowledge base. **Solutions:** -1. **Check sync status**: +1. **Check index status**: ```bash basic-memory status ``` @@ -209,19 +209,16 @@ The [Cloud Sync Guide's troubleshooting section](/cloud/cloud-sync#troubleshooti 2. **Archive old content** - Move inactive notes to archive folder -3. **Adjust sync delay** - Add to `~/.basic-memory/config.json`: - ```json - { - "sync_delay": 2000 - } +3. **Coalesce rapid file saves** - Increase the index delay when an editor writes the same file repeatedly in a short burst: + ```bash + bm config set index_delay 2000 ``` -4. **Increase thread pool** for large knowledge bases: - ```json - { - "sync_thread_pool_size": 8 - } +4. **Tune concurrent accepted writes carefully** - For workloads that submit many local writes at once, increase materialization workers from the default of `4`: + ```bash + bm config set materialization_workers 8 ``` + This affects local write materialization, not general indexing concurrency. ### High Memory Usage diff --git a/content/10.reference/6.configuration.md b/content/10.reference/6.configuration.md index 682b2df..c739f37 100644 --- a/content/10.reference/6.configuration.md +++ b/content/10.reference/6.configuration.md @@ -13,6 +13,62 @@ Environment variables override config file values. --- +## Manage settings from the CLI + +Use `bm config` for scalar settings instead of editing JSON by hand: + +```bash +bm config list +bm config get reranker_enabled +bm config set reranker_enabled true +bm config unset reranker_enabled +``` + +`set` validates the complete configuration before writing. `list` shows whether each effective value comes from a default, `config.json`, or an environment override. Secret fields are masked and credentials embedded in URLs are redacted from `list` and `get` output. + +Structured project mappings still use `bm project`; `bm config` intentionally handles only scalar settings. + +--- + +## New v0.23 settings + +All 28 new settings can be written with `bm config set ` or supplied as `BASIC_MEMORY_` environment variables. + +| Key | Default | Purpose | +|---|---|---| +| `reranker_enabled` | `false` | Enable cross-encoder reranking for vector and hybrid candidates. | +| `reranker_provider` | `fastembed` | Select local FastEmbed or API-backed LiteLLM reranking. | +| `reranker_model` | `jinaai/jina-reranker-v1-tiny-en` | Select the reranker model; LiteLLM uses `provider/model`. | +| `reranker_candidates` | `20` | Set the leading retrieval window rescored before returning a page. | +| `reranker_max_document_chars` | `0` | Cap characters per candidate; `0` sends full matched text. | +| `reranker_timeout` | `30.0` | Limit each LiteLLM rerank request in seconds; FastEmbed ignores it. | +| `reranker_api_base` | unset | Set a custom LiteLLM reranker endpoint. | +| `reranker_api_key` | unset | Pass a reranker credential directly to LiteLLM. | +| `semantic_vector_index` | `pgvector` | Select pgvector or Milvus for Postgres; SQLite remains sqlite-vec. | +| `milvus_uri` | unset | Connect to Milvus, Milvus Lite, or Zilliz Cloud. | +| `milvus_token` | unset | Authenticate to Milvus or Zilliz. | +| `milvus_timeout_seconds` | `30.0` | Limit each Milvus client operation. | +| `milvus_collection_prefix` | `basic_memory` | Prefix project-isolated Milvus collections. | +| `milvus_database` | `default` | Select a Milvus database; ignored by Milvus Lite. | +| `semantic_embedding_api_base` | unset | Set a custom LiteLLM embedding endpoint. | +| `semantic_embedding_api_key` | unset | Pass an embedding credential directly to LiteLLM. | +| `semantic_embedding_document_prefix` | unset | Prefix indexed chunks for asymmetric embedding models. | +| `semantic_embedding_query_prefix` | unset | Prefix queries for asymmetric embedding models. | +| `redis_url` | unset | Enable optional Redis caching for standalone MCP reads. | +| `redis_max_connections` | `20` | Bound the standalone read-cache connection pool. | +| `materialization_workers` | `4` | Bound local accepted-write materialization concurrency. | +| `sqlite_synchronous` | `NORMAL` | Set SQLite `PRAGMA synchronous`. | +| `sqlite_mmap_size` | `268435456` | Set SQLite `PRAGMA mmap_size` in bytes. | +| `sqlite_wal_autocheckpoint` | `1000` | Set SQLite WAL checkpoint frequency in pages. | +| `sqlite_page_size` | `4096` | Set SQLite page size for a new database or the next `VACUUM`. | +| `index_delay` | `1000` | Wait this many milliseconds after file changes before indexing. | +| `index_changes` | `true` | Index local file changes in real time. | +| `cli_output_style` | `rich` | Select Rich or plain interactive `bm tool` output. | + +Detailed constraints and operating guidance follow below. + +--- + ## Config file location Default location: @@ -112,6 +168,28 @@ Enable vector/hybrid retrieval. Semantic search is enabled and included by default in all standard installs. Embeddings are generated automatically on first startup. If sqlite-vec fails to load, search gracefully falls back to text-only mode. +### `semantic_vector_index` + +Vector index used by Postgres deployments. + +- Type: `"pgvector" | "milvus"` +- Default: `"pgvector"` +- Env: `BASIC_MEMORY_SEMANTIC_VECTOR_INDEX` + +SQLite always uses sqlite-vec and ignores this selector. After changing the Postgres index, run `bm reindex --embeddings`; before it succeeds, vector search returns nothing and hybrid search serves full-text results only. + +### Milvus settings + +| Key | Type | Default | Environment variable | +|---|---|---|---| +| `milvus_uri` | string or null | `null` | `BASIC_MEMORY_MILVUS_URI` | +| `milvus_token` | string or null | `null` | `BASIC_MEMORY_MILVUS_TOKEN` | +| `milvus_timeout_seconds` | float greater than 0 | `30.0` | `BASIC_MEMORY_MILVUS_TIMEOUT_SECONDS` | +| `milvus_collection_prefix` | string | `"basic_memory"` | `BASIC_MEMORY_MILVUS_COLLECTION_PREFIX` | +| `milvus_database` | string | `"default"` | `BASIC_MEMORY_MILVUS_DATABASE` | + +Install `basic-memory[milvus]` before selecting Milvus. `milvus_uri` accepts a remote Milvus/Zilliz endpoint or a Milvus Lite path. Lite requires a `.db` suffix and an existing parent directory, creates a directory tree at that path, and ignores `milvus_database`. See [Postgres and Milvus](/local/postgres-and-milvus). + ### `semantic_embedding_provider` Embedding provider. @@ -145,6 +223,17 @@ Embedding batch size. - Default: `2` - Env: `BASIC_MEMORY_SEMANTIC_EMBEDDING_BATCH_SIZE` +### LiteLLM endpoint and asymmetric-model settings + +| Key | Type | Default | Environment variable | Purpose | +|---|---|---|---|---| +| `semantic_embedding_api_base` | string or null | `null` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_API_BASE` | Custom OpenAI-compatible or self-hosted LiteLLM endpoint. | +| `semantic_embedding_api_key` | string or null | `null` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_API_KEY` | Credential passed directly to LiteLLM; normal provider env vars work when unset. | +| `semantic_embedding_document_prefix` | string or null | `null` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_DOCUMENT_PREFIX` | Literal prefix added to indexed chunks. | +| `semantic_embedding_query_prefix` | string or null | `null` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_QUERY_PREFIX` | Literal prefix added to queries. | + +Use the prefix pair for models whose document and query roles are expressed in input text. Rebuild embeddings after changing either prefix. + ### `semantic_vector_k` Vector candidate count. @@ -169,6 +258,31 @@ Semantic search dependencies are included by default in all standard installs. --- +## Reranker settings + +Reranking is disabled by default and applies only to vector and hybrid search. Enable the local provider with: + +```bash +bm config set reranker_enabled true +``` + +| Key | Type | Default | Environment variable | +|---|---|---|---| +| `reranker_enabled` | boolean | `false` | `BASIC_MEMORY_RERANKER_ENABLED` | +| `reranker_provider` | string (`fastembed` or `litellm`) | `"fastembed"` | `BASIC_MEMORY_RERANKER_PROVIDER` | +| `reranker_model` | string | `"jinaai/jina-reranker-v1-tiny-en"` | `BASIC_MEMORY_RERANKER_MODEL` | +| `reranker_candidates` | integer greater than 0 | `20` | `BASIC_MEMORY_RERANKER_CANDIDATES` | +| `reranker_max_document_chars` | integer at least 0 | `0` | `BASIC_MEMORY_RERANKER_MAX_DOCUMENT_CHARS` | +| `reranker_timeout` | float greater than 0 | `30.0` | `BASIC_MEMORY_RERANKER_TIMEOUT` | +| `reranker_api_base` | string or null | `null` | `BASIC_MEMORY_RERANKER_API_BASE` | +| `reranker_api_key` | string or null | `null` | `BASIC_MEMORY_RERANKER_API_KEY` | + +FastEmbed downloads the configured local model on first use. LiteLLM requires an explicit `provider/model` name and uses normal provider credentials when `reranker_api_key` is unset. Increase the candidate window to widen recall at a latency cost; use a positive document-character cap to bound long-note cost. FastEmbed ignores `reranker_timeout` because it runs locally. + +See [Semantic Search](/concepts/semantic-search#enable-reranking) for provider setup, measured latency, tuning, and fail-fast `503`/`502` behavior. + +--- + ## Permalink and file behavior ### `permalinks_include_project` @@ -221,39 +335,37 @@ Generate kebab-case filenames for tool-created notes. --- -## Sync and watch settings +## Indexing and watch settings -### `sync_changes` +### `index_changes` -Enable filesystem-to-index synchronization. +Index local file changes in real time. - Type: boolean - Default: `true` -- Env: `BASIC_MEMORY_SYNC_CHANGES` +- Env: `BASIC_MEMORY_INDEX_CHANGES` + +The old `sync_changes` config key and `BASIC_MEMORY_SYNC_CHANGES` environment variable are migrated automatically. If both old and new names are present, the new name wins. -### `sync_delay` +### `index_delay` -Debounce delay for sync in milliseconds. +Wait this many milliseconds after a file change before indexing it. - Type: integer - Default: `1000` -- Env: `BASIC_MEMORY_SYNC_DELAY` +- Env: `BASIC_MEMORY_INDEX_DELAY` -### `sync_thread_pool_size` +The old `sync_delay` config key and `BASIC_MEMORY_SYNC_DELAY` environment variable are migrated automatically. If both old and new names are present, the new name wins. -Thread pool size for file operations. - -- Type: integer -- Default: `4` -- Env: `BASIC_MEMORY_SYNC_THREAD_POOL_SIZE` +### `materialization_workers` -### `sync_max_concurrent_files` +Bound the number of accepted writes that materialize to local storage concurrently. -Maximum number of files to process concurrently during sync operations. +- Type: integer greater than `0` +- Default: `4` +- Env: `BASIC_MEMORY_MATERIALIZATION_WORKERS` -- Type: integer -- Default: `10` -- Env: `BASIC_MEMORY_SYNC_MAX_CONCURRENT_FILES` +This controls local write materialization, not general indexing concurrency. The removed `sync_thread_pool_size` and `sync_max_concurrent_files` settings are silently ignored; use this setting only when tuning concurrent accepted writes. ### `skip_initialization_sync` @@ -353,6 +465,49 @@ Connection URL for the database when using the Postgres backend. Only required when `database_backend` is `postgres`. Example: `postgresql+asyncpg://user:pass@host:5432/dbname` +### SQLite tuning + +These settings map directly to SQLite pragmas. The defaults are suitable for most installations. + +| Key | Type | Default | Environment variable | Guidance | +|---|---|---|---|---| +| `sqlite_synchronous` | `OFF`, `NORMAL`, `FULL`, or `EXTRA` | `NORMAL` | `BASIC_MEMORY_SQLITE_SYNCHRONOUS` | `NORMAL` is the safe WAL-mode default. Testing found no meaningful gain from `OFF`, which weakens durability. | +| `sqlite_mmap_size` | integer at least `0` | `268435456` | `BASIC_MEMORY_SQLITE_MMAP_SIZE` | Memory-map up to this many bytes; `0` disables memory mapping. | +| `sqlite_wal_autocheckpoint` | integer at least `0` | `1000` | `BASIC_MEMORY_SQLITE_WAL_AUTOCHECKPOINT` | Checkpoint after this many WAL pages; `0` disables automatic checkpoints. Larger values can reduce checkpoint stalls but allow a larger WAL. | +| `sqlite_page_size` | integer from `512` to `65536` | `4096` | `BASIC_MEMORY_SQLITE_PAGE_SIZE` | Applies to a fresh database or after `VACUUM`; changing it does not rewrite an existing database immediately. | + +--- + +## Optional Redis read cache + +Standalone MCP servers can cache read results in Redis. Install the optional dependency and set a URL: + +```bash +pip install "basic-memory[redis]" +bm config set redis_url redis://localhost:6379/0 +``` + +| Key | Type | Default | Environment variable | +|---|---|---|---| +| `redis_url` | string or null | `null` | `BASIC_MEMORY_REDIS_URL` | +| `redis_max_connections` | integer greater than `0` | `20` | `BASIC_MEMORY_REDIS_MAX_CONNECTIONS` | + +The cache is optional and limited to standalone MCP read paths. An unset URL disables it. If Redis is unavailable, Basic Memory fails open to authoritative storage reads rather than failing the request. + +--- + +## CLI output settings + +### `cli_output_style` + +Choose the default presentation for interactive `bm tool` commands. + +- Type: `"rich" | "plain"` +- Default: `"rich"` +- Env: `BASIC_MEMORY_CLI_OUTPUT_STYLE` + +This setting affects interactive `search`, `read-note`, `build-context`, and `recent-activity` output. Redirected output defaults to JSON. Explicit `--json` and `--plain` flags take precedence over TTY detection and this setting. + --- ## Cloud settings @@ -408,8 +563,10 @@ When loading older config files, Basic Memory automatically migrates legacy keys - `project_modes` - `cloud_projects` - `default_project_mode` +- `sync_changes` to `index_changes` +- `sync_delay` to `index_delay` -After migration, config is re-saved in current format. +The corresponding `BASIC_MEMORY_SYNC_CHANGES` and `BASIC_MEMORY_SYNC_DELAY` environment variables are also accepted as migration aliases, but the new environment names win when both are set. Removed `sync_thread_pool_size` and `sync_max_concurrent_files` keys are ignored. After file-based migration, config is re-saved in current format. --- @@ -418,3 +575,5 @@ After migration, config is re-saved in current format. - [CLI Reference](/reference/cli-reference) - [MCP Tools Reference](/reference/mcp-tools-reference) - [Semantic Search](/concepts/semantic-search) +- [Postgres and Milvus](/local/postgres-and-milvus) +- [Upgrade to v0.23](/whats-new/v0-23-upgrade) diff --git a/content/10.reference/7.docker.md b/content/10.reference/7.docker.md index 0a8938b..eb15389 100644 --- a/content/10.reference/7.docker.md +++ b/content/10.reference/7.docker.md @@ -64,9 +64,9 @@ services: - ./knowledge:/app/data:rw environment: - BASIC_MEMORY_DEFAULT_PROJECT=main - - BASIC_MEMORY_SYNC_CHANGES=true + - BASIC_MEMORY_INDEX_CHANGES=true - BASIC_MEMORY_LOG_LEVEL=INFO - - BASIC_MEMORY_SYNC_DELAY=1000 + - BASIC_MEMORY_INDEX_DELAY=1000 command: ["basic-memory", "mcp", "--transport", "sse", "--host", "0.0.0.0", "--port", "8000"] restart: unless-stopped healthcheck: @@ -101,9 +101,9 @@ Configuration comes from these environment variables — the image itself sets ` | `BASIC_MEMORY_HOME` | `/app/data/basic-memory` | Basic Memory home inside the container | | `BASIC_MEMORY_PROJECT_ROOT` | `/app/data` | Root directory for mounted projects | | `BASIC_MEMORY_DEFAULT_PROJECT` | `main` in the Compose example | Default project name | -| `BASIC_MEMORY_SYNC_CHANGES` | `true` in the Compose example | Enable file-change sync behavior | +| `BASIC_MEMORY_INDEX_CHANGES` | `true` in the Compose example | Enable real-time indexing of file changes | | `BASIC_MEMORY_LOG_LEVEL` | `INFO` in the Compose example | Log verbosity | -| `BASIC_MEMORY_SYNC_DELAY` | `1000` in the Compose example | File sync delay in milliseconds | +| `BASIC_MEMORY_INDEX_DELAY` | `1000` in the Compose example | File-change indexing delay in milliseconds | Mount two kinds of storage: diff --git a/nuxt.config.ts b/nuxt.config.ts index e11a92b..7db9032 100644 --- a/nuxt.config.ts +++ b/nuxt.config.ts @@ -13,7 +13,8 @@ export default defineNuxtConfig({ '/cloud/edit-locally-and-in-the-app': { redirect: { to: '/cloud/cloud-sync', statusCode: 301 } }, '/cloud/user-guide': { redirect: { to: '/cloud/cloud-guide', statusCode: 301 } }, '/teams/partners': { redirect: { to: '/partners/about', statusCode: 301 } }, - '/concepts/canvas': { redirect: { to: '/local/canvas', statusCode: 301 } }, + '/concepts/canvas': { redirect: { to: '/whats-new/v0-23-upgrade', statusCode: 301 } }, + '/local/canvas': { redirect: { to: '/whats-new/v0-23-upgrade', statusCode: 301 } }, '/integrations/claude-research-mode': { redirect: { to: '/integrations/claude-desktop', statusCode: 301 } }, }, app: { From f484c02111f59689d6be96301ccf97029f3c8f2f Mon Sep 17 00:00:00 2001 From: phernandez Date: Thu, 20 Aug 2026 23:14:08 -0500 Subject: [PATCH 2/5] docs: extend v0.23 docs with inspector, Team WebDAV sync, known issues - Document bm inspect chunks and bm inspect query (basic-memory #1249/#1250) in the CLI reference, add an Inspecting retrieval section and FAQ pointer to Semantic Search, and mention the inspector in the changelog. - Correct Team workspace push/pull for the WebDAV transport (#1263): Team transfers no longer use bm cloud setup or storage credentials; each request is authorized per project (read to pull, edit to push). Updated Cloud Sync, Cloud CLI, and the CLI reference. - Mention list_directory sorting (#1268) in the MCP tools reference and broaden the FTS relaxation note to non-Latin scripts (#1269). - Add Known issues in v0.23.0 to the upgrade guide: FastEmbed asymmetric model prefixes (#1264, fix tracked for v0.23.1) and macOS file-identity edge cases (#1275, #1281), plus a troubleshooting entry for duplicate notes on macOS. - Renumber What's New pages so the upgrade guide no longer shares the 2. prefix with the Teams page (routes unchanged). Signed-off-by: phernandez --- content/02.whats-new/3.changelog.md | 6 +- ...{2.v0-23-upgrade.md => 4.v0-23-upgrade.md} | 22 +++++++ ...{4.hermes-plugin.md => 5.hermes-plugin.md} | 0 .../02.whats-new/{5.cloud.md => 6.cloud.md} | 0 .../{6.agent-skills.md => 7.agent-skills.md} | 0 ...penclaw-plugin.md => 8.openclaw-plugin.md} | 0 ...friendly-docs.md => 9.ai-friendly-docs.md} | 0 content/03.cloud/05.cloud-sync.md | 19 +++++- content/03.cloud/12.cloud-cli.md | 4 +- content/07.concepts/7.semantic-search.md | 24 +++++++ content/10.reference/1.cli-reference.md | 66 ++++++++++++++++++- content/10.reference/2.mcp-tools-reference.md | 5 +- content/10.reference/5.troubleshooting.md | 12 ++++ 13 files changed, 149 insertions(+), 9 deletions(-) rename content/02.whats-new/{2.v0-23-upgrade.md => 4.v0-23-upgrade.md} (72%) rename content/02.whats-new/{4.hermes-plugin.md => 5.hermes-plugin.md} (100%) rename content/02.whats-new/{5.cloud.md => 6.cloud.md} (100%) rename content/02.whats-new/{6.agent-skills.md => 7.agent-skills.md} (100%) rename content/02.whats-new/{7.openclaw-plugin.md => 8.openclaw-plugin.md} (100%) rename content/02.whats-new/{8.ai-friendly-docs.md => 9.ai-friendly-docs.md} (100%) diff --git a/content/02.whats-new/3.changelog.md b/content/02.whats-new/3.changelog.md index 5a9e33a..d8c483a 100644 --- a/content/02.whats-new/3.changelog.md +++ b/content/02.whats-new/3.changelog.md @@ -11,9 +11,11 @@ Notable changes in each Basic Memory release, most recent first. For complete pa *[v0.23.0](https://github.com/basicmachines-co/basic-memory/releases/tag/v0.23.0)* -Semantic search takes a major step forward in v0.23. Opt-in cross-encoder reranking can rescore vector and hybrid candidates locally with FastEmbed or through LiteLLM, while Postgres deployments can move vector storage from pgvector to Milvus, Milvus Lite, or Zilliz Cloud. File-watcher writes now receive embeddings, non-BGE FastEmbed models are normalized correctly, SQLite full-text search covers complete long notes and relaxed CJK queries, and reindex failures are reported honestly. +Semantic search takes a major step forward in v0.23. Opt-in cross-encoder reranking can rescore vector and hybrid candidates locally with FastEmbed or through LiteLLM, while Postgres deployments can move vector storage from pgvector to Milvus, Milvus Lite, or Zilliz Cloud. File-watcher writes now receive embeddings, non-BGE FastEmbed models are normalized correctly, SQLite full-text search covers complete long notes and relaxes non-Latin queries (CJK, Cyrillic, and other scripts), and reindex failures are reported honestly. -The operator surface expands with **`bm config`** for validated settings, **`bm hook`** for harness lifecycle capture, Rich interactive output for **`bm tool`**, a diagnostics MCP tool, a usable local Postgres path, Cloud share management, and targeted **`bm cloud prune`** cleanup. +The operator surface expands with **`bm inspect`** — a [retrieval inspector](/reference/cli-reference#retrieval-inspection-commands) that shows how the index sees a note (`bm inspect chunks`) and traces a query's retrieval stages (`bm inspect query`) — plus **`bm config`** for validated settings, **`bm hook`** for harness lifecycle capture, Rich interactive output for **`bm tool`**, a diagnostics MCP tool, a usable local Postgres path, Cloud share management, and targeted **`bm cloud prune`** cleanup. + +Team workspaces gain working **`bm cloud push`**/**`pull`**: transfers now run over the cloud's permissioned WebDAV API with per-project access checks, so every member can sync — previously these commands required owner-only storage credentials on Team workspaces. `list_directory` learns title and updated-time sorting for one-read folder navigation. Underneath those features, the indexing and persistence core moves to generation-versioned relations and observations, compare-and-swap materialization, consistent lock ordering, and batched resolution. Observations and relations now land with accepted writes, eliminating the deadlock and stale-projection failures that appeared under concurrent agent workloads. diff --git a/content/02.whats-new/2.v0-23-upgrade.md b/content/02.whats-new/4.v0-23-upgrade.md similarity index 72% rename from content/02.whats-new/2.v0-23-upgrade.md rename to content/02.whats-new/4.v0-23-upgrade.md index dd7c4d9..e2f847e 100644 --- a/content/02.whats-new/2.v0-23-upgrade.md +++ b/content/02.whats-new/4.v0-23-upgrade.md @@ -73,6 +73,28 @@ Five revisions run automatically on first start: The final repair is not reversible through a downgrade. Markdown files remain the source of truth and the index is rebuildable, but back up `memory.db` before the upgrade if you want a pre-migration database snapshot. +## Known issues in v0.23.0 + +### Asymmetric FastEmbed models need manual prefixes + +The FastEmbed provider does not yet apply query/passage prefixes automatically, so asymmetric embedding models — notably the `intfloat/multilingual-e5` family, the strongest local option for multilingual notes — run off-label and retrieval quality collapses ([#1264](https://github.com/basicmachines-co/basic-memory/issues/1264); fix tracked for v0.23.1). Until then, set the literal prefixes yourself: + +```bash +bm config set semantic_embedding_query_prefix "query: " +bm config set semantic_embedding_document_prefix "passage: " +``` + +The trailing spaces are meaningful and preserved. Both prefixes are part of the embedding provider identity, so setting or changing them re-embeds your notes — run `bm reindex --embeddings` afterwards. The symmetric default model (`bge-small-en-v1.5`) needs no prefixes. + +### macOS file-identity edge cases + +Two open issues can produce duplicate entities for a single file on macOS: + +- **Unicode normalization (NFC/NFD):** APFS preserves whichever byte form a filename was written with, and some cross-platform sync tools (notably Syncthing) rewrite names to the other form. File-path lookups are byte-wise, so the same file can be indexed twice, accumulating `-1`-suffixed permalinks and breaking `edit_note` on the affected notes ([#1275](https://github.com/basicmachines-co/basic-memory/issues/1275)). +- **Case-only renames:** on a case-insensitive APFS volume, renaming `config.md` to `Config.md` via `move_note` is refused as "destination already exists"; case-only folder renames can silently not change the folder's case on disk and leave duplicate index entries behind ([#1281](https://github.com/basicmachines-co/basic-memory/issues/1281)). + +Workarounds: avoid case-only and normalization-only renames from MCP tools (rename through an intermediate name instead), and run `bm reindex --full -p ` to collapse duplicates if they appear. + ## Continue setup - [Semantic Search](/concepts/semantic-search) — enable and tune reranking. diff --git a/content/02.whats-new/4.hermes-plugin.md b/content/02.whats-new/5.hermes-plugin.md similarity index 100% rename from content/02.whats-new/4.hermes-plugin.md rename to content/02.whats-new/5.hermes-plugin.md diff --git a/content/02.whats-new/5.cloud.md b/content/02.whats-new/6.cloud.md similarity index 100% rename from content/02.whats-new/5.cloud.md rename to content/02.whats-new/6.cloud.md diff --git a/content/02.whats-new/6.agent-skills.md b/content/02.whats-new/7.agent-skills.md similarity index 100% rename from content/02.whats-new/6.agent-skills.md rename to content/02.whats-new/7.agent-skills.md diff --git a/content/02.whats-new/7.openclaw-plugin.md b/content/02.whats-new/8.openclaw-plugin.md similarity index 100% rename from content/02.whats-new/7.openclaw-plugin.md rename to content/02.whats-new/8.openclaw-plugin.md diff --git a/content/02.whats-new/8.ai-friendly-docs.md b/content/02.whats-new/9.ai-friendly-docs.md similarity index 100% rename from content/02.whats-new/8.ai-friendly-docs.md rename to content/02.whats-new/9.ai-friendly-docs.md diff --git a/content/03.cloud/05.cloud-sync.md b/content/03.cloud/05.cloud-sync.md index 49ae46a..163b74b 100644 --- a/content/03.cloud/05.cloud-sync.md +++ b/content/03.cloud/05.cloud-sync.md @@ -128,7 +128,7 @@ bm cloud setup 4. Configures a tenant-scoped rclone remote for the workspace (`basic-memory-cloud` for your Personal workspace) ::note -**Team workspaces need their own setup.** Run `bm cloud setup --workspace ` once per team workspace before push/pull — otherwise the commands abort with "Workspace is not set up for sync." +**Team workspaces skip this step.** `bm cloud setup` provisions the rclone remote used by Personal-workspace transfers. On a Team workspace, `push` and `pull` run over the cloud's permissioned WebDAV API instead — no rclone remote and no storage credentials — and every request is authorized against your access to that specific project: read access is enough to `pull`, edit access is required to `push`. New in v0.23; previously Team push/pull required owner-only storage credentials, so most members couldn't run them at all. :: ### 3. Add Projects with Sync @@ -224,6 +224,8 @@ bm cloud push --name research Both commands are additive — they never delete files on the destination. New and changed files transfer; if any file differs on both sides, the command aborts and lists the conflicts, like a rejected `git push`. +The transport depends on the workspace, but the commands and flags are identical. Personal-workspace transfers run through the tenant-scoped rclone remote configured by `bm cloud setup`. Team-workspace transfers (v0.23+) run over the cloud's WebDAV API, where each request is checked against your access to that project — members with read access can `pull`, members with edit access can `push`, and no `bm cloud setup` is needed. + **Resolving conflicts:** re-run with `--on-conflict` to choose what survives. The value names what is kept, so it reads the same in both directions: | Value | Behavior | @@ -415,6 +417,17 @@ bm cloud sync-setup research ~/Documents/research bm cloud pull --name research ``` +### Workspace Is Not Set Up for Sync + +**Problem:** "Workspace '...' is not set up for sync." + +**Solution:** A Personal workspace's rclone remote hasn't been provisioned yet: +```bash +bm cloud setup +``` + +Team workspaces never need `bm cloud setup` — their `push`/`pull` run over the permissioned WebDAV API, so this error only appears on the Personal path. + --- ## Sign out @@ -432,7 +445,7 @@ Logout removes your stored OAuth tokens (a saved API key survives and keeps rout - **Authentication**: OAuth 2.1 with PKCE flow - **Tokens**: Stored securely in `~/.basic-memory/basic-memory-cloud.json` - **Transport**: All data encrypted in transit (HTTPS) -- **Credentials**: Scoped S3 credentials (read-write to your tenant only) +- **Credentials**: Personal transfers use scoped S3 credentials (read-write to your tenant only); Team transfers use your OAuth session over WebDAV with per-project access checks - **Isolation**: Your data isolated from other tenants - **Ignore patterns**: Sensitive files excluded via `.bmignore` @@ -496,7 +509,7 @@ bm project ls --name **Basic Memory Cloud uses project-scoped sync:** 1. **Sign in** - `bm cloud login` -2. **Install rclone** - `bm cloud setup` +2. **Install rclone** - `bm cloud setup` (Personal workspaces; Team push/pull needs no setup) 3. **Add projects with sync** - `bm project add research --cloud --local-path ~/Documents/research` 4. **Preview the first pull** - `bm cloud pull --name research --dry-run` 5. **Pull the project down** - `bm cloud pull --name research` diff --git a/content/03.cloud/12.cloud-cli.md b/content/03.cloud/12.cloud-cli.md index 49e223a..627ea31 100644 --- a/content/03.cloud/12.cloud-cli.md +++ b/content/03.cloud/12.cloud-cli.md @@ -156,7 +156,7 @@ See [Local & Cloud Routing](/cloud/routing) for the routing model, precedence, a Use Cloud Sync when you want a local Markdown folder and a Cloud project to stay aligned. The additive `push` and `pull` commands work the same on personal and team workspaces. These commands assume the Cloud project already exists — create one in the web app or with `bm project add research --cloud`: ```bash -bm cloud setup +bm cloud setup # Personal workspaces only — Team push/pull needs no setup bm cloud sync-setup research ~/Documents/research # Fetch cloud changes, then upload local ones @@ -165,6 +165,8 @@ bm cloud push --name research --dry-run bm cloud push --name research ``` +On Team workspaces (v0.23+), `push` and `pull` run over the cloud's permissioned WebDAV API — no rclone remote or storage credentials — with each request authorized against your access to the specific project. + See [Cloud Sync](/cloud/cloud-sync) for the full workflow reference, including conflict handling with `--on-conflict`. The Personal-workspace mirror command can delete cloud files, including files that become newly ignored: diff --git a/content/07.concepts/7.semantic-search.md b/content/07.concepts/7.semantic-search.md index 15e971c..ea7c21f 100644 --- a/content/07.concepts/7.semantic-search.md +++ b/content/07.concepts/7.semantic-search.md @@ -291,6 +291,8 @@ code: | Results found by both keyword and meaning match rank highest. The dominant signal (whichever source scored higher) is preserved, while the weaker signal adds a 30% bonus. Items found by only one source keep their original score. +Question-form queries rarely match every keyword, so when the strict all-words text query comes back empty, hybrid search retries it as a relaxed any-word query rather than letting the text half go silent. As of v0.23 this relaxation works for queries in any script — CJK, Cyrillic, Greek, Arabic, and other non-Latin alphabets included. + ### Deduplication Each chunk has a content hash. When notes are re-synced or reindexed, unchanged chunks skip re-indexing. Only modified content triggers new embeddings. Editing one note in a thousand-note knowledge base only re-indexes the chunks that changed. @@ -414,6 +416,27 @@ bm reindex -p my-project --- +## Inspecting retrieval + +Two read-only CLI commands, new in v0.23, show what search actually did — so you can tell an indexing problem from a ranking problem: + +```bash +# Is this note indexed, embedded, and current? +bm inspect chunks "my-note" + +# What stages did this query run, and why did results rank the way they did? +bm inspect query "how do we handle auth" --mode hybrid +bm inspect query "auth" --mode vector --show-misses +``` + +`inspect chunks` shows a note's search rows and vector chunks with a per-chunk status (`ready`, `pending`, `stale`, or `orphaned`). `orphaned` means the chunk was embedded under a different model or vector index than the one configured — the classic "I switched models and now vector search finds nothing" state, fixed by `bm reindex --embeddings`. + +`inspect query` traces one search execution — matching, filtering, fusion, reranking, and the final page — with scores and rerank movement, captured from the same call that returned the results. Add `--show-misses` to see why near-miss candidates were dropped. + +See the [CLI Reference](/reference/cli-reference#retrieval-inspection-commands) for all flags and the JSON output contract. + +--- + ## FAQ ### Is this RAG? @@ -440,6 +463,7 @@ A few things to try: - **Lower the similarity threshold.** The default `0.55` filters out loosely related results. Try `0.3` if you want broader recall. See [Configuration](#configuration). - **Enable reranking.** If retrieval finds the right material but orders it poorly, opt in with `bm config set reranker_enabled true`, then measure the added latency on your notes. - **Check that embeddings are built.** Run `bm reindex --embeddings` to make sure the index is up to date. +- **Inspect what retrieval did.** `bm inspect query "your query" --show-misses` shows every stage and why candidates were dropped; `bm inspect chunks ` shows whether a note you expected is embedded and current. See [Inspecting retrieval](#inspecting-retrieval). - **Try a different search mode.** If you're looking for an exact term, use text search. If you're exploring a concept, use vector search directly instead of hybrid. ### Can I use a different embedding model? diff --git a/content/10.reference/1.cli-reference.md b/content/10.reference/1.cli-reference.md index 9408142..1a8f024 100644 --- a/content/10.reference/1.cli-reference.md +++ b/content/10.reference/1.cli-reference.md @@ -177,6 +177,68 @@ bm format notes/ # Format all files in a directory --- +## Retrieval inspection commands + +Read-only diagnostics for search, new in v0.23. When a note doesn't come back for a query you expected it to, `bm inspect chunks` answers "is this note indexed, embedded, and current?" and `bm inspect query` answers "what did retrieval actually do with my query?". + +### `bm inspect chunks` + +Show how the retrieval index decomposes one note into search rows and vector chunks. Accepts the same identifiers as `read_note`: a title, permalink, or `memory://` URL. + +```bash +bm inspect chunks "coffee-brewing-methods" +bm inspect chunks "Coffee Brewing Methods" --project research +bm inspect chunks coffee-brewing-methods --json +``` + +The output shows the note's entity, observation, and relation search rows with the vector chunks each row produced, plus a readiness summary and a per-chunk status: + +| Status | Meaning | +|--------|---------| +| `ready` | Embedded under the configured model and vector index — visible to vector search | +| `pending` | Awaiting embedding — not yet visible to vector search | +| `stale` | The stored chunk no longer matches the note's current search rows | +| `orphaned` | Embedded under a different model or vector index than the one configured — retrieval cannot see it; rebuild with `bm reindex --embeddings` | + +A freshness state (`fresh`, `not_indexed`, `index_behind_rows`, `rows_behind_file`, or `unknown`) distinguishes a chunking problem from an indexing-lag problem, with the checksum or fingerprint evidence for any non-fresh state. + +| Flag | Description | +|------|-------------| +| `--json` | Output raw JSON — the stable machine-readable schema (automatic when piped) | +| `--plain` | Undecorated plain-text output | +| `--project` | Target a specific project | +| `--project-id` | Project external ID (UUID); takes precedence over `--project` | +| `--local` | Force local routing (ignore cloud mode) | +| `--cloud` | Force cloud routing | + +### `bm inspect query` + +Run a search and trace its execution: every retrieval stage — matching, hydration, filtering, fusion, reranking, and the final page window — that produced the result page. The trace is captured by the same call that returns the results, so it reflects the actual execution rather than a reconstruction, and ordinary searches are unaffected. + +```bash +bm inspect query "how do we handle auth" +bm inspect query "auth" --mode vector --show-misses +bm inspect query "auth" --mode hybrid --page 1 --page-size 10 --json +``` + +The output shows the engine identity (vector index, embedding model, fusion formula, minimum similarity, reranker status), a stage table with in/out/dropped counts and timings, and the ranked results with final scores and rerank movement (Δ). + +| Flag | Description | +|------|-------------| +| `--mode` | Retrieval mode: `text` (default), `vector`, or `hybrid` | +| `--show-misses` | Also render bounded rejected candidates and why each was dropped (vector and hybrid modes) | +| `--show-ids` | Include stable entity IDs in human output | +| `--page` | Result page to inspect (default `1`) | +| `--page-size` | Results per page (default `10`) | +| `--json` | Output raw JSON — every captured candidate, stable schema (automatic when piped) | +| `--plain` | Undecorated plain-text output | +| `--project` | Target a specific project | +| `--project-id` | Project external ID (UUID); takes precedence over `--project` | +| `--local` | Force local routing (ignore cloud mode) | +| `--cloud` | Force cloud routing | + +--- + ## Configuration commands ### `bm config list` @@ -418,7 +480,7 @@ bm cloud api-key create "my-laptop" ### Setup and upload ```bash -bm cloud setup +bm cloud setup # provision the rclone remote (Personal workspaces; Team push/pull needs no setup) bm cloud upload ~/my-notes --project research --create-project ``` @@ -448,7 +510,7 @@ bm cloud check --name research bm cloud sync-setup research ~/Documents/research ``` -`push`/`pull` are additive (never delete on the destination) and abort on conflicts by default, git-style. `sync` is a destructive mirror and is blocked on Team workspaces. In v0.23 it also deletes previously synced files that become ignored by `.bmignore`; preview with `--dry-run`. `prune` performs only that ignore-based remote cleanup and asks for confirmation unless you pass `--yes`. See the [Cloud Sync guide](/cloud/cloud-sync) for details. +`push`/`pull` are additive (never delete on the destination) and abort on conflicts by default, git-style. On Team workspaces they run over the cloud's permissioned WebDAV API (v0.23+) — no `bm cloud setup` required, and access is checked per project. `sync` is a destructive mirror and is blocked on Team workspaces. In v0.23 it also deletes previously synced files that become ignored by `.bmignore`; preview with `--dry-run`. `prune` performs only that ignore-based remote cleanup and asks for confirmation unless you pass `--yes`. See the [Cloud Sync guide](/cloud/cloud-sync) for details. ### Public shares diff --git a/content/10.reference/2.mcp-tools-reference.md b/content/10.reference/2.mcp-tools-reference.md index 1c56ad9..f117705 100644 --- a/content/10.reference/2.mcp-tools-reference.md +++ b/content/10.reference/2.mcp-tools-reference.md @@ -254,19 +254,22 @@ Cloud purge behavior applies when the MCP session is hosted or factory-routed to ### `list_directory` -List directory contents with optional depth and glob filter. +List directory contents with optional depth, glob filter, and sorting. | Parameter | Type | Required | Notes | |---|---|---|---| | `dir_name` | string | No | Directory path to list (root if omitted) | | `depth` | integer | No | How many levels deep to list | | `file_name_glob` | string | No | Glob pattern to filter files (e.g., `*.md`, `schemas/*`) | +| `sort` | string | No | Ordering: `title_asc`, `title_desc`, `updated_asc`, or `updated_desc`. Omit for the default filename ordering | | `page` | integer | No | One-indexed page; default `1` | | `page_size` | integer | No | Nodes per page; default `10`, maximum `200`. Aliases: `limit`, `per_page` | | `output_format` | `"text"` or `"json"` | No | JSON includes structured pagination metadata | Large listings are bounded. Continue with the next `page` instead of assuming one response contains every file. File nodes include `external_id` in both text and JSON output. +Explicit sorts (v0.23) order folders before files, apply deterministically before pagination, and use note titles for `title_*` modes. Folders have no canonical update timestamp, so `updated_*` modes keep folders name-ascending while still listing them first. + ### `list_workspaces` List available cloud workspaces. Returns workspace names and tenant IDs for the authenticated user. diff --git a/content/10.reference/5.troubleshooting.md b/content/10.reference/5.troubleshooting.md index abe5356..fcd9e58 100644 --- a/content/10.reference/5.troubleshooting.md +++ b/content/10.reference/5.troubleshooting.md @@ -134,6 +134,18 @@ bm cloud login Plain `basic-memory reset` only deletes the index — pass `--reindex` to rebuild it from your files. May take time for large knowledge bases. :: +### Duplicate Notes on macOS + +**Problem:** The same file appears twice — duplicate search results, `-1`-suffixed permalinks, or `edit_note` failing on an affected note. Typically triggered by cross-platform sync tools that rewrite Unicode filename forms ([#1275](https://github.com/basicmachines-co/basic-memory/issues/1275)) or by case-only renames on case-insensitive APFS ([#1281](https://github.com/basicmachines-co/basic-memory/issues/1281)). + +**Solution:** Collapse the duplicates with a full rebuild, and avoid case-only or normalization-only renames until the fixes land: + +```bash +basic-memory reindex --full -p +``` + +See [Known issues in v0.23.0](/whats-new/v0-23-upgrade#known-issues-in-v0230) for details. + ### Cloud Sync Issues **Problem:** Push or pull not working — setup errors, conflicts, or "no local sync path configured". From 312f522c283267e211d8c8c00597865ec54be3ee Mon Sep 17 00:00:00 2001 From: phernandez Date: Sun, 23 Aug 2026 19:15:21 -0500 Subject: [PATCH 3/5] docs: reword E5 prefix known issue after the #1302 resolution The prefix configuration is now the documented, supported setup for asymmetric FastEmbed models (#1264 closed by basic-memory#1302); automatic per-model-family defaults remain tracked in #1284. Signed-off-by: phernandez --- content/02.whats-new/4.v0-23-upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/02.whats-new/4.v0-23-upgrade.md b/content/02.whats-new/4.v0-23-upgrade.md index e2f847e..71fe5ba 100644 --- a/content/02.whats-new/4.v0-23-upgrade.md +++ b/content/02.whats-new/4.v0-23-upgrade.md @@ -77,7 +77,7 @@ The final repair is not reversible through a downgrade. Markdown files remain th ### Asymmetric FastEmbed models need manual prefixes -The FastEmbed provider does not yet apply query/passage prefixes automatically, so asymmetric embedding models — notably the `intfloat/multilingual-e5` family, the strongest local option for multilingual notes — run off-label and retrieval quality collapses ([#1264](https://github.com/basicmachines-co/basic-memory/issues/1264); fix tracked for v0.23.1). Until then, set the literal prefixes yourself: +The FastEmbed provider does not apply query/passage prefixes automatically, so asymmetric embedding models — notably the `intfloat/multilingual-e5` family, the strongest local option for multilingual notes — run off-label and retrieval quality collapses without them ([#1264](https://github.com/basicmachines-co/basic-memory/issues/1264)). The prefix configuration below is the supported setup for these models; per-model-family automatic defaults are tracked separately in [#1284](https://github.com/basicmachines-co/basic-memory/issues/1284). Set the literal prefixes yourself: ```bash bm config set semantic_embedding_query_prefix "query: " From ccbae2ed00b22719fd69a95d3b85ad99ad896efd Mon Sep 17 00:00:00 2001 From: phernandez Date: Sun, 23 Aug 2026 21:42:50 -0500 Subject: [PATCH 4/5] docs: add v0.23.0 what's new page and bump version badge Add /whats-new/v0-23 as the first What's New page, covering reranked semantic search (MRR 0.616 -> 0.719 with rerank on), pluggable vector indexes with first-party Milvus, the concurrency-safe indexing core, the bm inspect retrieval inspector, Team workspace push/pull over WebDAV, the bm config / diagnostics / bm hook operator surface, cloud share/prune/Redis additions, and the automatic migrations with a link to the upgrade guide. Renumber the whats-new pages with zero-padded prefixes so the new page sorts first and 10/11 sort after 9 (routes unchanged - Nuxt strips numeric prefixes). Point the welcome-page What's New callout at the new page instead of the changelog. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01G4rbaeHJN3L7CREp5v38J9 Signed-off-by: phernandez --- content/0.welcome.md | 2 +- content/02.whats-new/01.v0-23.md | 74 +++++++++++++++++++ ...ions.md => 02.comments-and-suggestions.md} | 0 ...e-mcp-app.md => 03.interactive-mcp-app.md} | 0 .../02.whats-new/{2.teams.md => 04.teams.md} | 0 .../{3.changelog.md => 05.changelog.md} | 0 ...4.v0-23-upgrade.md => 06.v0-23-upgrade.md} | 0 ...5.hermes-plugin.md => 07.hermes-plugin.md} | 0 .../02.whats-new/{6.cloud.md => 08.cloud.md} | 0 .../{7.agent-skills.md => 09.agent-skills.md} | 0 ...enclaw-plugin.md => 10.openclaw-plugin.md} | 0 ...riendly-docs.md => 11.ai-friendly-docs.md} | 0 12 files changed, 75 insertions(+), 1 deletion(-) create mode 100644 content/02.whats-new/01.v0-23.md rename content/02.whats-new/{0.comments-and-suggestions.md => 02.comments-and-suggestions.md} (100%) rename content/02.whats-new/{1.interactive-mcp-app.md => 03.interactive-mcp-app.md} (100%) rename content/02.whats-new/{2.teams.md => 04.teams.md} (100%) rename content/02.whats-new/{3.changelog.md => 05.changelog.md} (100%) rename content/02.whats-new/{4.v0-23-upgrade.md => 06.v0-23-upgrade.md} (100%) rename content/02.whats-new/{5.hermes-plugin.md => 07.hermes-plugin.md} (100%) rename content/02.whats-new/{6.cloud.md => 08.cloud.md} (100%) rename content/02.whats-new/{7.agent-skills.md => 09.agent-skills.md} (100%) rename content/02.whats-new/{8.openclaw-plugin.md => 10.openclaw-plugin.md} (100%) rename content/02.whats-new/{9.ai-friendly-docs.md => 11.ai-friendly-docs.md} (100%) diff --git a/content/0.welcome.md b/content/0.welcome.md index 328ecd7..2bb7275 100644 --- a/content/0.welcome.md +++ b/content/0.welcome.md @@ -11,7 +11,7 @@ seo: **Basic Memory** is real memory for your AI — a knowledge base you own. Built-in AI memory is small, vendor-owned, and usually invisible. Basic Memory is large, structured, and inspectable: a knowledge graph written in plain Markdown. Open it in any editor, hand it to any AI tool, and keep it forever. ::note{icon="i-lucide-megaphone"} -**New in v0.23:** the search release — reranking, Milvus, and a deadlock-free indexing core. See the [Changelog](/whats-new/changelog). Also new in Basic Memory Cloud: [Comments and Suggestions](/whats-new/comments-and-suggestions) and the [Interactive MCP App](/whats-new/interactive-mcp-app) for ChatGPT and Claude. +**New in v0.23:** the search release — reranking, Milvus, and a deadlock-free indexing core. See [What's New in v0.23](/whats-new/v0-23). Also new in Basic Memory Cloud: [Comments and Suggestions](/whats-new/comments-and-suggestions) and the [Interactive MCP App](/whats-new/interactive-mcp-app) for ChatGPT and Claude. :: ## Pick your path diff --git a/content/02.whats-new/01.v0-23.md b/content/02.whats-new/01.v0-23.md new file mode 100644 index 0000000..76584cf --- /dev/null +++ b/content/02.whats-new/01.v0-23.md @@ -0,0 +1,74 @@ +--- +title: Basic Memory v0.23 +description: Reranked semantic search, pluggable vector indexes with Milvus, deadlock-free concurrent indexing, and a retrieval inspector. +--- + +**Basic Memory v0.23** shipped in August 2026 ([v0.23.0 release notes](https://github.com/basicmachines-co/basic-memory/releases/tag/v0.23.0)). The theme is search you can trust: semantic search matures with cross-encoder reranking and pluggable vector indexes, the indexing core stops deadlocking under concurrent writers, and a new inspector shows exactly why a search returned what it did. + +## Search that ranks + +### Opt-in cross-encoder reranking + +Vector and hybrid search can now rescore their leading candidates with a cross-encoder that reads the query and each candidate together: + +```bash +bm config set reranker_enabled true +``` + +The default provider runs a local FastEmbed ONNX model — no API key, no per-query cost — and LiteLLM connects hosted rerankers from Cohere, Jina, and Voyage. In our benchmark evaluation, reranking improved mean reciprocal rank from 0.616 to 0.719 — a 17% improvement in how high the right note lands in the results. Reranking is off by default and downloads its local model on the first reranked search. See [Enable reranking](/concepts/semantic-search#enable-reranking). + +### Pluggable vector indexes, with first-party Milvus + +Postgres deployments can now choose where vectors live: the new `semantic_vector_index` setting selects `pgvector` (the default) or Milvus, Milvus Lite, and Zilliz Cloud via `pip install "basic-memory[milvus]"`. A vector-index identity and readiness manifest lets search tell "no ready index" apart from "no results", and SQLite keeps sqlite-vec. After switching backends, rebuild with `bm reindex --embeddings`. See [Run Postgres with Milvus](/local/postgres-and-milvus). + +### Embedding and full-text correctness + +A batch of fixes makes search results more trustworthy across the board: + +- Notes edited directly on disk are vector-embedded when the file watcher picks them up — no more waiting for a reindex +- FastEmbed embeddings are L2-normalized for non-BGE models, so semantic scores no longer silently degrade +- SQLite full-text search covers complete note content (previously text beyond ~6,000 characters was invisible), and non-Latin queries — CJK, Cyrillic, Greek, Hebrew, Arabic, and more — relax correctly instead of silently degrading +- `bm reindex --embeddings` exits nonzero on real failures and reports which index it wrote + +## Indexing that no longer deadlocks + +Underneath search, the indexing and persistence core is rebuilt around generation-versioned relations and observations with compare-and-swap publication. A stale indexing pass can never clobber or deadlock against a newer write, and observations and relations land with the accepted write instead of waiting for the next file re-index. The deadlock and lost-observation failures that appeared under concurrent multi-agent write loads are gone, and a one-time migration repairs historical duplicate observation rows left behind by the old behavior. + +## See what search sees + +The new retrieval inspector answers "why did search return that?" from the command line: + +- **`bm inspect chunks `** shows a note exactly as the index sees it — its search rows, the vector chunks each row produced, and per-chunk ready/pending/stale/orphaned status — separating chunking problems from freshness problems in one command +- **`bm inspect query ""`** captures an execution trace from the same search call that returns the results: full-text and vector candidates, fusion, filtering, reranking, and the final page window, with stable JSON output and `--show-misses` for rejected candidates + +See [Retrieval inspection commands](/reference/cli-reference#retrieval-inspection-commands). + +## Team workspace push and pull for every member + +`bm cloud push` and `bm cloud pull` now work on Team workspaces for every member. Transfers run over the cloud's permissioned WebDAV API with per-project access checks — previously these commands required owner-only storage credentials, and every other member got a 403. Flags and semantics are unchanged: transfers stay additive (nothing is ever deleted) and `--on-conflict` still defaults to `fail`. See [Push and pull](/cloud/cloud-sync#push-and-pull-additive-git-style). + +## A real operator front door + +Day-to-day operation gets first-class commands: + +- **`bm config`** — `list` effective settings with environment overrides marked, `get`, `set` (validated through the config model), and `unset`. See [Configuration commands](/reference/cli-reference#configuration-commands) +- **Rich `bm tool` output** — interactive commands render panels, tables, and trees on a TTY; piped output stays machine-readable, with `--plain` and `--json` overrides +- **`basic_memory_diagnostics`** — a new MCP tool that reports version and system info for bug reports +- **`bm status`** — redesigned around project index status +- **`bm hook`** — harness lifecycle capture moves into the package: SessionStart and PreCompact handling for Claude Code and Codex, bounded envelope capture into a local inbox, and `bm hook status`, `flush`, `install`, and `remove` for standalone setups. See [Harness Capture](/integrations/harness-capture) + +## Cloud additions + +- **`bm cloud share`** — create, list, update, and revoke public share links from the CLI. See [Manage public share links](/cloud/cloud-cli#manage-public-share-links) +- **`bm cloud prune`** — targeted cleanup that deletes cloud files matching this machine's `.bmignore`, without a full mirror sync. See [Prune newly ignored cloud files](/cloud/cloud-sync#prune-newly-ignored-cloud-files) +- **Optional Redis read cache** — standalone MCP servers can cache reads via `pip install "basic-memory[redis]"` and `redis_url`. See [Optional Redis read cache](/reference/configuration#optional-redis-read-cache) + +## Upgrading + +Five database migrations run automatically on first start, including a one-time repair that removes duplicate observation rows and purges their stale full-text search entries. Most local setups need no manual action, but v0.23 also removes MCP tools and legacy API routes and changes several CLI, config, sync, and pagination contracts. + +::note{icon="i-lucide-arrow-up-circle"} +Before upgrading an integration or automation, read [Upgrade to v0.23](/whats-new/v0-23-upgrade) for the breaking changes, required actions, and known issues. +:: + +For the complete list of changes, see the [Changelog](/whats-new/changelog) and the [v0.23.0 release on GitHub](https://github.com/basicmachines-co/basic-memory/releases/tag/v0.23.0). diff --git a/content/02.whats-new/0.comments-and-suggestions.md b/content/02.whats-new/02.comments-and-suggestions.md similarity index 100% rename from content/02.whats-new/0.comments-and-suggestions.md rename to content/02.whats-new/02.comments-and-suggestions.md diff --git a/content/02.whats-new/1.interactive-mcp-app.md b/content/02.whats-new/03.interactive-mcp-app.md similarity index 100% rename from content/02.whats-new/1.interactive-mcp-app.md rename to content/02.whats-new/03.interactive-mcp-app.md diff --git a/content/02.whats-new/2.teams.md b/content/02.whats-new/04.teams.md similarity index 100% rename from content/02.whats-new/2.teams.md rename to content/02.whats-new/04.teams.md diff --git a/content/02.whats-new/3.changelog.md b/content/02.whats-new/05.changelog.md similarity index 100% rename from content/02.whats-new/3.changelog.md rename to content/02.whats-new/05.changelog.md diff --git a/content/02.whats-new/4.v0-23-upgrade.md b/content/02.whats-new/06.v0-23-upgrade.md similarity index 100% rename from content/02.whats-new/4.v0-23-upgrade.md rename to content/02.whats-new/06.v0-23-upgrade.md diff --git a/content/02.whats-new/5.hermes-plugin.md b/content/02.whats-new/07.hermes-plugin.md similarity index 100% rename from content/02.whats-new/5.hermes-plugin.md rename to content/02.whats-new/07.hermes-plugin.md diff --git a/content/02.whats-new/6.cloud.md b/content/02.whats-new/08.cloud.md similarity index 100% rename from content/02.whats-new/6.cloud.md rename to content/02.whats-new/08.cloud.md diff --git a/content/02.whats-new/7.agent-skills.md b/content/02.whats-new/09.agent-skills.md similarity index 100% rename from content/02.whats-new/7.agent-skills.md rename to content/02.whats-new/09.agent-skills.md diff --git a/content/02.whats-new/8.openclaw-plugin.md b/content/02.whats-new/10.openclaw-plugin.md similarity index 100% rename from content/02.whats-new/8.openclaw-plugin.md rename to content/02.whats-new/10.openclaw-plugin.md diff --git a/content/02.whats-new/9.ai-friendly-docs.md b/content/02.whats-new/11.ai-friendly-docs.md similarity index 100% rename from content/02.whats-new/9.ai-friendly-docs.md rename to content/02.whats-new/11.ai-friendly-docs.md From c3eba72883c923778658046b80d89965a5e21f0b Mon Sep 17 00:00:00 2001 From: phernandez Date: Sun, 23 Aug 2026 21:43:40 -0500 Subject: [PATCH 5/5] docs: name the LoCoMo benchmark and cross-link the v0.23 pages Signed-off-by: phernandez --- CLAUDE.md | 2 +- content/02.whats-new/01.v0-23.md | 2 +- content/02.whats-new/05.changelog.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 8e6d0ce..76e1068 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -179,7 +179,7 @@ pip install basic-memory When updating docs for a new Basic Memory release, update all of the following: 1. **Welcome page What's New callout** — `content/0.welcome.md`: update the version text (e.g., `v0.22`) and the headline; the link stays `/whats-new/changelog` -2. **Changelog page** — `content/2.whats-new/1.changelog.md`: add a section for the new minor/major version at the top, linking the GitHub release tag(s). Patch release: add a short bullet list under its minor version's section instead. The `::github-releases` block at the bottom auto-fetches full release notes from the GitHub API — no manual update needed there. There are no per-version pages — deep links to version-specific behavior point at GitHub release tags +2. **Changelog page** — `content/2.whats-new/1.changelog.md`: add a section for the new minor/major version at the top, linking the GitHub release tag(s). Patch release: add a short bullet list under its minor version's section instead. The `::github-releases` block at the bottom auto-fetches full release notes from the GitHub API — no manual update needed there. Since v0.23, significant releases also get a per-version What's New page (`content/02.whats-new/01.v0-XX.md`, zero-padded prefixes so 10+ pages sort correctly); the welcome-page `::note` callout links to it 3. **Feature docs** — if the release adds user-facing features, update the relevant guide and reference pages (`content/3.cloud/`, `content/9.reference/`, etc.) 4. **Deploy** — push to main auto-deploys to development; production requires manual workflow dispatch via GitHub Actions diff --git a/content/02.whats-new/01.v0-23.md b/content/02.whats-new/01.v0-23.md index 76584cf..51d0499 100644 --- a/content/02.whats-new/01.v0-23.md +++ b/content/02.whats-new/01.v0-23.md @@ -15,7 +15,7 @@ Vector and hybrid search can now rescore their leading candidates with a cross-e bm config set reranker_enabled true ``` -The default provider runs a local FastEmbed ONNX model — no API key, no per-query cost — and LiteLLM connects hosted rerankers from Cohere, Jina, and Voyage. In our benchmark evaluation, reranking improved mean reciprocal rank from 0.616 to 0.719 — a 17% improvement in how high the right note lands in the results. Reranking is off by default and downloads its local model on the first reranked search. See [Enable reranking](/concepts/semantic-search#enable-reranking). +The default provider runs a local FastEmbed ONNX model — no API key, no per-query cost — and LiteLLM connects hosted rerankers from Cohere, Jina, and Voyage. On the LoCoMo long-conversation retrieval benchmark, reranking improved mean reciprocal rank from 0.616 to 0.719 — a 17% improvement in how high the right note lands in the results. Reranking is off by default and downloads its local model on the first reranked search. See [Enable reranking](/concepts/semantic-search#enable-reranking). ### Pluggable vector indexes, with first-party Milvus diff --git a/content/02.whats-new/05.changelog.md b/content/02.whats-new/05.changelog.md index d8c483a..e6fc1ff 100644 --- a/content/02.whats-new/05.changelog.md +++ b/content/02.whats-new/05.changelog.md @@ -19,7 +19,7 @@ Team workspaces gain working **`bm cloud push`**/**`pull`**: transfers now run o Underneath those features, the indexing and persistence core moves to generation-versioned relations and observations, compare-and-swap materialization, consistent lock ordering, and batched resolution. Observations and relations now land with accepted writes, eliminating the deadlock and stale-projection failures that appeared under concurrent agent workloads. -This release also removes MCP tools and legacy API routes and changes several CLI, config, sync, edit, and pagination contracts. Read [Upgrade to v0.23](/whats-new/v0-23-upgrade) before upgrading an integration or automation. +This release also removes MCP tools and legacy API routes and changes several CLI, config, sync, edit, and pagination contracts. Read [What's New in v0.23](/whats-new/v0-23) for the highlights and [Upgrade to v0.23](/whats-new/v0-23-upgrade) before upgrading an integration or automation. ---