Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
## 变更描述

<!-- 用一两段说明这个 PR 改了什么、为什么改。遵循 ci-cd-and-quality.md 的
Proof before trust:PR 文本不是修改成立的证据,检查清单与 CI 才是。 -->

**What**(改了什么):

**Why**(为什么改 / 解决什么问题):

**Changes**(关键改动点,按文件或模块列出):

---

## 完成检查项

<!-- 合并前逐项核对。GitHub 拥有 PR/CI 状态;本清单只是让提交者自己先过一遍。 -->

### 本地质量检查

- [ ] `npm run verify:static` 通过(build / package:check / tsc / lint / format:check / docs:check / agent:context:check / complexity:gate)
- [ ] `npm run test:product` 通过(或按改动路由跑 `npm run agent:verify -- <路径>`)
- [ ] 文档改动跑过 `npm run docs:check`;决策/设计改动遵循 doc-maintenance 规范
- [ ] 新代码方法圈复杂度不超阈值(CodeFactor / `npm run complexity:gate`)

### RCP(Repository Control Plane)

- [ ] 首个实质写入前已在 `repo-development` 黑板登记 in-flight goal(条目已 resolve)
- [ ] `npm run agent:verify -- <改动路径>` 跑过 reconcile,`.nmg/verification/latest.json` 覆盖改动路由
- [ ] 只提交本 PR 拥有的文件;未吞并行 Agent 的暂存/工作树改动

### CI 完成确认

<!-- CI 完成后无需逐个 job 轮询:读 CI Status Snapshot 即可确认。 -->
- [ ] CI Status Snapshot(`.nmg-ci/status.json`)结论为 `workflow.conclusion: "success"` 且 `failures: []`
—— 或 `gh pr checks <PR> | Select-String "All checks passed"` 出现且为 pass
- [ ] CodeFactor 通过

> CI Status Snapshot 是 GitHub 状态的只读观察(`authority: observation-only`),
> 不是授权或合并决定;合并仍需显式操作。
18 changes: 17 additions & 1 deletion docs/design/file-content-source-design.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,11 @@
# File content source for search

**Status:** proposed
**Status:** superseded
**Updated:** 2026-09-01
**Superseded by:** [memory-anchors-design.md](memory-anchors-design.md) — the
full-text file index is dropped in favor of sparse, Agent-authored anchors as an
independent searchable source. This document is kept for lineage; its
"files are not memory" and separated-presentation conclusions carry forward.

This document proposes giving `nmg search` a second content source — the
project's own files and documents — so an Agent does not have to re-discover the
Expand Down Expand Up @@ -136,6 +140,12 @@ already provide the observation seam.
- Not a document-management system: no file versions, no document lifecycle.
- Lexical-first by default; vector is an optional future enhancement, not a
requirement (per §4 finding).
- **No structured-unit parsing for now** (surveyed 2026-09-02): code-symbol
units (tree-sitter) and Markdown section anchors (mdast) are reliable in
principle, but document-side "concept units" have no deterministic parser —
prose structure lives at the semantic layer, not the syntax layer. We leave
an extractor protocol slot (crawler → per-type extractor → unit rows) and do
not implement it until a real gap or a mature tool demands it.

## 6. MVP path

Expand All @@ -155,3 +165,9 @@ already provide the observation seam.
- Scope auto-growth cap and decay (avoid unbounded scope growth).
- Whether the file index should be per-project (`.nmg/`) or share the daemon
store; per-project keeps it isolated and deletable.
- **Structured-unit evolution (deferred, protocol slot only)**: the current
index is whole-file blob. A future extractor layer could emit unit rows
(`path, kind, name, start_line, end_line`) for code (tree-sitter) and
Markdown (mdast heading sections), giving file:line anchors instead of
trigram fragments. Document "concept units" have no deterministic parser and
are deliberately not pursued; semantic understanding stays with embeddings.
182 changes: 182 additions & 0 deletions docs/design/memory-anchors-design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,182 @@
# Memory anchors: bookmarks as a searchable source

**Status:** proposed
**Updated:** 2026-09-02

This document supersedes the file-content-source design
([file-content-source-design.md](file-content-source-design.md)) as the owner of
"how an Agent reaches file content through NMG". It records a first-principles
redesign reached through an extended design discussion (surveyed 2026-09-02).

## 1. Problem

The Agent repeatedly "searches around" for things it already knows exist. NMG
memory stores *long-lived, low-churn* facts — "there is a budget mechanism". But
memory does not say *where* that mechanism lives in the current file tree, so
every session re-discovers file locations by hand (`grep` / `read` / `glob`).

The previous answer was a **file content index**: passively scan files, index
their full text, and search them as a second source
([file-content-source-design.md](file-content-source-design.md)). Experience
with the MVP showed this direction is usable but not good: whole-file blobs,
trigram fragments with no anchors, and — decisively — **a whole index to
maintain** (scope file, incremental crawler, content hashes, file FTS, scope
observer). The maintenance cost outweighs the retrieval benefit for a memory
system whose files are already reachable by path.

## 2. First-principles reframing

Three layers, each answering one question:

```text
memory (long-lived, low-churn) — "there is this thing"
anchor (bookmark, position) — "the content of that thing is here"
file (content host) — the actual bytes
```

The Agent does **not** need to remember which file, at which line, holds which
content — that is high-churn, fragile knowledge. It needs to remember *that the
thing exists* (memory) and have a cheap, objective way to reach *its content*
(anchor → file). The anchor is the bridge; it is an **external buffer layer**,
not memory content and not a file-content replica.

## 3. Decisions

### 3.1 No file full-text search

The file content source (full-text index over files, `.nmg-search-scope`,
incremental crawler, file FTS) is **dropped** as a maintained feature. Files are
not indexed, not crawled, not searched by NMG. They remain reachable through
anchors (and through the Agent's own `read`/`grep` tools, which NMG does not
replace).

Rationale: maintaining a file index is expensive and its marginal value over
anchors + direct tool access is low. The file is the content host; NMG only
needs *pointers into it*.

### 3.2 Anchors are an independent, searchable source

An **anchor** (a bookmark) is a first-class row, not a field glued onto a
memory. Anchors live in their own store and are **searched alongside memory** —
a single query returns memory hits *and* anchor hits.

```text
nmg search "budget"
├── memory source: FTS over memory_records (existing)
└── anchor source: FTS over anchors (label/path) (new)
```

An anchor row carries:

| field | meaning |
| --------- | ---------------------------------------------------------- |
| `path` | file the anchor points into (project-relative) |
| `snippet` | short content excerpt used for *relocation*, not line |
| `label` | Agent-written one-liner (searchable) |
| `kind` | e.g. `code`, `doc`, `note` (optional) |
| `memory_id` | optional back-pointer to the memory that raised it |

Anchors are searchable independently: even with no matching memory (or after a
memory is superseded), a matching anchor is still found.

### 3.3 Anchors are content-anchored, not line-anchored

An anchor stores a **content snippet**, never a line number. Line numbers drift
on every edit; content is relocatable. To resolve an anchor, locate its snippet
in the current file (exact match → position; fuzzy fallback → nearest match;
absent → anchor is stale). This is the established pattern from
[gptme hash-anchored editing](https://github.com/gptme/gptme/blob/ae707fc8233e77d4da97fc74f94db1eaff1e381a/gptme/tools/_anchored.py),
[agentic-bookmarks self-healing anchors](https://github.com/super-mega-lab/agentic-bookmarks),
and [haido `hash_at_link` drift detection](https://github.com/lebac-svg/haido/blob/HEAD/docs/DESIGN.md):
never persist a position that the file can invalidate; persist content and
relocate.

Staleness is **objective**: resolve on read; if the snippet no longer exists,
report the anchor as stale (memory stays valid — only the position is gone).

### 3.4 Markers are the index pointer between memory and anchor

NMG memory already carries an open-string metadata channel — `MemoryMarker`
(`kind` open, `attributes` key/value), used today by `board_origin` and
`retrieveHint`. The memory↔anchor link rides the same channel:

```jsonc
markers: [{
"kind": "anchor_ref",
"attributes": { "anchorId": "…" }
}]
```

The marker is a *pointer*; the anchor row is the *content*. This keeps the
schema untouched (no migration) and gives RAII for free: markers follow their
memory through supersede/delete.

### 3.5 Writing is active; recall is active + passive

- **Write (active):** when the Agent records a memory that refers to a file
location, it optionally supplies an anchor (`nmg remember … --anchor
path:label`, or a dedicated `nmg anchor` action). Writing memory is already an
active act; adding an anchor is the same act, one extra field. No observer, no
auto-extraction.
- **Recall (active + passive):** active search queries both sources; passive
automatic recall can surface anchors alongside memory. The marker lets recall
walk memory → anchor → file position when needed.

## 4. Architecture

```text
┌─────────────── search ────────────────────────────────┐
│ query → memory hits (existing) + anchor hits (new) │
└───────────────────────────┬───────────────────────────┘
┌───────────────────┴───────────────────┐
▼ ▼
┌───────────────┐ ┌───────────────────┐
│ memory_records│ │ anchors │
│ (LTG/STG) │ marker anchor_ref │ (path, snippet, │
│ │ ──────────────────► │ label, memory_id)│
└───────────────┘ └─────────┬─────────┘
│ resolve snippet
file (content host,
never indexed by NMG)
```

## 5. Boundaries

- Anchors are **not memory**: no LTG/STG semantics, no provenance/verification
on anchor content. The `memory_id` back-pointer is optional linkage, not
memory content.
- Files are **not indexed**: NMG stores pointers, never file content. The
"files are not memory" red line is preserved by construction.
- Anchors are **sparse and Agent-authored**: no parser, no crawler, no
tree-sitter, no mdast. Document "concept units" remain out of scope (no
deterministic parser; see the superseded design's structured-unit note).

## 6. Relation to the superseded design

[file-content-source-design.md](file-content-source-design.md) proposed a
full-text file index as a second search source. This design keeps its
conclusions that are still true — files are not memory; scope discipline
matters; separated presentation is sane — but **drops the file index itself** in
favor of sparse, Agent-authored anchors. The maintenance-heavy machinery
(`.nmg-search-scope`, incremental crawler, file FTS, scope observer) is not part
of this design.

## 7. Open questions (deferred)

- Anchor store location (separate table in the memory DB vs project-local
file). TBD at implementation.
- Snippet length and relocation tolerance (exact vs fuzzy).
- Whether anchors get a TTL or are retired by staleness only.
- Presentation: anchors shown as a third partition, or merged with memory.

## 8. Research basis (surveyed 2026-09-02)

| Reference | What it validates |
| --- | --- |
| [haido DESIGN.md](https://github.com/lebac-svg/haido/blob/HEAD/docs/DESIGN.md) | Anchored memory with objective staleness (`hash_at_link`), not TTLs or LLM self-reflection; anchors drift/missing/moved; recall ranks anchors before full text. Closest full implementation to this design. |
| [gptme `_anchored.py`](https://github.com/gptme/gptme/blob/ae707fc8233e77d4da97fc74f94db1eaff1e381a/gptme/tools/_anchored.py) | Hash-anchored, content-based editing — content anchors survive edits, line numbers do not. |
| [agentic-bookmarks](https://github.com/super-mega-lab/agentic-bookmarks) | Durable bookmarks with self-healing anchors that survive refactors. |
| [ai-memory ARCHITECTURE](https://github.com/akitaonrails/ai-memory/blob/v1.8.0/docs/ARCHITECTURE.md) | Markdown wiki as source of truth, SQLite as derived index — validates "pointers, not replicas". |
| [quote-anchored citations ADR](https://zby.github.io/commonplace/reference/adr/023-quote-anchored-citations-for-code-grounded-reviews/) | Cite by quoted content, not line number, for code-grounded references. |
2 changes: 2 additions & 0 deletions dsh/dsh-nmg/src/plugin/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1155,6 +1155,7 @@ export function apply(ctx: Context): () => void {
residence: { type: 'string', enum: ['ltg', 'stg'], description: 'ltg (durable) or stg (session/task-local).' },
writeReason: { type: 'string', description: 'Durable-write justification.' },
scope: { type: 'object', additionalProperties: true, description: 'Applicability scope, e.g. {"project":"nmg"}.' },
anchors: { type: 'array', maxItems: 10, items: { type: 'object', properties: { path: { type: 'string', description: 'Project-relative file path the bookmark points into.' }, snippet: { type: 'string', description: 'Exact short content excerpt from that file used for relocation.' }, label: { type: 'string', description: 'Optional one-line searchable label.' }, kind: { type: 'string', description: 'Optional kind: code | doc | note.' } }, required: ['path', 'snippet'] }, description: 'Optional file bookmarks (anchors): point this memory at file locations you actually saw. One bookmark per {path, snippet} — snippet is the relocation key, so copy it verbatim from the file.' },
},
required: [],
},
Expand Down Expand Up @@ -1200,6 +1201,7 @@ export function apply(ctx: Context): () => void {
return 'nmg_remember save requires statement and nodeName.'
}
const params: Record<string, any> = { statement: args.statement, nodeName: args.nodeName, projectDir: workspaceRoot }
if (args.anchors && Array.isArray(args.anchors)) params.anchors = args.anchors
if (args.memoryType) params.memoryType = args.memoryType
if (args.recallTriggers) params.recallTriggers = args.recallTriggers
if (args.stateKey) params.stateKey = args.stateKey
Expand Down
32 changes: 31 additions & 1 deletion src/cli/commands.ts
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,7 @@ export const NMG_CLI_COMMANDS: readonly CliCommandSpec[] = [
"opened-at",
"related-memory",
"recall-trigger",
"anchor",
],
flags: [],
usageDetail: `Remember options:
Expand All @@ -222,7 +223,9 @@ export const NMG_CLI_COMMANDS: readonly CliCommandSpec[] = [
--write-source SOURCE Submission channel; defaults to user for the CLI
--external-source REF External provenance: web:URL or file:PATH
--retrieved-at ISO External retrieval timestamp (default: now)
--content-hash HASH Optional external content hash`,
--content-hash HASH Optional external content hash
--anchor PATH::SNIPPET[::LABEL] Repeatable file bookmark: path, ::, content
snippet (relocation key), optional ::LABEL`,
buildParams: rememberParams,
},
{
Expand Down Expand Up @@ -1021,11 +1024,38 @@ function rememberParams(values: OptionValues): NmgRememberParams {
openedAt: firstOption(values, "opened-at"),
relatedMemoryIds: values.options.get("related-memory"),
recallTriggers: values.options.get("recall-trigger"),
anchors: parseAnchorOptions(values.options.get("anchor")),
markers: externalMarker,
projectDir: optionalResolvedPath(firstOption(values, "project-dir")),
}) as unknown as NmgRememberParams;
}

/** Parse repeatable `--anchor PATH::SNIPPET[:LABEL]` options into AnchorInput
* entries. The snippet is everything up to the next `::` (or the end); the
* optional label follows a second `::`. Paths must not contain `::`. */
function parseAnchorOptions(
raw: unknown,
): Array<{ path: string; snippet: string; label?: string }> | undefined {
const values = Array.isArray(raw) ? raw : raw === undefined ? [] : [raw];
const anchors: Array<{ path: string; snippet: string; label?: string }> = [];
for (const value of values) {
const text = String(value ?? "");
const parts = text.split("::");
if (parts.length < 2 || !parts[0]!.trim() || !parts[1]!.trim()) {
throw new Error(
"--anchor must be PATH::SNIPPET (path then :: then content snippet), optionally ::LABEL",
);
}
const path = parts[0]!.trim();
const snippet = parts[1]!.trim();
const label = parts.length > 2 ? parts.slice(2).join("::").trim() : undefined;
const anchor: { path: string; snippet: string; label?: string } = { path, snippet };
if (label) anchor.label = label;
anchors.push(anchor);
}
return anchors.length > 0 ? anchors : undefined;
}

function resolutionParams(
values: OptionValues,
action: "resolve" | "reopen",
Expand Down
31 changes: 25 additions & 6 deletions src/cli/main.ts
Original file line number Diff line number Diff line change
Expand Up @@ -434,6 +434,29 @@ function parseOptions(args: readonly string[]): OptionValues {
return { flags, options, positionals };
}

/** Append the FILES / ANCHORS source-section lines of a search result. Kept as
* its own helper so humanResult's branching stays under the complexity gate. */
function sourceSectionLines(
files: Array<{ path: string; excerpt: string }> | undefined,
anchors: Array<{ path: string; label: string; line?: number; stale?: boolean }> | undefined,
): string[] {
const lines: string[] = [];
if (files && files.length > 0) {
lines.push("FILES:");
for (const file of files) {
lines.push(`${file.path}\t${file.excerpt}`);
}
}
if (anchors && anchors.length > 0) {
lines.push("ANCHORS:");
for (const anchor of anchors) {
const position = anchor.stale ? "(stale)" : anchor.line ? `:${anchor.line}` : "";
lines.push(`${anchor.path}${position}\t${anchor.label}`);
}
}
return lines;
}

function humanResult(value: unknown): string {
const result = value as Record<string, unknown>;
if (result.action === "discover" && Array.isArray(result.agents)) {
Expand Down Expand Up @@ -582,18 +605,14 @@ function humanResult(value: unknown): string {
node: { canonicalName: string };
}>;
files?: Array<{ path: string; excerpt: string }>;
anchors?: Array<{ path: string; label: string; line?: number; stale?: boolean }>;
timings?: { timings?: Record<string, number>; totalMs?: number };
};
const lines = context.results.map(
({ memory, node }) =>
`${memory.id}\t${memory.memoryType}\tL${memory.tier}\t${node.canonicalName}\t${memory.statement}`,
);
if (context.files && context.files.length > 0) {
lines.push("FILES:");
for (const file of context.files) {
lines.push(`${file.path}\t${file.excerpt}`);
}
}
lines.push(...sourceSectionLines(context.files, context.anchors));
if (context.timings) {
const sections = Object.entries(context.timings.timings ?? {})
.sort((left, right) => right[1] - left[1])
Expand Down
Loading