Repository navigation
feat(search): count #tags written in note bodies, the way Obsidian does - #717
aliasunder wants to merge 40 commits into
Conversation
parseBodyTags returns the #tags written in a note's body, in the order and spelling Obsidian's Tags view lists them. It runs remark-parse 8.0.3, the parser Obsidian's metadata cache forked, with Obsidian's syntax (tags, wikilinks and embeds, highlights, %% comments, math, footnotes, callouts, block IDs) added as tokenizers, and three ported remark files carrying the changes Obsidian makes to blockquotes, lists and indentation. The Obsidian parts are written from Obsidian's observed behaviour only. The test vectors are Obsidian 1.14.4's own answers for 485 inputs, produced by scripts/obsidian-tag-oracle.ts and scripts/generate-body-tag-vectors.ts; scripts/compare-body-tags-random.ts compares the parser with Obsidian on random documents. remark-parse is pinned, kept out of Dependabot, and its trim@0.0.1 dependency is overridden to 1.0.1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Notes gain a body_tags column, filled from each note's body by the body-tag parser running in a worker thread, so a slow note never blocks requests. A note still parsing after 2 seconds is indexed without body tags and logged as an error. The startup rebuild parses before its write transaction; the file watcher parses before its event check, so a delete or newer edit during a slow parse wins, and a newer event drops a parse still waiting in the queue. Tag search, the tags filter on full-text search and its vector-only mirror, tag listing and the untagged count read the tags property and the body tags together. Results return body_tags beside tags, each spelling once, and leave it out when a note has none. The rebuilt-index log line counts each parse outcome. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The seven tools that return notes or count tags say that a tag matches in the tags property or as a #tag in the body, and that results carry body_tags beside tags: tags is the property vault_update_properties edits, body_tags the tags written in the text, edited with vault_replace_in_note. Each edit-tool name appears only while that tool is served. The tool list grows by 1,301 characters on the default combination, within the current size cap. ARCHITECTURE.md describes the body tags column, the parser and the worker; AGENTS.md gains the re-verification recipe for remark-parse and new Obsidian releases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ctly Both files pass the body tags indexing would compute to upsertNote and the in-process parser to rebuilds. The heading-array test asserts the exact task list instead of a length and its first element, and the reranker test asserts the exact documents the reranker receives instead of a prefix on each. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vault_search_by_property and vault_list_property_values match the tags property only, while tag search and tag listing now also count #tags in note bodies. vault_search_by_property's routing sentence says so, and vault_list_property_values' example names its result as tags property values. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Random comparisons with Obsidian 1.14.4 found the body-tag parser disagreeing on a few block shapes: - a %% comment opens a block only after spaces, never a tab; - a table's alignment row needs no closing pipe; - a footnote definition strips at most four columns of indent from each line, and any text after its colon lets lazy lines join it. The vectors gain Obsidian's answers for those shapes and for three list inputs where the ported remove-indentation file differs from remark's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
These tests write and read up to 40 real files. In a full parallel run on macOS the largest batch already takes over 4 seconds of vitest's default 5, and the body-tag tests' extra load pushed it over. The file's other tests that do real file work already allow 15 seconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
![umm-actually[bot]](https://avatars.githubusercontent.com/in/4233716?s=60&v=4){kind=small}
|
listAllTags counts body-tag occurrences in a note already carrying the same property tag when computing 'most-used spelling'
The new occurrence CTE unions property tags and body tags per note, then the 'spelling shown' picks the tag with the most occurrences (a repeat within one note counting again, per the comment). Because body tags store every occurrence, a note with Failure scenario: A vault user writes Suggested fixAdd a vector/test with `tags: [Topic]` in frontmatter and `#topic` repeated twice in the body, asserting which spelling the result shows, derived from Obsidian.umm-actually · z-ai/glm-5.3-flash |
|
Consolidate duplicated title-stripping logic between reviewLeadLine and dailyNoteBlock
reviewLeadLine splits on Failure scenario: The prompt's title or daily-note header format changes; tests using hardcoded strings keep passing while the prompt's actual output diverges, because two independent test-side literals drifted in the same direction. Suggested fixDerive both from a single constant (e.g. the review title string used in the prompt) or assert against a small builder function.umm-actually · z-ai/glm-5.3-flash |
|
buildExpectedSurvey's optional memory key does not enforce section presence order
The helper builds the full expected output including the '## Memory' section only when Failure scenario: A future change to the builder's memory branch (e.g. wrapping memory in data markers) is made in the builder but the MEMORY_ENABLED=false test's hand-assembled expectation doesn't include the new marker, hiding a real prompt-output mismatch. Suggested fixHave the MEMORY_ENABLED=false test also call buildExpectedSurvey with memory omitted and assert the full text, so the builder's optional-branch logic is exercised, not reimplemented.umm-actually · z-ai/glm-5.3-flash |
|
Duplicated FOLDERS/tags/propertyKeys literals between DEFAULT_FIXTURE_SECTIONS and individual test bodies
DEFAULT_FIXTURE_SECTIONS defines folders/tags/propertyKeys/recent/orphans strings, and several separate Failure scenario: Adding a third fixture note or changing frontmatter updates the shared constant but leaves a retyped per-test expectation stale, so one test fails while the others pass — misleading which behavior broke. Suggested fixReuse DEFAULT_FIXTURE_SECTIONS properties in each per-section test instead of retyping the string, so a fixture change updates one place.umm-actually · z-ai/glm-5.3-flash |
|
umm-actually re-reviewed at No new findings (24 tracked finding(s) across all runs). Context notes
umm-actually · z-ai/glm-5.3-flash |
- Listen for "error" on the worker during a parse, so a worker that dies on an uncaught error (running out of memory) settles its note "failed" and logs it instead of exiting the server. - Stop listening for the reply when the time limit fires, so a reply arriving while the worker stops cannot settle the note "parsed". - Settle a note "closed" without an error log when close() stops its worker during start-up, and post nothing to a worker that reports ready after close(). - Skip the body-tag parse of a note read that a newer event replaced. Ship-Check: pr-review · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Startup no longer waits for the body-tag parser. The rebuild writes each note with the body tags stored before it ran, and the server starts answering at once. A background pass then parses every note again, one at a time, so a note the file watcher parses waits behind at most one note of the pass. The pass writes only tags that changed, skips a note whose rebuild version a later write replaced, stops when the parser closes, and logs its counts as "body-tag pass complete". A note that hits the 2-second parse limit now delays only the background pass, never the server's start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y-tag comparison Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comments now state the facts readers paused on: which listAllTags, why handlers stay attached, what an abort skips, the rollback the body_tags default serves, and the queue order behind the one-note wait. Stale startup claims in server.ts are corrected, the indexing flow and diagram in ARCHITECTURE.md gain the body-tag pass, and AGENTS.md says why knip ignores remark-parse. Structure fixes split a two-line guard, name the tool-text clauses and the random draws, and narrow the worker's port directly. Ship-Check: code-quality · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Readability changes from code review, with no change to the parser's answers: one locator helper replaces the twice-written back-up-one arithmetic, the tokenizer order is built in named steps, bare offsets and delimiter lengths are named, the table check reads only the lines it needs, the list port uses early continues, and comments explain the remaining rules. The digits-only name test and the ASCII digit check each live in one place. Ship-Check: triage · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The background pass logs "body-tag pass stopped because the parser closed" with the notes it reached, instead of a "complete" line whose counts no longer add up. - A stored body_tags value that is JSON but not a list of names now logs a warning, as text that is not JSON already did. - BODY_TAG_PARSE_TIMEOUT_MS lives in body-tag-parser.ts, so the server and the search eval read one constant. - The oracle script treats only a missing tags field as no tags, so a malformed reply fails with its own error. Ship-Check: triage · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… oracle helpers - body-tag parser: a worker that exits mid-parse or fails to start, a stale or malformed reply, and close() while a note is in the worker (the old test closed before the note was posted) - body-tag pass: a note removed before the pass reaches it, a parse that throws, stored tags that are not JSON or not a list, and the column migration on an older database - file watcher: a note whose parse passes the limit is still indexed - oracle script: loadObsidianTagOracle against a stand-in bundle, listedTagNames and readOracleInputs - hasTagCandidate's vector check now asserts on every vector Ship-Check: test-audit · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…notes - A line after a one-column list item that starts with a marker that is not a bullet (`**`, `---`, `1.x`) again counts that marker's width toward its indent, as remark-parse 8 does, so it continues the item the way Obsidian reads it. A readability change had dropped that. - An inline footnote's closing bracket search skips code spans, and an unclosed backtick run ends the footnote, so its tags fall back to document order as in Obsidian. The vectors gain Obsidian's answers for 23 list shapes and 12 footnote shapes. Ship-Check: triage · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- body-tag-parser: a throw from the Worker constructor now settles the note failed and logs it. Before, it rejected the queue's drain, which crashed the server on an unhandled rejection and left every later parse pending. - ARCHITECTURE.md: a #tag in a Markdown link's text counts, as Obsidian reads it; only wikilinks and link destinations hide one. - compare-body-tags-random: refuse a --documents or --seed value that Number() cannot read, instead of comparing zero documents and passing. Ship-Check: bug-check · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A worker that fails to start now settles a note "unavailable" instead of "failed", since it says nothing about the note. The background pass stops on it, so every note keeps the body tags stored before the restart, rather than having them overwritten with none and one error logged per note. The watcher still indexes a changed note without body tags in that case. Ship-Check: triage · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…te columns - The SIGTERM handler closes the body-tag parser before requests drain, so the background pass writes nothing to the index while the server stops. The handler takes named parameters now that it has three. - One NOTE_METADATA_COLUMNS constant replaces the column list seven note-metadata queries each spelled out. - Stored body tags are read with the existing parseStringArray, and a damaged value logs one warning before the note is read as having none. Ship-Check: design-pass · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An inline `$…$` span now honours a backslash at its first character, so `$\$m$` is math as in Obsidian. The list port reads its bullet marker through one helper instead of two copies of the rule. The vectors gain Obsidian's answers for ten math-escape shapes and three empty-bracket list items. Ship-Check: pr-monitor · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A parser throw leaves the worker in place; only the limit, a crash or an exit start a fresh one. Ship-Check: pr-monitor · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The body-tag, trash-sweep and config-read tests wait on background work in a spawned server, whose log line or index write can take longer than vi.waitFor's default one second on a loaded runner. Ship-Check: pr-monitor · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Dispositions for the four PR-level findings from the review at
🔍 ship-check · pr-monitor · claude-opus-5-5 |
The shutdown tests check that the body-tag parser closes before the drain, use a non-default force-exit delay so ignoring it fails, and flush the rejected parser close without running the force-exit timer. A stored body tag list holding a non-name now reads as none, with a warning. Ship-Check: test-audit · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ored JSON - server.ts: destructure the shutdown handler's params at the factory, as createErrorMiddleware does, and separate its steps with blank lines. - search-index.ts: name readStoredBodyTags' input bodyTagsJson, and separate rebuildFromVault's opening steps. - search-helpers.ts: drop "(private)" from the JSON column parsers header now that parseStringArray is exported, list every column it parses, and name the `notes` table in NOTE_METADATA_COLUMNS' comment. - body-tag-parser.ts: fix the run-on sentence about which failures replace the worker. - ARCHITECTURE.md: say the pass writes nothing while requests drain. Ship-Check: code-quality · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r shutdown Once SIGTERM closes the body-tag parser, every parse resolves "closed", and the file watcher wrote each note changed during the drain with no body tags. The next startup carries the stored tags until its pass parses the note again, so tag queries missed that note's tags until then. The watcher now leaves the note's row as it was; the next startup's rebuild reads the note from disk. Ship-Check: bug-check · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the parser closes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…search text The list port's spacing scan runs only after a bullet was read, so it sits inside that block instead of behind a second check of the same marker. The unescaped-search helper's parameter is named for what it is. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Add the body-tag modules to the search layer's module table
The PR adds body-tag-parser.ts and body-tag-worker.ts to src/vault-mcp/search/ and updates AGENTS.md's tree and .devin/wiki.json to name them, but ARCHITECTURE.md's module-decomposition section still says the search query and indexing layer is split into seven modules and its table omits both new files. A reader using that section as the search-layer module map misses the worker-thread parsing design. Failure scenario: A contributor opens ARCHITECTURE.md → Module decomposition to map the search/ folder before changing indexing behavior; the table lists seven modules with no body-tag entries, so they miss the two new modules the folder actually contains and edit against a stale map, while the same PR's AGENTS.md tree lists them. Suggested fixAdd rows for body-tag-parser.ts and body-tag-worker.ts to the table and update the 'seven modules' count — or extend the parenthetical to include the body-tag parser among the modules described above (Body tags column).umm-actually · z-ai/glm-5.3-flash |
…cannot answer The watcher stored no body tags for a note it re-indexed while the body-tag worker could not start, and the startup pass already keeps every note's stored tags in that case. upsertNote now takes "keep-stored", which keeps the tags stored for the path (none for a new path), and the watcher passes it when the worker cannot start or the server's shutdown has closed the parser. The closed case now indexes the edit too, instead of leaving the row as it was. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ils to start Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ocs with keep-stored The stored-tag reader's doc now covers the watcher's keep-stored write as well as the rebuild, the upsert doc names the outcomes that pass keep-stored, and ARCHITECTURE.md says a watcher re-index still tries a fresh worker. Ship-Check: code-quality · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…es clear stored tags - Split the keep-stored index test into one test per behavior, and add a case for a damaged stored value: a warning, no tags, and the row repaired. - The watcher's over-limit test now starts from a tagged note and covers a failed parse too, so it fails if those outcomes keep the stored tags. - Pin the shutdown handler's 10-second default force-exit timeout. Ship-Check: test-audit · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…annot answer A re-indexed note with no tag candidate gets no tags even while the worker cannot start, a parse after close() resolves closed even for such a note, a removal also replaces a dropped parse, and a stopped startup pass leaves the tags it already wrote in place. Ship-Check: bug-check · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Disposition for the PR-level finding from the review at
🔍 ship-check · pr-monitor · claude-opus-5-5 |
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s with --out Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s rebuild steps - noteHasTagCondition's doc comment states that the predicate must contain exactly one `?`, in bullets with complete sentences. - search-eval separates stopping the body-tag pass from waiting on the embedding pass with blank lines. - AGENTS.md step 4 says what disagreements.json holds in a full sentence. Ship-Check: code-quality · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Add the missing 'Upgrading remark-parse' section to AGENTS.md
The AGENTS.md tree comment points readers to a section named 'Upgrading remark-parse' for re-checking body-tag parity, and the PR description says that section gives the steps to run after a remark-parse bump or an Obsidian release — but no such section exists anywhere in the provided AGENTS.md. The re-check procedure the PR's whole verification story (oracle scripts, 555 vectors, 300k random docs) depends on is therefore documented nowhere, so the dangling reference cannot be followed. Failure scenario: A maintainer (or a Dependabot PR after a version drift) bumps remark-parse to a newer release; following the tree's pointer they look for AGENTS.md → 'Upgrading remark-parse', find no such section, and merge without re-running the oracle/vector/random-document re-check — silently breaking Obsidian tag parity, which is the exact regression class the PR's vector suite was built to catch. Suggested fixAdd the 'Upgrading remark-parse' section (steps: re-run scripts/obsidian-tag-oracle.ts, regenerate obsidian-body-tag-vectors.json via generate-body-tag-vectors.ts, and re-run compare-body-tags-random.ts after a remark-parse bump or an Obsidian release), or drop the '(see "Upgrading remark-parse")' pointer and fold the steps into an existing section.umm-actually · z-ai/glm-5.3-flash |
… shortcut test for what it checks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Disposition for the PR-level finding from the review at
🔍 ship-check · pr-monitor · claude-opus-5-5 |
…erator accepts them - compare-body-tags-random --out labelled each record "seed S, case N". Case numbers restart at 1 on every run while the fixture keeps earlier runs' labels, so the vector generator rejected the file the remark-parse upgrade steps pass it (duplicate labels: random comparison, seed 1, case 1). Each label now carries the first 12 hex digits of its input's SHA-256. - search-eval's comment says closing the parser stops the body-tag pass after at most one note: a first note with no tag candidate finishes before the eval closes the parser. Ship-Check: bug-check · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What does this PR do?
Tags written in a note's text (
#project) now count as tags, the way Obsidian counts them. Tag search, tag listing and thetagsfilter on full-text search match them, and results return them asbody_tagsbesidetags.parseBodyTags(obsidian-markdown/body-tags.ts) runs remark-parse 8.0.3, the parser Obsidian's metadata cache is built on. Obsidian's own syntax is added as tokenizers inobsidian-remark-parser.ts: tags, wikilinks and embeds, highlights,%%comments, math, footnotes, callouts and block IDs. Three ported remark files inremark-ports/carry Obsidian's changes to blockquotes, lists and indentation. The Obsidian parts are written from Obsidian's observed behaviour only.obsidian-body-tag-vectors.json) are Obsidian 1.14.4's own answers for 555 inputs.scripts/obsidian-tag-oracle.tsasks Obsidian,generate-body-tag-vectors.tsrewrites the answers, andcompare-body-tags-random.tscompares the parser with Obsidian on random documents; the last run, 300,000 documents against the final parser, found no disagreement. Removing any Obsidian-specific piece (the code-span, tag or comment tokenizers, or any of the three ported files) fails at least one vector. AGENTS.md → "Upgrading remark-parse" gives the steps to re-check after a remark-parse bump or an Obsidian release.search/body-tag-parser.ts,body-tag-worker.ts), one note at a time, with a 2-second limit per note. remark-parse slows down sharply on some long runs of punctuation (an 8 KB line ofa_btakes Obsidian itself about 2 seconds), and the worker keeps that off the thread that answers requests. A note that passes the limit is indexed without body tags and logged as an error, and the worker restarts; so does a note whose worker crashes. When the worker cannot start, every note keeps its stored tags, including a note the file watcher re-indexes. A note with no#followed by a tag character skips the worker.body_tagscolumn holds every occurrence.body-tag pass completelog line counts each outcome. A slow note costs background time only.vault_search_by_tag,vault_list_tags,vault_search'stagsfilter (and its vector-only results) and the untagged count in thevault-orientationprompt read the tags property and body tags together. A note with the same tag in both places counts once.tagsis the propertyvault_update_propertiesedits,body_tagsthe tags written in the text, edited withvault_replace_in_note. Each edit-tool name appears only while that tool is served. The default tool list grows by 1,375 characters, to 3,965 per tool against the 3,997 cap.remark-parseis pinned at 8.0.3 and kept out of Dependabot, since its version is part of matching Obsidian; itstrim@0.0.1dependency is overridden to 1.0.1.Tool-definition review: the seven changed tools passed a cold and a diff read with one defect, now fixed:
vault_search_by_propertydid not say itstagskey misses body tags. A diff read of that fix and of a relabelledvault_list_property_valuesexample found nothing.On SIGTERM the shutdown handler closes the parser before requests drain: the background pass stops, and a note saved while the server stops keeps its stored tags.
Live test
Tested head
dc85f0f1: test deploy run 41130011695,ghcr.io/aliasunder/vault-cortex:testwith image revisiondc85f0f179b4e683f1c4ffec4260a1a6cacab463, against a real vault of 2,743 notes, compared with Obsidian 1.14.4. The "before" column ran on the image the instance served before the deploy, which has no body tags.vault_search_by_tag("shoutout")[]shoutout, each withshoutoutinbody_tagsbody_tagsagainst Obsidian's metadata cachevault_search_by_tag({ tag, exact: true })for every tag (582) against Obsidian's notes for that tagvault_list_tagsagainst Obsidian's Tags viewproject, which no note carries directly: it is only the parent ofproject/...tags#prbody717in prose,#prcomment717in%% %%,`#prcode717`and[[x#prlink717]]body_tagsis["prbody717", "prcomment717"]; tag search finds the note for those two and not the other two; after the note is deleted, tag search finds nothingbody_tagskey (0 of 2,714 such notes carry an empty one)file watcher startedtorebuilt indexbody-tag pass completeType of change
Checklist
npm testpassesnpm run buildsucceeds🤖 Generated with Claude Code