When --update reports that most of the corpus changed, the flow goes straight
from detection to chunking and subagent dispatch. Nothing prompts a check of
where the change is, and a manifest diff cannot tell "this content was
rewritten" from "this content just entered the scan root": both are simply
absent from the manifest.
Related, but a different cause: #2459 and #2033 (false "changed" reports from
kind="semantic"), #2870 and #2908 (files leaving scope), #2239 (other
pre-dispatch gaps on --update). This issue is about genuinely new files that
are new only because the corpus boundary moved.
What happened
Six days after a build, --update reported 799 of 936 files new or changed.
That reads as "the corpus was rewritten" and seems to justify a large extraction.
Grouping the changed documents by their first path component under the scan
root showed that 240 of 277 changed documents (3.31 of 4.14 MB) sat in one
folder that had been moved into the scan root a few days earlier. Nothing in
it was knowledge the graph was meant to hold. Adding that folder to
.graphifyignore took the corpus from 936 to 289 files and the pending work
from 799 to 152 files.
The breakdown took one short script. Without it, the run would have dispatched
several times more subagents than needed, on content that should not have been
in the graph.
Expected
Before spending on a diff that covers most of the corpus, show the shape of the
change so the user can tell a boundary change from a content change.
Proposal (references/update.md, between the detection block and the
code_only check)
When new_total exceeds a fraction of the corpus (for example half of
total_files):
- Group
new_files by first path component relative to the scan root (files
directly in the root count as (root)), with file count and total size per
group. The full build already does this: Step 2 of SKILL.md, when
total_files > 500, ranks the top 5 first-level subdirectories and asks which
one to run on before continuing. --update has no equivalent gate, although
a large diff costs the same as a large build.
- Print the top groups and ask whether the dominant one is intended corpus or a
directory that was moved or cloned in.
- If it should be excluded, offer to append it to
.graphifyignore and rerun
detection before any chunking.
This is cheap (no LLM, one pass over paths already in
.graphify_incremental.json) and runs only when the diff is large enough to be
expensive.
Environment: observed on graphifyy 0.9.55, graphify install --platform claude,
Claude Code on Windows 11; now on 0.9.65. references/update.md checked in
v0.9.72: no such step (the file is unchanged since v0.9.61).
When
--updatereports that most of the corpus changed, the flow goes straightfrom detection to chunking and subagent dispatch. Nothing prompts a check of
where the change is, and a manifest diff cannot tell "this content was
rewritten" from "this content just entered the scan root": both are simply
absent from the manifest.
Related, but a different cause: #2459 and #2033 (false "changed" reports from
kind="semantic"), #2870 and #2908 (files leaving scope), #2239 (otherpre-dispatch gaps on
--update). This issue is about genuinely new files thatare new only because the corpus boundary moved.
What happened
Six days after a build,
--updatereported 799 of 936 files new or changed.That reads as "the corpus was rewritten" and seems to justify a large extraction.
Grouping the changed documents by their first path component under the scan
root showed that 240 of 277 changed documents (3.31 of 4.14 MB) sat in one
folder that had been moved into the scan root a few days earlier. Nothing in
it was knowledge the graph was meant to hold. Adding that folder to
.graphifyignoretook the corpus from 936 to 289 files and the pending workfrom 799 to 152 files.
The breakdown took one short script. Without it, the run would have dispatched
several times more subagents than needed, on content that should not have been
in the graph.
Expected
Before spending on a diff that covers most of the corpus, show the shape of the
change so the user can tell a boundary change from a content change.
Proposal (
references/update.md, between the detection block and thecode_onlycheck)When
new_totalexceeds a fraction of the corpus (for example half oftotal_files):new_filesby first path component relative to the scan root (filesdirectly in the root count as
(root)), with file count and total size pergroup. The full build already does this: Step 2 of
SKILL.md, whentotal_files> 500, ranks the top 5 first-level subdirectories and asks whichone to run on before continuing.
--updatehas no equivalent gate, althougha large diff costs the same as a large build.
directory that was moved or cloned in.
.graphifyignoreand rerundetection before any chunking.
This is cheap (no LLM, one pass over paths already in
.graphify_incremental.json) and runs only when the diff is large enough to beexpensive.
Environment: observed on graphifyy 0.9.55,
graphify install --platform claude,Claude Code on Windows 11; now on 0.9.65.
references/update.mdchecked inv0.9.72: no such step (the file is unchanged since v0.9.61).