Repository navigation
Large Files: when zone mode helps, War and Peace, -ZONES - #254
Merged
ddehilster merged 3 commits intoOct 28, 2026
Merged
Conversation
- Memory goes by the skim tree as well as the largest zone: the passes before the zone line build a tree of the whole document that zones cannot free. Check zone spans in final.tree. - War and Peace beside Pride and Prejudice: 1,910 MB to 455 MB at the same time (1,120 s vs 1,106 s); final tree 345 MB to 1.3 MB. - The oversized-zone story: before parse-en-us v1.1.6 a "1." line made one zone of 71% of the book, and left 18% of its words UNKNOWN. - parse-en-us is zoned by default from v1.1.5. - Compiled analyzers run zones from engine 4.2.49 (the page said they ignore them). - Checklist: zoned but memory barely drops. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New section: -ZONES 40-45 runs only those zones; the others get only the skim passes, -DEV writes trees for every zone that runs, and the KB and G() start without what the skipped zones would have added. War and Peace: five zones in 27 s against 18 minutes for the whole book. Also a debugging pointer and a checklist row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ddehilster
changed the title
Member
Author
|
Added a section on |
A single document fits whole further than expected (War and Peace in 1.9 GB on 32-bit; about 20 MB of text on a 16 GB 64-bit machine), and zoning is no faster (1,120 s vs 1,106 s) and no more accurate. What it buys: -DEV on a large document (unzoned, War and Peace would write some 45 GB of trees), rerunning one part with -ZONES (27 s against 18 min), memory when many documents run at once or memory is limited (1,910 MB to 455 MB), and documents too big to fit whole. The checklist now says to run a document whole unless one of those applies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ddehilster
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updates the Large Files help page with what Pride and Prejudice and War and Peace showed about zone mode, and documents
-ZONES(engine 4.2.50).When zone mode helps (new section): an honest account.
-DEVon a large document: unzoned, War and Peace would write some 45 GB of trees.-ZONES: 27 s against 18 min.What memory goes by: the skim tree (the passes before the
zoneline build a tree of the whole document, which zones cannot free) as well as the largest zone, and how to check zone spans infinal.tree.War and Peace beside Pride and Prejudice, and the oversized-zone story. Before parse-en-us v1.1.6 (VisualText/parse-en-us#13), a
1.line made one zone of 71% of the book, and left 18% of its words UNKNOWN.Running only some zones (
-ZONES, engine 4.2.50, VisualText/nlp-engine#810): syntax, what the other zones get,-DEVtrees for every zone that runs, and the caveat that the KB andG()start without the skipped zones.Also:
Merge after engine 4.2.50 is released.
🤖 Generated with Claude Code