From 3ea611ad78e8eca26aaac7d79f8989a0c315fbeb Mon Sep 17 00:00:00 2001 From: David de Hilster Date: Tue, 27 Oct 2026 19:42:29 -0400 Subject: [PATCH 1/3] Large Files: what zone-mode memory goes by, War and Peace - Memory goes by the skim tree as well as the largest zone: the passes before the zone line build a tree of the whole document that zones cannot free. Check zone spans in final.tree. - War and Peace beside Pride and Prejudice: 1,910 MB to 455 MB at the same time (1,120 s vs 1,106 s); final tree 345 MB to 1.3 MB. - The oversized-zone story: before parse-en-us v1.1.6 a "1." line made one zone of 71% of the book, and left 18% of its words UNKNOWN. - parse-en-us is zoned by default from v1.1.5. - Compiled analyzers run zones from engine 4.2.49 (the page said they ignore them). - Checklist: zoned but memory barely drops. Co-Authored-By: Claude Opus 5.5 --- Help/markdown/vscode/largefiles.md | 52 +++++++++++++++++++++++++----- 1 file changed, 44 insertions(+), 8 deletions(-) diff --git a/Help/markdown/vscode/largefiles.md b/Help/markdown/vscode/largefiles.md index 495b797..eb1ad7a 100644 --- a/Help/markdown/vscode/largefiles.md +++ b/Help/markdown/vscode/largefiles.md @@ -60,16 +60,50 @@ How it works: file to file in a folder run. Node offsets count from the start of the document, and there is one set of output files. - **Let the tree go.** After its passes a zone keeps its node, attributes and span, - and the tree under it is freed. Memory goes by the **largest zone**, not the - document. While a zone runs, the other zones are hidden, earlier ones included. + and the tree under it is freed. While a zone runs, the other zones are hidden, + earlier ones included. - **Wrap up.** The passes after an `endzone` line run once, after the last zone, on the zone nodes and the whole KB. Put whole-document work there: totals, a summary, dumping the KB. -On *Pride and Prejudice* (738 KB) with parse-en-us zoned by paragraph -(`_TEXTZONE`), peak memory went from 473 MB to 153 MB, time from 275 to 283 -seconds, 99.995% of words got the same part-of-speech tag, and the KB at the end -was byte-identical. +### What memory goes by + +Peak memory in zone mode is the larger of two things: + +- **The skim tree.** The passes before the `zone` line build a tree of the whole + document, at least its tokens, and that tree exists before any zone runs. Zones + cannot free it. Keep the skim passes few and cheap. +- **The largest zone.** Each zone's full tree exists only while its passes run. + +So zone mode only works as well as the skim divides the document. Look at the zone +spans in `final.tree`: if one zone covers most of the document, that zone is +analyzed whole and memory is barely lower than without zones. + +parse-en-us runs in zone mode by default from v1.1.5, one zone per paragraph +(`_TEXTZONE`). On two novels, peak memory and time: + +| Book | Without zones | Zoned | +|---|---|---| +| *Pride and Prejudice*, 738 KB, 127,360 words | 473 MB, 275 s | 153 MB, 283 s | +| *War and Peace*, 3.3 MB, 563,286 words | 1,910 MB, 1,120 s | 455 MB, 1,106 s | + +Zoning takes about the same time either way. *War and Peace* ran on a 32-bit +build, whose limit is 2 GB: without zones it used almost all of it, so a somewhat +longer book would have failed, while zoned it used under a quarter. Its final +tree went from 345 MB to 1.3 MB. + +On *Pride and Prejudice*, 99.995% of words got the same part-of-speech tag and the +KB at the end was byte-identical. On *War and Peace*, 97.5% did; the changes go +both ways, and the zoned run leaves fewer words UNKNOWN (1.6% against 2.1%). +*War and Peace* is about 4.4 times longer, so its skim tree is too, and that, not +its paragraphs, sets its 455 MB. + +*War and Peace* also showed the cost of one oversized zone. In parse-en-us before +v1.1.6, a line starting `1.` inside running text (Tolstoy's list of the seven +Freemason virtues) began a zone that ran to the end of the book: 2.3 MB, 71% of it. +Zoned, memory fell only to 776 MB. The same huge "paragraph" had also left 18% of +the book's words tagged UNKNOWN, with or without zones. With the zone stopping at +the next paragraph, the largest zone is 4.4 KB and 1.6% of words are UNKNOWN. ### Example: one zone per form @@ -150,8 +184,9 @@ makes the cover page a `_form` of its own. `final.tree` shows the zone nodes. - **No zones found.** If the skim builds no node of the zone type, the passes after the `zone` line run on the whole document, and a warning says so. -- **Compiled analyzers** ignore the `zone` and `endzone` lines and run every pass on - the whole document. +- **Compiled analyzers** run zones the same way, from engine 4.2.49. Before that + they ignored the `zone` and `endzone` lines and ran every pass on the whole + document. ### Splitting a file into pieces @@ -217,6 +252,7 @@ covers this in full; in short: |---|---| | is a few hundred KB or less | nothing: run it whole | | has parts that stand alone (forms, sections, paragraphs) | zone mode: a skim pass, `zone _part`, `endzone` | +| is zoned but memory barely drops | look in `final.tree` for one zone covering most of the document, and fix the skim | | comes as many files, or must be split | stream the pieces through `-STDIN` | | might hit a looping rule or run too long | `-TIMELIMIT`; the rec pass limit is on by default | | uses a large dictionary or KB | name it `…full.dict` / `…full.kbb`; look keys up with `findconcept` | From c3c66b142934270847c71fd951d6826e0ab6e0eb Mon Sep 17 00:00:00 2001 From: David de Hilster Date: Tue, 27 Oct 2026 20:30:17 -0400 Subject: [PATCH 2/3] Large Files: running only some zones with -ZONES (engine 4.2.50) New section: -ZONES 40-45 runs only those zones; the others get only the skim passes, -DEV writes trees for every zone that runs, and the KB and G() start without what the skipped zones would have added. War and Peace: five zones in 27 s against 18 minutes for the whole book. Also a debugging pointer and a checklist row. Co-Authored-By: Claude Opus 5.5 --- Help/markdown/vscode/largefiles.md | 31 +++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/Help/markdown/vscode/largefiles.md b/Help/markdown/vscode/largefiles.md index eb1ad7a..7345f05 100644 --- a/Help/markdown/vscode/largefiles.md +++ b/Help/markdown/vscode/largefiles.md @@ -181,13 +181,41 @@ makes the cover page a `_form` of its own. - **Debugging.** With `-DEV`, the first three zones write their pass trees as `z001-ana040.tree`, `z002-ana040.tree` and so on; later zones write none, which keeps a run over thousands of zones from writing hundreds of thousands of files. - `final.tree` shows the zone nodes. + `final.tree` shows the zone nodes. To debug a later zone, run just that zone with + `-ZONES` (below). - **No zones found.** If the skim builds no node of the zone type, the passes after the `zone` line run on the whole document, and a warning says so. - **Compiled analyzers** run zones the same way, from engine 4.2.49. Before that they ignored the `zone` and `endzone` lines and ran every pass on the whole document. +### Running only some zones (engine 4.2.50) + +When one part of a large document goes wrong, rerun only that part. `-ZONES` names +the zones to run: + +``` +nlp -ANA parse-en-us -ZONES 4100-4104 -DEV wp.txt +``` + +- **Which zones.** A zone number (`-ZONES 7`), a range (`40-45`) or a list + (`3,40-45`). Zones count from 1 in document order: zone 40 is the 40th zone node + in `final.tree`, and its pass trees are named `z040-ana###.tree`. +- **The other zones** get only the skim passes. The passes after `endzone` still run. + `$isfirstzone` and `$islastzone` mark the first and last zone that runs, so + per-document setup and wrap-up code still works. +- **With `-DEV`**, every zone that runs writes its pass trees, not only the first + three, so you can step through the trees of the zone you care about. +- **It is quick.** The skim and the analyzer load still cover the whole document, + but nothing else does. On *War and Peace*, five zones took 27 seconds, against + 18 minutes for the whole book (117 seconds with `-DEV`). +- **Not quite a full run.** The KB and `G()` start without what the skipped zones + would have added, so a zone whose analysis depends on earlier zones can come out + differently. Check a fix with a full run before you rely on it. + +`stderr` says how many zones ran, and a spec that is not zone numbers, such as +`3-1`, stops the run with an error. + ### Splitting a file into pieces Without zone mode, a document can be split into files and streamed through one @@ -253,6 +281,7 @@ covers this in full; in short: | is a few hundred KB or less | nothing: run it whole | | has parts that stand alone (forms, sections, paragraphs) | zone mode: a skim pass, `zone _part`, `endzone` | | is zoned but memory barely drops | look in `final.tree` for one zone covering most of the document, and fix the skim | +| goes wrong in one part, and a full run is slow | rerun that part with `-ZONES 40-45 -DEV` | | comes as many files, or must be split | stream the pieces through `-STDIN` | | might hit a looping rule or run too long | `-TIMELIMIT`; the rec pass limit is on by default | | uses a large dictionary or KB | name it `…full.dict` / `…full.kbb`; look keys up with `findconcept` | From 7f8280fb23cfcb2c2f487c49d309220626079a97 Mon Sep 17 00:00:00 2001 From: David de Hilster Date: Tue, 27 Oct 2026 20:53:49 -0400 Subject: [PATCH 3/3] Large Files: when zone mode helps, and when it does not A single document fits whole further than expected (War and Peace in 1.9 GB on 32-bit; about 20 MB of text on a 16 GB 64-bit machine), and zoning is no faster (1,120 s vs 1,106 s) and no more accurate. What it buys: -DEV on a large document (unzoned, War and Peace would write some 45 GB of trees), rerunning one part with -ZONES (27 s against 18 min), memory when many documents run at once or memory is limited (1,910 MB to 455 MB), and documents too big to fit whole. The checklist now says to run a document whole unless one of those applies. Co-Authored-By: Claude Opus 5.5 --- Help/markdown/vscode/largefiles.md | 36 ++++++++++++++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-) diff --git a/Help/markdown/vscode/largefiles.md b/Help/markdown/vscode/largefiles.md index 7345f05..1c071c5 100644 --- a/Help/markdown/vscode/largefiles.md +++ b/Help/markdown/vscode/largefiles.md @@ -28,6 +28,38 @@ analyzer can exhaust memory. The largest input the engine accepts is **2,000,000,000 bytes**, on every platform. A larger file is refused with a message on stderr. +### When zone mode helps + +A single document fits whole further than you might expect. *War and Peace* +(3.3 MB, 563,286 words) runs through parse-en-us in one tree, in 1.9 GB, even on a +32-bit build. At about 700 MB per MB of text, a 64-bit machine with 16 GB holds +roughly 20 MB of text whole. For one document of ordinary length on a 64-bit +machine, zone mode is insurance, not a necessity. + +Zone mode does **not** make a run faster: the same work is done either way +(*War and Peace*: 1,120 seconds whole, 1,106 zoned). It is not a way to get better +results either; nearly every word gets the same tag. + +What it does buy: + +- **Debugging a large document.** `-DEV` writes the whole tree after every pass. + Unzoned, *War and Peace*'s tree is 345 MB, and parse-en-us has about 130 passes: + some 45 GB of log files, so in practice a book cannot be debugged at all. Zoned, + `-DEV` writes trees only for the first three zones, or for the zones you pick. +- **Rerunning one part.** With `-ZONES` (engine 4.2.50), the part of a document + that went wrong reruns alone, with full `-DEV` trees: five zones of *War and + Peace* in 27 seconds, against 18 minutes for the book. This is the main reason + to zone an analyzer you are still hardening. +- **Memory when it is short.** Memory goes from the whole document to its skim + tree and largest zone: *War and Peace* went from 1,910 MB to 455 MB. That matters + when many documents run at once (a server, a batch run in parallel, a Python + service), under a memory-limited container or cloud function, or on a 32-bit + build. +- **Documents too big to fit whole.** Tens of MB and up, with a large analyzer: + long legal or regulatory filings, logs, concatenated corpora. + +If none of these apply, run the document whole. + ### Zone mode (engine 4.2.47) Zone mode handles a large input the way a person reads one: **skim** it to find its @@ -278,8 +310,8 @@ covers this in full; in short: | The input... | Do this | |---|---| -| is a few hundred KB or less | nothing: run it whole | -| has parts that stand alone (forms, sections, paragraphs) | zone mode: a skim pass, `zone _part`, `endzone` | +| fits in memory, runs alone, and needs no debugging piece by piece | nothing: run it whole | +| is large and has parts that stand alone (forms, sections, paragraphs), and you need to debug it, run many at once, or save memory | zone mode: a skim pass, `zone _part`, `endzone` | | is zoned but memory barely drops | look in `final.tree` for one zone covering most of the document, and fix the skim | | goes wrong in one part, and a full run is slow | rerun that part with `-ZONES 40-45 -DEV` | | comes as many files, or must be split | stream the pieces through `-STDIN` |