docs: lazy *full.kbb keys from any pass, << to absolute paths (engine 4.2.39)

Help/markdown/New_in_Current_Version.md

@@ -20,6 +20,8 @@ The ANSI version shares the same source code with the Unicode version, but repli
 
 ## Miscellaneous
 
+(4.2.39) **CHANGED:** a lazily loaded `…full.kbb` answers [findconcept](findconcept.md), [getconcept](getconcept.md) and [down](down.md) in any pass. `findconcept` and `getconcept` search the file for a key not in memory yet, and the first `down` on its top concept loads the whole file. Before, a later pass found only keys the tokenizer or a [dictfindword](dictfindword.md) call had loaded, and `down` listed only those. The tokenizer and `dictfindword` now search only dictionary files: a top concept of `dictionary`, or a same-named `.dict` beside the `.kbb`. A domain knowledge base such as `en-synonyms-full.kbb` is no longer searched for every word and spelling candidate, which made one spelling pass go from 0.1 s to 12 s; look its keys up with `findconcept` instead. See [Lazy Loading](vscode/lazyload.md#looking-up-keys-from-any-pass-4239). **FIXED:** `"path" << x` with an absolute path, such as `G("$apppath") + "/out.txt"`, wrote nothing and said nothing. The path is now used as given, as [openfile](openfile.md) does, and a file that can't be opened is reported once, in `err.log` and on standard error.
+
 (4.2.38) **FIXED:** three knowledge-base load failures that gave no error. A `.kbb` line of 2,048 or more characters ended the load: every concept after it was dropped. A `.kbb` or `.dict` value of 512 or more characters overflowed a buffer, and the engine crashed while loading `kb/user`. `.dict` lines that long were skipped, with a message in `cgerr.log` only. Lines and values of any length now load. Every quoted `.kbb` value followed by more text on its line (a `,`, `]` or space) also loaded a stray number `0` after it, so `term=["a b","c d"]` loaded `a b`, `0`, `c d`, `0`. `SaveKB` in `KBFuncs.nlp` writes values in exactly that shape, so a saved knowledge base picked up the extra values when it was loaded again. They are no longer added, but a `.kbb` that was loaded and then saved again before this fix has them written into it: rebuild it from its source data. **CHANGED:** an NLP++ runtime error, such as [findconcept](findconcept.md) given no concept, still skips the rest of its `@CODE` region (or, in `@POST`, the rest of the pass), and the run still exits with code 0. Until now the error went only to the input's `_log/err.log`, so a run that had skipped work looked successful. The engine now also prints one line to standard error, naming the region, pass, pass file, line and input, for example `[Error in NLP++ @CODE: pass 2 (loop) line 13, rest of the region skipped. Input: text.txt. Details in err.log.]`. It is printed once per input for each pass and line, so an error in a rule that matches many times prints one line. Compiled analyzers do not print it.
 
 (4.2.37) **FIXED:** [jsonwrite](jsonwrite.md) wrote an array followed by a plain sibling -- a list of items, then a field -- with a stray `}` and no comma after the `]`, so the file was not JSON. Adjacent arrays with different base names (`item1`, `item2`, `thing1`) were merged into one, an array element with no attributes or children ended in a trailing comma, and a name with interior digits was split in the wrong place (`a1b2` became array `a1`, id `12`). `JsonKB`, `JsonKBRecurse` and `SeparateConCount` in `KBFuncs.nlp` had the same bugs and were fixed with it, so the builtin and the NLP++ functions still write identical output. In `KBFuncs.nlp` too, `StripEndDigits` no longer drops the last letter of a name that does not end in digits (`item` stayed `ite`), and `JsonKBRecurse` no longer writes a stray `hier.txt` debug file. Refresh an analyzer's copy of `KBFuncs.nlp` to get these.

Help/markdown/dictfindword.md

@@ -24,6 +24,8 @@ Handle to the word concept if *wordString* is found in the dictionary, NULL othe
 
 ## Remarks
 
+Among lazily loaded `…full.kbb` files, dictfindword searches only dictionaries: a file whose top concept is `dictionary`, or one with a same-named `.dict` file beside it. Before engine 4.2.39 it searched every `…full.kbb`, so it also found keys of a domain knowledge base such as `en-synonyms-full.kbb`; use [findconcept](findconcept.md) on that file's top concept instead. See [Lazy Loading](vscode/lazyload.md#looking-up-keys-from-any-pass-4239).
+
 The dictionary is a specialized part of the KB hierarchy used to store and lookup word forms. You can see it with the KB editor, under **concept > sys > dict**.
 
 ## Example

Help/markdown/down.md

@@ -24,6 +24,8 @@ First child concept.
 
 ## Remarks
 
+If currentConcept is the top concept of a lazily loaded `…full.kbb` file, the first call loads the whole file, so down and [next](next.md) see every key in it, not only those already looked up (engine 4.2.39). See [Lazy Loading](vscode/lazyload.md#looking-up-keys-from-any-pass-4239).
+
 If currentConcept is a bad concept, a warning is written to output log. If there is no child of currentConcept, a warning is written to the output log.
 
 ## Example

Help/markdown/fileout.md

@@ -26,7 +26,7 @@ Old-style.  Use NLP++ syntax such as
 
 "file.txt" << "hello\n";
 
-to create files in the current output area.  Or, use **openfile** to place files anywhere on your file system.
+to create files in the current output area.  An absolute path is used as given, so `L("f") = G("$apppath") + "/out.txt"; L("f") << "hello\n";` writes to the analyzer folder (engine 4.2.39: it used to write nothing). If the file can't be opened, the engine says so once, in `err.log` and on standard error.  Or, use **openfile** to place files anywhere on your file system.
 
 ## Example
 

Help/markdown/findconcept.md

@@ -40,6 +40,8 @@ Numerical version: If a number is passed in instead of a string, the concept wit
 
 ## Remarks
 
+If **parentConcept** is the top concept of a lazily loaded `…full.kbb` file and the name is not in memory yet, findconcept searches that file for it and loads it, with everything under it (engine 4.2.39). With an index instead of a name, it first loads the whole file. See [Lazy Loading](vscode/lazyload.md#looking-up-keys-from-any-pass-4239).
+
 If **parentConcept** is not in the KB, that is, if you don't have a handle on a valid concept, an error is thrown in the output log window: "Arg must be Sem object." If an index is chosen outside of the set of existing concepts, findconcept returns 0.
 
 ## Example

Help/markdown/getconcept.md

@@ -26,6 +26,8 @@ returnedConcept - type: con
 
 ## Remarks
 
+If **parentConcept** is the top concept of a lazily loaded `…full.kbb` file, getconcept first looks for the name in that file and returns what it finds there, rather than making an empty concept (engine 4.2.39). See [Lazy Loading](vscode/lazyload.md#looking-up-keys-from-any-pass-4239).
+
 All concepts have names (see `makeconcept`) and need to be rooted in the concept hierarchy. This function returns a concept given its name and its parent concept. If the parent concept is bad, an error appears in the log output window. If the parent concept does not have a child concept named conceptNameString, this function creates one with that name. If the parent concept does have a child concept named conceptNamestring, this function returns that concept.
 
 ## Example

Help/markdown/vscode/lazyload.md

@@ -50,6 +50,41 @@ eager one did.
 The attributes arrive on the node either way — a lazily loaded featured dictionary
 still gives a word its `vform`, `tense` and `number`. Only the timing changes.
 
+## Looking up keys from any pass (4.2.39)
+
+A lazily loaded `…full.kbb` answers the ordinary knowledge-base functions in any
+pass, not only during tokenization:
+
+- **[findconcept](../findconcept.md) and [getconcept](../getconcept.md)**: when the
+  parent is the file's top concept and the key is not in memory yet, the engine
+  searches that file for it, loads the key with everything indented under it, and
+  returns it. A name the file doesn't have is remembered, so it is searched once.
+  Names with a space are found whether or not the file quotes them.
+- **[down](../down.md)**: the first call on the file's top concept loads the whole
+  file, so `down` and [next](../next.md) list every key once. That costs what
+  loading the file eagerly would, so scan the keys only when you need all of them.
+  [findconcept](../findconcept.md) with a number does the same.
+
+The tokenizer and [dictfindword](../dictfindword.md) search a `…full.kbb` for every
+word only when it is a dictionary: its top concept is `dictionary` (as in
+`en-full.kbb`), or a `.dict` file with the same name sits beside it (`en-full.kbb`
+with `en-full.dict`). Any other `…full.kbb` is a domain knowledge base, such as
+`en-lemmas-full.kbb`, `en-synonyms-full.kbb` or a code index: look its keys up with
+`findconcept` on its top concept.
+
+```
+@CODE
+L("lemmas") = findconcept(findroot(),"lemmas");    # en-lemmas-full.kbb
+L("con") = findconcept(L("lemmas"),"running");      # loaded now, if not already
+if (L("con"))
+    "out.txt" << strval(L("con"),"root") << "\n";  # run
+@@CODE
+```
+
+Before 4.2.39, `findconcept` found only keys the tokenizer or a `dictfindword` call
+had already loaded, `down` listed only those, and every `…full.kbb` was searched for
+every word and spelling candidate in the input.
+
 ## Loading a KB or dictionary on demand
 
 When only some inputs need a large domain knowledge base or dictionary, you can load it