diff --git a/Help/markdown/New_in_Current_Version.md b/Help/markdown/New_in_Current_Version.md index 2b05688..9313371 100644 --- a/Help/markdown/New_in_Current_Version.md +++ b/Help/markdown/New_in_Current_Version.md @@ -20,6 +20,8 @@ The ANSI version shares the same source code with the Unicode version, but repli ## Miscellaneous +(4.2.39) **CHANGED:** a lazily loaded `…full.kbb` answers [findconcept](findconcept.md), [getconcept](getconcept.md) and [down](down.md) in any pass. `findconcept` and `getconcept` search the file for a key not in memory yet, and the first `down` on its top concept loads the whole file. Before, a later pass found only keys the tokenizer or a [dictfindword](dictfindword.md) call had loaded, and `down` listed only those. The tokenizer and `dictfindword` now search only dictionary files: a top concept of `dictionary`, or a same-named `.dict` beside the `.kbb`. A domain knowledge base such as `en-synonyms-full.kbb` is no longer searched for every word and spelling candidate, which made one spelling pass go from 0.1 s to 12 s; look its keys up with `findconcept` instead. See [Lazy Loading](vscode/lazyload.md#looking-up-keys-from-any-pass-4239). **FIXED:** `"path" << x` with an absolute path, such as `G("$apppath") + "/out.txt"`, wrote nothing and said nothing. The path is now used as given, as [openfile](openfile.md) does, and a file that can't be opened is reported once, in `err.log` and on standard error. + (4.2.38) **FIXED:** three knowledge-base load failures that gave no error. A `.kbb` line of 2,048 or more characters ended the load: every concept after it was dropped. A `.kbb` or `.dict` value of 512 or more characters overflowed a buffer, and the engine crashed while loading `kb/user`. `.dict` lines that long were skipped, with a message in `cgerr.log` only. Lines and values of any length now load. Every quoted `.kbb` value followed by more text on its line (a `,`, `]` or space) also loaded a stray number `0` after it, so `term=["a b","c d"]` loaded `a b`, `0`, `c d`, `0`. `SaveKB` in `KBFuncs.nlp` writes values in exactly that shape, so a saved knowledge base picked up the extra values when it was loaded again. They are no longer added, but a `.kbb` that was loaded and then saved again before this fix has them written into it: rebuild it from its source data. **CHANGED:** an NLP++ runtime error, such as [findconcept](findconcept.md) given no concept, still skips the rest of its `@CODE` region (or, in `@POST`, the rest of the pass), and the run still exits with code 0. Until now the error went only to the input's `_log/err.log`, so a run that had skipped work looked successful. The engine now also prints one line to standard error, naming the region, pass, pass file, line and input, for example `[Error in NLP++ @CODE: pass 2 (loop) line 13, rest of the region skipped. Input: text.txt. Details in err.log.]`. It is printed once per input for each pass and line, so an error in a rule that matches many times prints one line. Compiled analyzers do not print it. (4.2.37) **FIXED:** [jsonwrite](jsonwrite.md) wrote an array followed by a plain sibling -- a list of items, then a field -- with a stray `}` and no comma after the `]`, so the file was not JSON. Adjacent arrays with different base names (`item1`, `item2`, `thing1`) were merged into one, an array element with no attributes or children ended in a trailing comma, and a name with interior digits was split in the wrong place (`a1b2` became array `a1`, id `12`). `JsonKB`, `JsonKBRecurse` and `SeparateConCount` in `KBFuncs.nlp` had the same bugs and were fixed with it, so the builtin and the NLP++ functions still write identical output. In `KBFuncs.nlp` too, `StripEndDigits` no longer drops the last letter of a name that does not end in digits (`item` stayed `ite`), and `JsonKBRecurse` no longer writes a stray `hier.txt` debug file. Refresh an analyzer's copy of `KBFuncs.nlp` to get these. diff --git a/Help/markdown/dictfindword.md b/Help/markdown/dictfindword.md index 659fe2d..05343f1 100644 --- a/Help/markdown/dictfindword.md +++ b/Help/markdown/dictfindword.md @@ -24,6 +24,8 @@ Handle to the word concept if *wordString* is found in the dictionary, NULL othe ## Remarks +Among lazily loaded `…full.kbb` files, dictfindword searches only dictionaries: a file whose top concept is `dictionary`, or one with a same-named `.dict` file beside it. Before engine 4.2.39 it searched every `…full.kbb`, so it also found keys of a domain knowledge base such as `en-synonyms-full.kbb`; use [findconcept](findconcept.md) on that file's top concept instead. See [Lazy Loading](vscode/lazyload.md#looking-up-keys-from-any-pass-4239). + The dictionary is a specialized part of the KB hierarchy used to store and lookup word forms. You can see it with the KB editor, under **concept > sys > dict**. ## Example diff --git a/Help/markdown/down.md b/Help/markdown/down.md index 6fbc7ec..8ddc76b 100644 --- a/Help/markdown/down.md +++ b/Help/markdown/down.md @@ -24,6 +24,8 @@ First child concept. ## Remarks +If currentConcept is the top concept of a lazily loaded `…full.kbb` file, the first call loads the whole file, so down and [next](next.md) see every key in it, not only those already looked up (engine 4.2.39). See [Lazy Loading](vscode/lazyload.md#looking-up-keys-from-any-pass-4239). + If currentConcept is a bad concept, a warning is written to output log. If there is no child of currentConcept, a warning is written to the output log. ## Example diff --git a/Help/markdown/fileout.md b/Help/markdown/fileout.md index 48fda18..ebcc649 100644 --- a/Help/markdown/fileout.md +++ b/Help/markdown/fileout.md @@ -26,7 +26,7 @@ Old-style.  Use NLP++ syntax such as "file.txt" << "hello\n"; -to create files in the current output area.  Or, use **openfile** to place files anywhere on your file system. +to create files in the current output area. An absolute path is used as given, so `L("f") = G("$apppath") + "/out.txt"; L("f") << "hello\n";` writes to the analyzer folder (engine 4.2.39: it used to write nothing). If the file can't be opened, the engine says so once, in `err.log` and on standard error. Or, use **openfile** to place files anywhere on your file system. ## Example diff --git a/Help/markdown/findconcept.md b/Help/markdown/findconcept.md index 55b436d..e49d439 100644 --- a/Help/markdown/findconcept.md +++ b/Help/markdown/findconcept.md @@ -40,6 +40,8 @@ Numerical version: If a number is passed in instead of a string, the concept wit ## Remarks +If **parentConcept** is the top concept of a lazily loaded `…full.kbb` file and the name is not in memory yet, findconcept searches that file for it and loads it, with everything under it (engine 4.2.39). With an index instead of a name, it first loads the whole file. See [Lazy Loading](vscode/lazyload.md#looking-up-keys-from-any-pass-4239). + If **parentConcept** is not in the KB, that is, if you don't have a handle on a valid concept, an error is thrown in the output log window: "Arg must be Sem object." If an index is chosen outside of the set of existing concepts, findconcept returns 0. ## Example diff --git a/Help/markdown/getconcept.md b/Help/markdown/getconcept.md index 1d1a4e4..69fd320 100644 --- a/Help/markdown/getconcept.md +++ b/Help/markdown/getconcept.md @@ -26,6 +26,8 @@ returnedConcept - type: con ## Remarks +If **parentConcept** is the top concept of a lazily loaded `…full.kbb` file, getconcept first looks for the name in that file and returns what it finds there, rather than making an empty concept (engine 4.2.39). See [Lazy Loading](vscode/lazyload.md#looking-up-keys-from-any-pass-4239). + All concepts have names (see `makeconcept`) and need to be rooted in the concept hierarchy. This function returns a concept given its name and its parent concept. If the parent concept is bad, an error appears in the log output window. If the parent concept does not have a child concept named conceptNameString, this function creates one with that name. If the parent concept does have a child concept named conceptNamestring, this function returns that concept. ## Example diff --git a/Help/markdown/vscode/lazyload.md b/Help/markdown/vscode/lazyload.md index e91330c..1709576 100644 --- a/Help/markdown/vscode/lazyload.md +++ b/Help/markdown/vscode/lazyload.md @@ -50,6 +50,41 @@ eager one did. The attributes arrive on the node either way — a lazily loaded featured dictionary still gives a word its `vform`, `tense` and `number`. Only the timing changes. +## Looking up keys from any pass (4.2.39) + +A lazily loaded `…full.kbb` answers the ordinary knowledge-base functions in any +pass, not only during tokenization: + +- **[findconcept](../findconcept.md) and [getconcept](../getconcept.md)**: when the + parent is the file's top concept and the key is not in memory yet, the engine + searches that file for it, loads the key with everything indented under it, and + returns it. A name the file doesn't have is remembered, so it is searched once. + Names with a space are found whether or not the file quotes them. +- **[down](../down.md)**: the first call on the file's top concept loads the whole + file, so `down` and [next](../next.md) list every key once. That costs what + loading the file eagerly would, so scan the keys only when you need all of them. + [findconcept](../findconcept.md) with a number does the same. + +The tokenizer and [dictfindword](../dictfindword.md) search a `…full.kbb` for every +word only when it is a dictionary: its top concept is `dictionary` (as in +`en-full.kbb`), or a `.dict` file with the same name sits beside it (`en-full.kbb` +with `en-full.dict`). Any other `…full.kbb` is a domain knowledge base, such as +`en-lemmas-full.kbb`, `en-synonyms-full.kbb` or a code index: look its keys up with +`findconcept` on its top concept. + +``` +@CODE +L("lemmas") = findconcept(findroot(),"lemmas"); # en-lemmas-full.kbb +L("con") = findconcept(L("lemmas"),"running"); # loaded now, if not already +if (L("con")) + "out.txt" << strval(L("con"),"root") << "\n"; # run +@@CODE +``` + +Before 4.2.39, `findconcept` found only keys the tokenizer or a `dictfindword` call +had already loaded, `down` listed only those, and every `…full.kbb` was searched for +every word and spelling candidate in the input. + ## Loading a KB or dictionary on demand When only some inputs need a large domain knowledge base or dictionary, you can load it