docs: .kbb/.dict lines of any length, NLP++ errors on stderr (engine 4.2.38)

Help/markdown/New_in_Current_Version.md

@@ -20,6 +20,8 @@ The ANSI version shares the same source code with the Unicode version, but repli
 
 ## Miscellaneous
 
+(4.2.38) **FIXED:** three knowledge-base load failures that gave no error. A `.kbb` line of 2,048 or more characters ended the load: every concept after it was dropped. A `.kbb` or `.dict` value of 512 or more characters overflowed a buffer, and the engine crashed while loading `kb/user`. `.dict` lines that long were skipped, with a message in `cgerr.log` only. Lines and values of any length now load. Every quoted `.kbb` value followed by more text on its line (a `,`, `]` or space) also loaded a stray number `0` after it, so `term=["a b","c d"]` loaded `a b`, `0`, `c d`, `0`. `SaveKB` in `KBFuncs.nlp` writes values in exactly that shape, so a saved knowledge base picked up the extra values when it was loaded again. They are no longer added, but a `.kbb` that was loaded and then saved again before this fix has them written into it: rebuild it from its source data. **CHANGED:** an NLP++ runtime error, such as [findconcept](findconcept.md) given no concept, still skips the rest of its `@CODE` region (or, in `@POST`, the rest of the pass), and the run still exits with code 0. Until now the error went only to the input's `_log/err.log`, so a run that had skipped work looked successful. The engine now also prints one line to standard error, naming the region, pass, pass file, line and input, for example `[Error in NLP++ @CODE: pass 2 (loop) line 13, rest of the region skipped. Input: text.txt. Details in err.log.]`. It is printed once per input for each pass and line, so an error in a rule that matches many times prints one line. Compiled analyzers do not print it.
+
 (4.2.37) **FIXED:** [jsonwrite](jsonwrite.md) wrote an array followed by a plain sibling -- a list of items, then a field -- with a stray `}` and no comma after the `]`, so the file was not JSON. Adjacent arrays with different base names (`item1`, `item2`, `thing1`) were merged into one, an array element with no attributes or children ended in a trailing comma, and a name with interior digits was split in the wrong place (`a1b2` became array `a1`, id `12`). `JsonKB`, `JsonKBRecurse` and `SeparateConCount` in `KBFuncs.nlp` had the same bugs and were fixed with it, so the builtin and the NLP++ functions still write identical output. In `KBFuncs.nlp` too, `StripEndDigits` no longer drops the last letter of a name that does not end in digits (`item` stayed `ite`), and `JsonKBRecurse` no longer writes a stray `hier.txt` debug file. Refresh an analyzer's copy of `KBFuncs.nlp` to get these.
 
 (4.2.15) **CHANGED:** a negative [strpiece](strpiece.md) position now counts from the end of the string: `strpiece("hello", 0, -2)` is `hell`. Code that computes a position as `strlength(s) - k` changes meaning when `s` is shorter than `k`: the position comes out negative and now wraps round to the end instead of running off the start, so `strpiece(",", 0, strlength(",") - 2)` is `,` rather than empty, and a loop that trims a trailing character this way never ends. Write the position from the end directly, `strpiece(s, 0, -2)`, which is the same for any string long enough and empty for one too short. `KBFuncs.nlp` and `UtilFuncs.nlp` were checked for this pattern and fixed where it could wrap (4.2.37).

Help/markdown/loaddict.md

@@ -32,6 +32,8 @@ A `.dict` file uses the flat `word attr=val ...` format, one word per line (see
 
 A `#` outside a quoted string starts a comment, and the engine skips the whole line, including any entry before the `#`. Put comments on lines of their own: `beta pos=verb # a comment` adds nothing for `beta`.
 
+Lines and values can be any length. Before engine 4.2.38 a line of 2,048 or more characters was skipped, with a message in `cgerr.log` only, and a quoted value of 512 or more characters crashed the engine.
+
 To load an indented hierarchy file instead, use [loadkbb](loadkbb.md).
 
 ## Example

Help/markdown/loadkbb.md

@@ -30,6 +30,8 @@ Returns **1** if the file was loaded successfully and **0** otherwise (for examp
 
 A `.kbb` file uses the indented "dictionary" format, where indentation expresses the concept hierarchy. `loadkbb` adds the concepts in the file to the KB already in memory, hanging them off the existing hierarchy. This is the same mechanism the engine uses when it loads `*.kbb` files during normal KB startup, exposed so an analyzer can pull in an additional `.kbb` on demand.
 
+Lines, concept names, attribute names and values can be any length. Before engine 4.2.38 a line of 2,048 or more characters silently ended the load, a value of 512 or more characters crashed the engine, and every quoted value followed by more text on its line also loaded a stray number `0`. A `.kbb` that was loaded and then saved again before 4.2.38 has those `0` values written into it, so rebuild it from its source data.
+
 To load a flat `word attr=val ...` dictionary file instead, use [loaddict](loaddict.md).
 
 ## Example