Repository navigation
.kbb/.dict lines and values of any length; NLP++ errors echoed to stderr (4.2.38) - #797
Merged
Merged
Conversation
Three silent failures in the KB file readers, found building large
knowledge bases for an ICD-10 coder. Each exited 0 with nothing on the
console.
readKBB read each line into a 2,048-byte buffer. A longer line set
failbit, peek() then returned EOF, and every line after it was dropped:
a 30-concept .kbb whose line 10 ran ~2,500 chars loaded 10 concepts.
It now reads std::string lines. So do readDict (which skipped such lines
with a cgerr message), the lazy *full.dict / *full.kbb loaders, and the
catenation that writes a dumped KB.
parseKBBLine copied names, attributes and values unbounded into
512-byte buffers, so a 600-char value overflowed the stack and the
engine aborted at startup ("buffer overflow detected" on Linux).
parseDictLine had the same unbounded copy for quoted values, and
addConceptByPath copied an end offset's worth of bytes as if it were a
length. All of them now copy into std::string.
Every quoted value with anything after it on the line (a ",", "]" or
space) also loaded a stray number 0 after it: ["a","b"] came in as
a, 0, b, 0. The empty text between the closing quote and the separator
was taken as one more value. It is now skipped. No .kbb shipped with the
engine or analyzers has this shape; concept-path arrays are unaffected.
Fixtures kbb-long-line and kbb-long-value cover both files and both
limits.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An NLP++ error, such as findconcept() given a null concept, abandons the rest of its @code region (or the pass, in @post), and the run still exits 0. The error trail went to the input's err.log only, so the run looked clean: in an ICD-10 build it dropped 27,845 of 163,354 lines. The semantics are unchanged. Parse::nlpppError now also writes one line to stderr naming the region, pass, pass file, line and input: [Error in NLP++ @code: pass 2 (loop) line 13, rest of the region skipped. Input: .../text.txt. Details in err.log.] for @code, @postcode, @post and @check. It prints once per input for each pass and line, so an error in a rule that fires on every match prints one line, not one per match. New CI step on Linux, Windows and macOS runs the kbb-long-line, kbb-long-value and code-region-error fixtures and asserts on out.txt and on stderr. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ddehilster
added a commit
to VisualText/visualtext-files
that referenced
this pull request
Oct 20, 2026
…4.2.38) New_in_Current_Version gets a 4.2.38 entry. loadkbb and loaddict say lines and values can be any length, and what happened before 4.2.38: a 2,048+ char .kbb line ended the load, a 512+ char value crashed the engine, a long .dict line was skipped, and every quoted .kbb value followed by more text loaded a stray 0. SaveKB writes that shape, so a .kbb loaded and saved again before the fix should be rebuilt. The entry also covers the one change analyzer authors can see: an NLP++ runtime error still skips the rest of its region and exits 0, but now prints one line to stderr, not only to err.log. Engine PR: VisualText/nlp-engine#797. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes four bugs that all failed silently: exit code 0 and nothing on the console. They were found while building large knowledge bases for an ICD-10 coding analyzer.
.kbb and .dict readers (
cs/libconsh/cg.cpp)readKBBread each line into a 2,048-byte buffer. A longer line madegetlinefail, and every line after it was dropped: a 30-concept.kbbloaded 10 concepts. Lines are now read intostd::string, both here and inreadDict(which used to skip such lines), the lazy*full.dict/*full.kbbloaders, and the code that writes a saved KB.parseKBBLinecopied names, attributes and values into 512-byte buffers with no bound. A 600-character value aborted the engine at startup (buffer overflow detectedon Linux).parseDictLineandaddConceptByPathhad the same problem. All three now usestd::string.0. Each quoted value followed by,,]or a space also loaded the number0, so["a","b"]loadeda, 0, b, 0. No.kbbshipped with the engine or the analyzers has this shape, and concept-path arrays are unaffected.NLP++ runtime errors now reach stderr
An NLP++ error, such as
findconcept()given a null concept, still stops the rest of its region the way it did before. Until now the only record was the input'serr.log. One line now also goes to stderr:It covers @code, @postcode, @post and @check, and prints once per input for each pass and line. The exit code does not change.
Tests
kbb-long-line,kbb-long-valueandcode-region-error, run by one new step in the Linux, Windows and macOS workflows. The step checksout.txtand the stderr line. Every check in it fails on 4.3.15 and passes on this branch.🤖 Generated with Claude Code