Skip to content

Bound every skill-package read at 8 MiB - #96

Merged
dacharyc merged 1 commit into
mainfrom
fix/issue-87-bounded-reads
Sep 20, 2026
Merged

dacharyc merged 1 commit into
mainfrom
fix/issue-87-bounded-reads

Conversation

@dacharyc

Copy link
Copy Markdown
Member

Fixes #87.

What was wrong

readFileWithCap truncated to 8 MiB only after util.SafeReadFile had already loaded the whole file, so the cap never bounded memory. And the cap only guarded token counting: the unclosed-fence check, the orphan scan (three read sites), and skillcheck's readers loaded reference files whole with no limit at all. Fixing only the token path would have left "bound memory during file reads" untrue of validate structure as a whole.

What changed

  • util.SafeReadFileN(root, path, limit) applies the existing regular-file and symlink-containment checks, then reads through an io.LimitReader. It reads one byte past the limit to report truncation without a second stat, and never more.
  • util.SafeReadFile now refuses files over util.MaxSkillFileBytes (8 MiB) with util.ErrFileTooLarge instead of reading them. Every existing caller is bounded through this change with no call-site edits; only the callers that should say something were touched.
  • Token counts read a bounded prefix. types.TokenCount gains a Truncated flag (also in JSON as "truncated": true), and a warning names the file and the limit. The per-file hard limit still fires on the prefix count, since 8 MiB of anything is far past 25,000 tokens.
  • Unclosed-fence check skips an oversized reference file with a warning rather than checking a partial read.
  • Orphan scan marks an oversized file as unscanned and warns, ahead of the orphan warnings its unseen links may cause, so a false "unreferenced" report has its explanation next to it.
  • SKILL.md over the limit fails to load with "reading SKILL.md: file exceeds the read limit: ... is larger than 8 MiB".
  • skillcheck readers (content and contamination analysis) skip oversized files silently, as they already did on any read error; the structure warnings above cover the same file.
  • README documents the limit under the structure checks.

Verification

Peak resident set on a skill with one 214 MiB reference file:

Build Peak RSS
main 940 MB
this branch 291 MB

The remaining 291 MB is the tokenizer working on the 8 MiB prefix, which is now the ceiling regardless of file size. The same run shows all three new warnings on the file.

Tests: SafeReadFileN truncation and containment, SafeReadFile refusing a sparse file one byte over the limit and accepting one exactly at it, and one test per check for the truncated-count warning, the skipped fence check, and the unscanned-file warning ordering. Sparse fixtures keep the over-limit tests free of disk cost. go test ./... and golangci-lint run are clean.

No CHANGELOG entry, following the release-commit pattern.

readFileWithCap truncated to 8 MiB only after SafeReadFile had loaded
the whole file, so the cap never bounded memory (#87). And the cap only
covered token counting: the markdown, orphan, and skillcheck readers
loaded reference files whole with no limit at all.

Add util.SafeReadFileN, which applies the existing regular-file and
containment checks and then reads through an io.LimitReader, so no more
than the limit ever enters memory. SafeReadFile now refuses files over
util.MaxSkillFileBytes (8 MiB) with ErrFileTooLarge instead of reading
them, which bounds every caller.

Say so where it matters: token counts on a truncated prefix carry a
Truncated flag (also in JSON output) and a warning; the unclosed-fence
check and orphan scan skip oversized files with a warning, because a
partial read would produce false results there. A SKILL.md over the
limit fails to load with a clear error.

On a 214 MiB reference file, peak RSS drops from 940 MB to 291 MB.

Fixes #87
@dacharyc
dacharyc merged commit 7ab1ccc into main Sep 20, 2026
3 checks passed
@dacharyc
dacharyc deleted the fix/issue-87-bounded-reads branch September 20, 2026 01:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bound memory during file reads, not just tokenization (8 MiB cap reads whole file first)

1 participant