Bound every skill-package read at 8 MiB - #96
Merged
Merged
Conversation
readFileWithCap truncated to 8 MiB only after SafeReadFile had loaded the whole file, so the cap never bounded memory (#87). And the cap only covered token counting: the markdown, orphan, and skillcheck readers loaded reference files whole with no limit at all. Add util.SafeReadFileN, which applies the existing regular-file and containment checks and then reads through an io.LimitReader, so no more than the limit ever enters memory. SafeReadFile now refuses files over util.MaxSkillFileBytes (8 MiB) with ErrFileTooLarge instead of reading them, which bounds every caller. Say so where it matters: token counts on a truncated prefix carry a Truncated flag (also in JSON output) and a warning; the unclosed-fence check and orphan scan skip oversized files with a warning, because a partial read would produce false results there. A SKILL.md over the limit fails to load with a clear error. On a 214 MiB reference file, peak RSS drops from 940 MB to 291 MB. Fixes #87
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #87.
What was wrong
readFileWithCaptruncated to 8 MiB only afterutil.SafeReadFilehad already loaded the whole file, so the cap never bounded memory. And the cap only guarded token counting: the unclosed-fence check, the orphan scan (three read sites), and skillcheck's readers loaded reference files whole with no limit at all. Fixing only the token path would have left "bound memory during file reads" untrue ofvalidate structureas a whole.What changed
util.SafeReadFileN(root, path, limit)applies the existing regular-file and symlink-containment checks, then reads through anio.LimitReader. It reads one byte past the limit to report truncation without a second stat, and never more.util.SafeReadFilenow refuses files overutil.MaxSkillFileBytes(8 MiB) withutil.ErrFileTooLargeinstead of reading them. Every existing caller is bounded through this change with no call-site edits; only the callers that should say something were touched.types.TokenCountgains aTruncatedflag (also in JSON as"truncated": true), and a warning names the file and the limit. The per-file hard limit still fires on the prefix count, since 8 MiB of anything is far past 25,000 tokens.Verification
Peak resident set on a skill with one 214 MiB reference file:
The remaining 291 MB is the tokenizer working on the 8 MiB prefix, which is now the ceiling regardless of file size. The same run shows all three new warnings on the file.
Tests:
SafeReadFileNtruncation and containment,SafeReadFilerefusing a sparse file one byte over the limit and accepting one exactly at it, and one test per check for the truncated-count warning, the skipped fence check, and the unscanned-file warning ordering. Sparse fixtures keep the over-limit tests free of disk cost.go test ./...andgolangci-lint runare clean.No CHANGELOG entry, following the release-commit pattern.