@ ci: fail the build on CP949-damaged source - #32
Merged
Merged
Conversation
lvis-plugin-ep shipped 245 mangled lines for three months. A commit about
deleting old pages was written by a tool that read the repo's UTF-8 Korean
through a CP949 decoder and wrote the result back; nothing noticed, because
nothing in CI reads Korean. Two of those lines were not comments — a room
alias table whose keys no user input could match, and a remote-work check
comparing against a literal that no longer existed.
Restoring the text was one repo's problem. Detecting it is every repo's, so
the check belongs here rather than in a test file each plugin has to grow its
own copy of.
The damage has a fingerprint and the gate looks for exactly that: a UTF-8 byte
pair CP949 can map lands in the CJK ideograph block, one it cannot becomes a
literal '?' welded to the surviving Hangul, and the orphaned third byte of a
syllable passes through as a raw C1 control character. All three are things a
repo that writes Korean and English never produces on purpose.
U+FFFD is deliberately not a signal. This transform never emits it, and the
code that defends against a different decoder's output — ep's and meeting's
`includes("�")` guards — names the character legitimately. Including it
would have failed two clean repos on their own correctness checks.
Deliberate mojibake does exist: lvis-plugin-git detects broken drafts by
matching what CP949 does to its headings. Those lines carry a
`[mojibake-sample]` tag, which is greppable, so the exemptions stay visible in
the file that needs them instead of accumulating in a line-number list here
that would rot on the next edit.
Verified against all seven plugin repos at main: clean, 4-55 files each. At
ep's commit before the restoration, 248 lines.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W9K9WE1ATSqYFStZGtLpNk
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
lvis-plugin-ep shipped 245 mangled lines for three months. A commit about
deleting old pages was written by a tool that read the repo's UTF-8 Korean
through a CP949 decoder and wrote the result back; nothing noticed, because
nothing in CI reads Korean. Two of those lines were not comments — a room
alias table whose keys no user input could match, and a remote-work check
comparing against a literal that no longer existed.
Restoring the text was one repo's problem. Detecting it is every repo's, so
the check belongs here rather than in a test file each plugin has to grow its
own copy of.
The damage has a fingerprint and the gate looks for exactly that: a UTF-8 byte
pair CP949 can map lands in the CJK ideograph block, one it cannot becomes a
literal '?' welded to the surviving Hangul, and the orphaned third byte of a
syllable passes through as a raw C1 control character. All three are things a
repo that writes Korean and English never produces on purpose.
U+FFFD is deliberately not a signal. This transform never emits it, and the
code that defends against a different decoder's output — ep's and meeting's
includes("�")guards — names the character legitimately. Including itwould have failed two clean repos on their own correctness checks.
Deliberate mojibake does exist: lvis-plugin-git detects broken drafts by
matching what CP949 does to its headings. Those lines carry a
[mojibake-sample]tag, which is greppable, so the exemptions stay visible inthe file that needs them instead of accumulating in a line-number list here
that would rot on the next edit.
Verified against all seven plugin repos at main: clean, 4-55 files each. At
ep's commit before the restoration, 248 lines.
🤖 Generated with Claude Code
https://claude.ai/code/session_01W9K9WE1ATSqYFStZGtLpNk