Skip to content

@ ci: fail the build on CP949-damaged source - #32

Merged
ken-jo merged 1 commit into
mainfrom
ci/encoding-gate
Aug 23, 2026
Merged

@ ci: fail the build on CP949-damaged source#32
ken-jo merged 1 commit into
mainfrom
ci/encoding-gate

Conversation

@ken-jo

@ken-jo ken-jo commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

lvis-plugin-ep shipped 245 mangled lines for three months. A commit about
deleting old pages was written by a tool that read the repo's UTF-8 Korean
through a CP949 decoder and wrote the result back; nothing noticed, because
nothing in CI reads Korean. Two of those lines were not comments — a room
alias table whose keys no user input could match, and a remote-work check
comparing against a literal that no longer existed.

Restoring the text was one repo's problem. Detecting it is every repo's, so
the check belongs here rather than in a test file each plugin has to grow its
own copy of.

The damage has a fingerprint and the gate looks for exactly that: a UTF-8 byte
pair CP949 can map lands in the CJK ideograph block, one it cannot becomes a
literal '?' welded to the surviving Hangul, and the orphaned third byte of a
syllable passes through as a raw C1 control character. All three are things a
repo that writes Korean and English never produces on purpose.

U+FFFD is deliberately not a signal. This transform never emits it, and the
code that defends against a different decoder's output — ep's and meeting's
includes("�") guards — names the character legitimately. Including it
would have failed two clean repos on their own correctness checks.

Deliberate mojibake does exist: lvis-plugin-git detects broken drafts by
matching what CP949 does to its headings. Those lines carry a
[mojibake-sample] tag, which is greppable, so the exemptions stay visible in
the file that needs them instead of accumulating in a line-number list here
that would rot on the next edit.

Verified against all seven plugin repos at main: clean, 4-55 files each. At
ep's commit before the restoration, 248 lines.
🤖 Generated with Claude Code

https://claude.ai/code/session_01W9K9WE1ATSqYFStZGtLpNk

lvis-plugin-ep shipped 245 mangled lines for three months. A commit about
deleting old pages was written by a tool that read the repo's UTF-8 Korean
through a CP949 decoder and wrote the result back; nothing noticed, because
nothing in CI reads Korean. Two of those lines were not comments — a room
alias table whose keys no user input could match, and a remote-work check
comparing against a literal that no longer existed.

Restoring the text was one repo's problem. Detecting it is every repo's, so
the check belongs here rather than in a test file each plugin has to grow its
own copy of.

The damage has a fingerprint and the gate looks for exactly that: a UTF-8 byte
pair CP949 can map lands in the CJK ideograph block, one it cannot becomes a
literal '?' welded to the surviving Hangul, and the orphaned third byte of a
syllable passes through as a raw C1 control character. All three are things a
repo that writes Korean and English never produces on purpose.

U+FFFD is deliberately not a signal. This transform never emits it, and the
code that defends against a different decoder's output — ep's and meeting's
`includes("�")` guards — names the character legitimately. Including it
would have failed two clean repos on their own correctness checks.

Deliberate mojibake does exist: lvis-plugin-git detects broken drafts by
matching what CP949 does to its headings. Those lines carry a
`[mojibake-sample]` tag, which is greppable, so the exemptions stay visible in
the file that needs them instead of accumulating in a line-number list here
that would rot on the next edit.

Verified against all seven plugin repos at main: clean, 4-55 files each. At
ep's commit before the restoration, 248 lines.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W9K9WE1ATSqYFStZGtLpNk
@ken-jo
ken-jo merged commit 74521c6 into main Aug 23, 2026
1 check passed
@ken-jo
ken-jo deleted the ci/encoding-gate branch August 23, 2026 23:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant