Repository navigation
Conversation
CommonMark's flanking rule leaves `**` literal when punctuation sits inside and a CJK letter outside, which is common in Chinese text. A zero-width mark inserted before parsing makes the delimiter valid; it is removed from the output and alignment still uses the original source. Replace the syntax tui-markdown keeps as text after alignment: rules, quote markers, bullets, task boxes and code fences. Replaced markers map to no source byte, so annotations stay anchored. Break CJK runs per character with basic kinsoku, keep Latin words whole, and give wrapped list and quote lines a hanging indent. Claude-Session: https://claude.ai/code/session_01XiN1EeethwfQuCmfBZXKZ2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Chinese Markdown comes out with raw syntax on screen, mostly because of one CommonMark rule. This PR renders it properly and wraps CJK text cleanly.
What was wrong
改造成**“最强妖怪”**。shows the**. CommonMark refuses a delimiter with punctuation on its inner side and a letter on its outer side. Chinese has no spaces to satisfy that, and the leftover**can then pair with the wrong partner further along the line.tui-markdownkeeps---,>,-,[ ]and code fences as text.-, leaving the marker alone on a row.What changed
cjk_emphasis.rs: puts a U+200B on the inner side of a refused delimiter before parsing, then strips it from the rendered text.tui-markdownruns its own parser and has no option for this, so the repair has to happen in its input. Alignment still runs against the original source, so offsets don't move. Delimiters in code, math and raw HTML are left alone, and text without a repairable CJK delimiter passes through unchanged.decorate.rs: runs after alignment and replaces the leftover syntax:>becomes│, bullets become•,◦or▪by depth, task boxes become☐or☑, and code fences are removed. Replaced markers map to no source byte, so annotations stay anchored.wrap.rs: breaks CJK per character with basic kinsoku, keeps Latin words whole, draws rules at full width, and gives wrapped list and quote lines a hanging indent.Results
On a 7 KB Chinese reply at 80 columns, comparing rows on screen:
**------bench samples/big.md, three runs each:Testing
cargo fmt --all --check,cargo clippy --workspace --all-targets -- -D warnings,cargo test --workspace(470 passed, 20 of them new),python3 herdr/test-manifest.pyand the release build.--snapshot <file> ... <select-quote>, selecting text inside repaired bold, and text after it in a wrapped list item, covers the expected number of characters, the same as the build before this change.•,◦,▪,☐,☑,│and─are East Asian Ambiguous width, so a terminal that draws them wide could misalign wrapped lines.For review
AGENTS.mdsays "we never interpret markdown ourselves".decorate.rsdoesn't parse Markdown, but it does read leading characters of the renderer's output, usingBlockKind, which is close to that line. The cleaner home would betui-markdownitself. I'm happy to rework this if you'd rather go that way.wrap.rsgrows from 208 to 384 lines, tests included, which is over the ~300 line signal. I can split it if you prefer.protectdoesn't exclude*inside link destinations or titles.strip_marksalso removes any U+200B that was already in the source.$math$and inline HTML. A paragraph right after a definition list is also shown twice, as it was before this PR.https://claude.ai/code/session_01XiN1EeethwfQuCmfBZXKZ2