Skip to content

fix(render): render CJK emphasis and leftover Markdown syntax - #135

Open
Manho wants to merge 1 commit into
plannotator:mainfrom
Manho:fix/cjk-markdown-rendering
Open

Manho wants to merge 1 commit into
plannotator:mainfrom
Manho:fix/cjk-markdown-rendering

Conversation

@Manho

@Manho Manho commented Nov 5, 2026

Copy link
Copy Markdown

Chinese Markdown comes out with raw syntax on screen, mostly because of one CommonMark rule. This PR renders it properly and wraps CJK text cleanly.

What was wrong

  • Bold stays literal. 改造成**“最强妖怪”**。 shows the **. CommonMark refuses a delimiter with punctuation on its inner side and a letter on its outer side. Chinese has no spaces to satisfy that, and the leftover ** can then pair with the wrong partner further along the line.
  • Syntax stays on screen. tui-markdown keeps ---, >, -, [ ] and code fences as text.
  • CJK wraps badly. A whole CJK run counted as one unbreakable token, so a long list item wrapped straight after its -, leaving the marker alone on a row.

What changed

  • cjk_emphasis.rs: puts a U+200B on the inner side of a refused delimiter before parsing, then strips it from the rendered text. tui-markdown runs its own parser and has no option for this, so the repair has to happen in its input. Alignment still runs against the original source, so offsets don't move. Delimiters in code, math and raw HTML are left alone, and text without a repairable CJK delimiter passes through unchanged.
  • decorate.rs: runs after alignment and replaces the leftover syntax: > becomes │, bullets become •, ◦ or ▪ by depth, task boxes become ☐ or ☑, and code fences are removed. Replaced markers map to no source byte, so annotations stay anchored.
  • wrap.rs: breaks CJK per character with basic kinsoku, keeps Latin words whole, draws rules at full width, and gives wrapped list and quote lines a hanging indent.

Results

On a 7 KB Chinese reply at 80 columns, comparing rows on screen:

Before After
Rows with literal ** 5 0
Rows with only --- 5 0
Rows with only - 20 0
Non-empty rows 148 124

--bench samples/big.md, three runs each:

Before After
render+align+wrap 417-424 ms 444-448 ms (about 6% slower)
reflow 85-93 ms 84-102 ms (overlapping, noisy)

Testing

  • Passed, on macOS with rustc 1.99.1: cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace (470 passed, 20 of them new), python3 herdr/test-manifest.py and the release build.
  • Passed by hand: opened a real reply in a Herdr pane and read the rendered screen. With --snapshot <file> ... <select-quote>, selecting text inside repaired bold, and text after it in a wrapped list item, covers the expected number of characters, the same as the build before this change.
  • Not run: Linux and Windows.
  • Not checked: how terminals draw the new glyphs. •, ◦, ▪, ☐, ☑, │ and ─ are East Asian Ambiguous width, so a terminal that draws them wide could misalign wrapped lines.

For review

  • AGENTS.md says "we never interpret markdown ourselves". decorate.rs doesn't parse Markdown, but it does read leading characters of the renderer's output, using BlockKind, which is close to that line. The cleaner home would be tui-markdown itself. I'm happy to rework this if you'd rather go that way.
  • wrap.rs grows from 208 to 384 lines, tests included, which is over the ~300 line signal. I can split it if you prefer.
  • Known limits:
    • protect doesn't exclude * inside link destinations or titles.
    • strip_marks also removes any U+200B that was already in the source.
    • Code inside a list item loses its indent, and a blank row can appear before it.
  • Still not rendered, unchanged by this PR: footnotes, definition lists, $math$ and inline HTML. A paragraph right after a definition list is also shown twice, as it was before this PR.

https://claude.ai/code/session_01XiN1EeethwfQuCmfBZXKZ2

CommonMark's flanking rule leaves `**` literal when punctuation sits
inside and a CJK letter outside, which is common in Chinese text. A
zero-width mark inserted before parsing makes the delimiter valid; it is
removed from the output and alignment still uses the original source.

Replace the syntax tui-markdown keeps as text after alignment: rules,
quote markers, bullets, task boxes and code fences. Replaced markers map
to no source byte, so annotations stay anchored.

Break CJK runs per character with basic kinsoku, keep Latin words whole,
and give wrapped list and quote lines a hanging indent.

Claude-Session: https://claude.ai/code/session_01XiN1EeethwfQuCmfBZXKZ2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant