Keep a comment that starts a line from hiding its formatting

[kylemcd]

Fixes #94.

Markdown reads a line whose text starts with <!-- as a raw HTML block, so Reading view shows the whole line as written. A comment that starts a paragraph puts its <!--c:ID--> marker exactly there. Every ==highlight==, **bold** and link on that line then comes out as plain text. Live Preview has its own parser, which is why editing looked fine. Rendering through Obsidian's own renderer shows the same for:

• the text of a list item, task, quote, callout title or body, or footnote;
• a soft-wrapped line of a paragraph, which also split the paragraph into three blocks;
• the header row of a table without outer pipes, which stopped rendering as a table.

Headings, table body rows and markers mid-line were fine. It has worked this way since the first release.

A second bug has the same cause. Obsidian's triple-click selects a line from its very start, so commenting on a whole list item, quote or heading put the marker in front of the - , > or ##. The line then stopped being a list item, quote or heading, in Live Preview as well as Reading view.

Before: a note written by 0.1.16, opened on this branch. Each card offers the new Repair.

After: the same note with a zero-width space in front of each marker, as new comments and Repair now write it.

What changes

• A zero-width space in front of the marker. When a new comment's text starts a line, the plugin writes a U+200B before <!--c:ID-->. The line reads as ordinary text again, and the marker as an inline comment. It renders as nothing in Obsidian, GitHub and other Markdown tools. Headings, table body rows and a marker alone on its line don't need one and don't get one.
• Markers stay off block markup. A selection that starts in front of a bullet, task box, >, callout [!type], footnote label or heading #s now anchors after them. A selection that ends at the start of the next line, or just after its bullet, ends on the text it covers. It isn't pulled back onto the end of a fence, rule or table row, where it would break that line. In that case it goes after the next line's markup and takes the zero-width space instead. A marker on a line of nothing but whitespace goes at the line's start, where it stays invisible.
• The editor treats the zero-width space and its marker as one. It hides them together, one arrow press crosses both, and Backspace or Delete can't remove the space on its own. Deleting a comment removes its space, unless another comment's marker still needs it.
• Existing notes are left alone until you ask.
  • A comment whose markers break their line says "This comment is breaking its line's formatting." on its card, with a one-click Repair.
  • Repair adds the zero-width space, or moves a marker from in front of the markup to after it.
  • Reading view cards now offer Repair too, for tables as well as lines, since that's where a broken line shows.
  • Repair comments in this note replaces Repair table comments in this note. It repairs both kinds in one write, runs from Reading view, and keeps the same command id, so hotkeys keep working.
  • Moving a closing marker back takes two edits. When a table repair touches one of them, the command repairs the table and asks you to run it again rather than make only one.
• The agent skill documents the zero-width space, and validate_comments.py flags a marker that starts a line without one, or that comes right after a backslash.

Also fixed here

These are the same kind of problem. I found them by rendering thousands of random comments with markdown-it, then checked each shape in Obsidian.

• Rules, setext underlines and blank lines. A selection that started or ended on ---, ===, an empty bullet or a blank line put a marker on that line. Anywhere on a rule or underline, a marker stops it rendering as one. These selections now anchor on the text beside them. A selection of nothing but a rule is refused.

  • The move stops at a line that opens a $$ math block, a %% comment block or an HTML block. A marker in front of one of those breaks the block, so a start on a blank line above one stays on the blank line, where it's invisible.

• Code blocks.

  • A comment on a paragraph followed directly by a code block wrote its thread inside the block whenever the code had a blank line. The thread then showed as code. It now goes after the closing fence.
  • A code block with no closing fence now refuses a comment instead of filling up with comment text.

• Reading view highlights. These cases had no highlight in Reading view:

  • a comment whose text carried formatting (==hl==, bold, links, inline code);
  • one that ran across a line break or into the next paragraph;
  • one in a syntax-highlighted code block;
  • one in a footnote.

  A repeated phrase also highlighted its first occurrence instead of the comment's. The highlight now finds the comment's rendered text and wraps it across elements. Obsidian rebuilds a syntax-highlighted block after the plugin has wrapped it, so a code comment wraps again once that happens.

  A comment over three or more blocks highlights its first and last. Obsidian keeps any rendered block whose source didn't change, so a highlighted middle block would keep its highlight after the comment was deleted.

  Obsidian renders all of a note's footnotes in one section of their own, after the note's last line, so the plugin found no comments there. Each footnote now maps back to its definition, which holds its comments. Add comment in reading view works on a footnote's text for the same reason. It used to say it couldn't map the selection.

• Highlights stay in sync with the note. Each Reading view refresh removes highlights whose comment is gone, and updates resolved status and preview on the rest. That covers a block Obsidian kept from before the change, which used to keep a deleted code comment's highlight, for example. The sync checks against the text the pane shows, not the file. A Reading pane beside an editing pane shows the editor's unsaved text, so a just-added comment would otherwise lose its highlight.

• Repair also moves a marker off a rule or underline it broke.

• Lines ending in a backslash. A backslash at the end of a line is a hard line break, and \< is an escape. A selection that stopped there, as Obsidian's triple-click does, or a closing marker pulled back there, put the marker right after the backslash. Reading view then showed the marker as text and dropped the line break. A marker now goes in front of the backslash, and Repair moves one that an older version wrote after it.

• Indented code. A comment on indented code put its markers into the code, where they showed as text. It now takes in the code block's whole lines.

  • For a top-level block with a blank line above it and room below, the markers go on those lines, and the code still renders as code. Reading view highlights it and shows its card.
  • Inside a list item, the markers stay on the code's own line, at the item's text column. That line shows as plain text, as it did on main, but the list around it doesn't change.
  • Telling indented code from a list item's or a footnote's indented text follows Obsidian's own rendering, which I checked in the app. It differs from CommonMark around comment lines, quotes, footnotes, raw HTML blocks and ordered lists.

• Raw HTML blocks. A marker in front of a block's opening tag (<details>, <div>, </details>) turned the line into a comment block. The lines after it then rendered as something else, and the zero-width space would have turned it into a paragraph. A marker now goes after the opening tag, and Repair moves one that an older version wrote in front of it. A selection of nothing but a lone tag is refused.

• Deleting a comment gives back the note exactly. Deleting a comment used to remove a line that held only its marker, even when that line was blank before the comment. That could join a code block to the paragraph next to it. Now only a code comment, which adds lines of its own around its block, takes them out again.

Testing

• npm run check passes (602 tests).

  • New tests cover each block kind, adding comments, deleting them back to the original text, repair, how the editor handles the zero-width space, and footnotes in Reading view.
  • With the editor's and creation's handling of the space switched off, 12 of the new tests fail.

• One expectation from Keep a marker outside a table where the selection put it #93 changes. A selection ending just after the bullet of a list item below a table still leaves its closing marker there, since pulling it back would put it after the table's closing pipe. It now gets the zero-width space, so the item keeps its formatting.

• In Obsidian, with this branch's build in the test vault:

  • Rendered each case above with and without the zero-width space.
  • Added comments through the composer on a paragraph's first word and on a triple-clicked list item. Both wrote the space after any markup.
  • Repaired 0.1.16-style comments from a Live Preview card and from a Reading view card.
  • Ran the command from Reading view. It repaired two lines and a table in one write.
  • The screenshots above show the space leaves nothing visible in either view.

• Reviews: a fresh agent with no context reviewed each commit, with reproductions and randomized tests.

  • The first pass found three bugs. The second commit fixes each one, with a regression test:

    • the whole-note repair could delete a closing marker when a table repair touched one of the two edits that move it;
    • a closing marker could still land in front of a bullet when a selection ended right after a code fence or --- rule;
    • on files with Windows line endings, --- didn't count as a rule, so the repair could add a marker to its end.

  • The second pass confirmed those fixes.

    • Across about 52,000 damaged notes, repeating the whole-note repair always finished within 4 runs.
    • Across about 66,000 random new comments, the repair flagged none.
    • It found that the second commit put a marker on a whitespace-only line after its indentation. Four or more columns of indentation make that a code block. The third commit fixes it.

  • The third pass confirmed that fix and found nothing new. Across 31,600 random comment creations rendered with markdown-it, this branch breaks rendering in fewer cases than main (8,230 against 24,174), and in none that main renders correctly.

  • A second fresh agent reviewed the "Also fixed here" commit and found two problems:

    • a start on a blank line could land in front of a $$ or %% block below it, which then showed as text;
    • the middle block of a comment over three blocks kept its highlight after the comment was deleted.

    Rechecking each fix, it found three more:

    • the HTML-block stop also caught inline HTML such as <b>Note:</b>;
    • a highlight could outlive a comment over two blocks;
    • syncing against the file removed a new comment's highlight in a Reading pane next to an unsaved editor.

    Each fix came with a test that failed first. Its last pass found nothing new.

  • Five more fresh agents then reviewed the whole branch, one after each round of fixes. They found 15 more problems, most of them in the handling of backslashes, indented code and raw HTML blocks above. Each fix came with a test that failed first. The last full review found one: running Add comment again on the selection that made an empty comment missed it and wrote a second comment over it. The next commit fixes that. A fresh review of that fix found it made Add comment slow on a select-all in a note with hundreds of empty comments: about a second per lookup, blocking the editor. The commit after brings it back to milliseconds. Its own review found that with three or more empty comments around the selection, the speed-up skipped the comment it had just made. The last commit fixes that: a lookup takes 16 ms, against 711 ms, with 250 empty comments in a 47 KB note, and the reviewer's fuzz of nested highlights finds no misses in 2,228 cases, against 440 before.

  • A fresh agent reviewed the footnote fix and found no bugs. Of its side notes, one was worth fixing here: a selected word that a footnote's label shares mapped into the label, so Add comment refused it. Another, a line right after an indented fence or heading in a footnote, renders inside the footnote in Obsidian, as the fix assumes. The third, a thread written between a footnote's paragraphs, is older and listed under "Known".

• Rendering check: the same markdown-it harness ran over the same kinds of random notes and selections. Where a new comment's markers land now breaks the rendering in 0 of 23,102 comments, against 8,230 before this round and 24,174 on main.

  • Some selections are now refused instead of written: those in code blocks with no closing fence, those holding nothing but a rule or a lone HTML tag, and those of indented code that leave no hidden place for a marker, such as code right before a rule.
  • In about 600 cases a comment's thread still sits between items of a loose list. markdown-it splits the list there, but Obsidian keeps it whole, which I checked in the app.

• A second harness adds hard line breaks, indented code, footnotes, tables and raw HTML blocks to the random notes, about 40,000 comments in all. It renders every case where markdown-it shows a difference in Obsidian itself, through the running app.

  • Judged by Obsidian, a new comment never renders worse than before this round. Against main, only the one rare layout under "Known" differs.
  • Deleting a comment gives back the original note in every case. On main, 9,464 deletions didn't.
  • Running Add comment again on the same selection always finds the empty comment it made.
  • Repair never loses an anchor, and no new comment is flagged as damaged, apart from 4 selections made inside an existing HTML comment.

• In Obsidian: I checked the note in the screenshot above. It covers:

  • formatted text and a repeated word;
  • a comment over two paragraphs;
  • a list item and a syntax-highlighted code line;
  • a marker after --- from an older version, repaired from its Reading view card;
  • deleting one paragraph of a comment over two, which cleared the other paragraph's highlight;
  • adding a comment in an editing pane beside a Reading pane, where the Reading pane kept the highlight before and after the autosave.

• In Obsidian, for this round: I added comments with the app's real triple-click on a line ending in a backslash, on top-level indented code and on indented code in a list item. I also added a code comment over a fenced block holding an HTML comment line. In Reading view:

  • the line break stayed;
  • both code blocks rendered intact;
  • each comment was highlighted and its card shown;
  • the list kept the paragraph after it separate;
  • no marker showed as text.

• In Obsidian, for footnotes: I made a note with five comments across four footnotes: mid-line, guarded at a footnote's start, over ==formatted== text, in a footnote's second paragraph, and in the footnote defined last. In Reading view each comment was highlighted in its own footnote, with its card, as in the screenshot above. That held:

  • with frontmatter, and in a note that ends without a line break;
  • with two footnotes holding the same word;
  • after edits that added lines above the footnotes and between them.

  Add comment in reading view on a footnote's first word wrote the marker into its definition, behind a zero-width space, and its highlight and card appeared. Deleting it from its card gave back the note byte for byte. On a word that the footnote's label also holds, it commented on the footnote's text, not the label.

• Known, not fixed here:

  • A comment inside an inline footnote (^[…]) has no highlight in Reading view, as on main.
  • A comment on the first paragraph of a footnote with several paragraphs writes its thread between them. That ends the footnote, so the paragraphs after it show as code, as on main.
  • Adding a comment in Reading view still needs plain text. A selection across formatting can't be mapped back to the Markdown.
  • In a loose list, a comment's thread goes between items. Obsidian keeps the list whole, but GitHub and other CommonMark tools split it there.
  • A thread that an older version wrote inside a code block stays there. The parser skips everything inside code, so Repair can't see it either.
  • A comment on indented code inside a list item, or with no blank line above the block, keeps its markers on a code line. That line shows as plain text rather than code, as it did on main.
  • One rare layout doesn't follow Obsidian's usual rules: an ordered list item, then an HTML comment line, then a line indented six spaces. A comment there can show its closing marker as code. Main hid the marker but showed the line as plain text.
  • A selection made inside an existing HTML comment or inside a tag's own text writes markers that break it, as on main. Repair can't fix comments that an older version wrote inside another HTML comment.