Skip to content

👌 Reject unterminated inline HTML in constant time - #428

Open
chrisjsewell wants to merge 3 commits into
masterfrom
claude/html-inline-quadratic
Open

👌 Reject unterminated inline HTML in constant time#428
chrisjsewell wants to merge 3 commits into
masterfrom
claude/html-inline-quadratic

Conversation

@chrisjsewell

@chrisjsewell chrisjsewell commented Sep 9, 2026

Copy link
Copy Markdown
Member

Summary

Follow-up to #423, covering the remaining quadratic in the same class that #423 deliberately left out because it is a different code path. With html=True (which the commonmark and gfm-like presets enable), a run of unterminated inline HTML openers in inline context is O(n²):

input ("x" + opener × 40 000) before after
<[CDATA[ ~245 s ~2.3 s
<!-- ~71 s ~0.25 s
<? ~30 s ~0.09 s
<!a ~3 s ~0.16 s

All four now grow ~2.0× per doubling. markdown-it (JavaScript) has the same quadratic shape here (with smaller constants), so unlike #423 this is inherited rather than port-specific; it is still worth fixing because the constants in Python are up to 50× worse.

Root cause

Every alternative of HTML_TAG_RE ends in a specific terminator: --> for comments, ?> for processing instructions, ]]> for CDATA, > for declarations and tags. The lazy sub-patterns ([\s\S]*?, [^>]*) rescan to the end of the input on every failed attempt, once per opener.

Fix

  • StateInline.html_terminator_last(term): a lazily-populated per-state cache of src.rfind(term), so each distinct terminator costs one scan per inline parse.
  • html_inline consults it before running the regex and returns False when the terminator the opener needs cannot occur at or after the earliest index it could legally start (e.g. ?> at pos+2 for <??>, ]]> at pos+9 for <[CDATA[]]>, --> at pos+4 unless the opener is one of the short forms <!--> / <!--->).

This is a pure quick-reject: it only skips attempts the regex could never match, so output is unchanged. The regexes themselves are untouched. The check uses src rather than posMax, matching how the existing HTML_TAG_RE.match(state.src, pos) already behaves inside link labels.

Verification

  • Output unchanged: 47,936-case differential (repo fixtures, CommonMark spec, targeted constructs, fuzzed inputs × 7 presets) comparing HTML, renderInline and full token streams: 0 differences. A further 4,730-case HTML-focused corpus (all "Raw HTML" spec examples, random strings over <>!?-[]CDATA/abz \n="', random tag concatenations) and 12,000 posMax-restricted variants (inside links, emphasis, blockquotes, tables, images): 0 differences.
  • Bounds proven, not just tested: an exhaustive check over all strings up to length 6 on the alphabet <>!?-[Aa/ plus 250,000 random strings, 1,040,507 < positions in total, found no position where the quick-reject fired but the regex would have matched.
  • No cost on real documents: spec.md and README.md render at 0.99–1.00× (min of 20 reps, interleaved).
  • Full suite passes; pre-commit (ruff 0.16.6 / mypy 2.3.1) clean; local docs build adds no warnings.

Tests (tests/test_html_inline.py)

Deterministic rather than timed, so they are immune to coverage and PyPy slowdowns (a first version used the global timeout as a guard and timed out on the PyPy job):

  • the shortest legal form of every construct (<a>, </a>, <!-->, <!--->, <!---->, <??>, <!a>, <[CDATA[]]>) still passes through unchanged;
  • for a 1,000-repetition run of each unterminated opener, HTML_TAG_RE.match is asserted to be invoked zero times (before the fix: once per opener) and the output is the escaped text;
  • a run of terminated constructs is asserted to still reach the regex and render as raw HTML.

With `html=True` (the `commonmark` and `gfm-like` presets), a run of
unterminated inline HTML openers in inline context was O(n^2): every
alternative of `HTML_TAG_RE` ends in a specific terminator, and the lazy
sub-patterns for comments (`-->`), processing instructions (`?>`), CDATA
(`]]>`) and declarations (`>`) rescan to the end of the input on each
failed attempt. `"x" + "<![CDATA[" * 40_000` took ~245s, `"<!--"` ~71s,
`"<?"` ~30s. markdown-it (JavaScript) has the same shape, with smaller
constants.

`StateInline` gains `html_terminator_last(term)`, a per-state cache of the
last index of a terminator in `src`, and `html_inline` uses it to bail out
before running the regex when the terminator the opener needs cannot occur
at or after the earliest position it could legally start. This is a pure
quick-reject: it only skips attempts the regex could never match, so
output is unchanged.

All four openers now grow ~2.0x per doubling; CDATA drops from ~245s to
~2.3s at 40k repetitions. Realistic documents are unaffected (spec.md and
README render at 0.99-1.00x).

Verification: a 47,936-case differential (repo fixtures, CommonMark spec,
targeted constructs and fuzzed inputs across seven presets) comparing
HTML, `renderInline` and full token streams shows zero differences; a
further 4,730-case HTML-focused corpus plus 12,000 wrapped (posMax-
restricted) variants also show zero differences; and an exhaustive check
over 1,040,507 `<` positions confirms the quick-reject never rejects a
position the regex would have matched.

Adds `tests/test_html_inline.py`: the shortest legal form of each
construct still passes through, and long unterminated runs of each opener
render within the global test timeout.
The wall-clock guard timed out on PyPy under coverage (no C tracer). The
quick-reject means HTML_TAG_RE is never invoked for an unterminated run,
which can be asserted directly and deterministically; a companion test
checks terminated constructs still reach the regex.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant