👌 Reject unterminated inline HTML in constant time - #428
Open
chrisjsewell wants to merge 3 commits into
Open
Conversation
With `html=True` (the `commonmark` and `gfm-like` presets), a run of unterminated inline HTML openers in inline context was O(n^2): every alternative of `HTML_TAG_RE` ends in a specific terminator, and the lazy sub-patterns for comments (`-->`), processing instructions (`?>`), CDATA (`]]>`) and declarations (`>`) rescan to the end of the input on each failed attempt. `"x" + "<![CDATA[" * 40_000` took ~245s, `"<!--"` ~71s, `"<?"` ~30s. markdown-it (JavaScript) has the same shape, with smaller constants. `StateInline` gains `html_terminator_last(term)`, a per-state cache of the last index of a terminator in `src`, and `html_inline` uses it to bail out before running the regex when the terminator the opener needs cannot occur at or after the earliest position it could legally start. This is a pure quick-reject: it only skips attempts the regex could never match, so output is unchanged. All four openers now grow ~2.0x per doubling; CDATA drops from ~245s to ~2.3s at 40k repetitions. Realistic documents are unaffected (spec.md and README render at 0.99-1.00x). Verification: a 47,936-case differential (repo fixtures, CommonMark spec, targeted constructs and fuzzed inputs across seven presets) comparing HTML, `renderInline` and full token streams shows zero differences; a further 4,730-case HTML-focused corpus plus 12,000 wrapped (posMax- restricted) variants also show zero differences; and an exhaustive check over 1,040,507 `<` positions confirms the quick-reject never rejects a position the regex would have matched. Adds `tests/test_html_inline.py`: the shortest legal form of each construct still passes through, and long unterminated runs of each opener render within the global test timeout.
The wall-clock guard timed out on PyPy under coverage (no C tracer). The quick-reject means HTML_TAG_RE is never invoked for an unterminated run, which can be asserted directly and deterministically; a companion test checks terminated constructs still reach the regex.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #423, covering the remaining quadratic in the same class that #423 deliberately left out because it is a different code path. With
html=True(which thecommonmarkandgfm-likepresets enable), a run of unterminated inline HTML openers in inline context is O(n²):"x" + opener × 40 000)<[CDATA[<!--<?<!aAll four now grow ~2.0× per doubling. markdown-it (JavaScript) has the same quadratic shape here (with smaller constants), so unlike #423 this is inherited rather than port-specific; it is still worth fixing because the constants in Python are up to 50× worse.
Root cause
Every alternative of
HTML_TAG_REends in a specific terminator:-->for comments,?>for processing instructions,]]>for CDATA,>for declarations and tags. The lazy sub-patterns ([\s\S]*?,[^>]*) rescan to the end of the input on every failed attempt, once per opener.Fix
StateInline.html_terminator_last(term): a lazily-populated per-state cache ofsrc.rfind(term), so each distinct terminator costs one scan per inline parse.html_inlineconsults it before running the regex and returnsFalsewhen the terminator the opener needs cannot occur at or after the earliest index it could legally start (e.g.?>atpos+2for<??>,]]>atpos+9for<[CDATA[]]>,-->atpos+4unless the opener is one of the short forms<!-->/<!--->).This is a pure quick-reject: it only skips attempts the regex could never match, so output is unchanged. The regexes themselves are untouched. The check uses
srcrather thanposMax, matching how the existingHTML_TAG_RE.match(state.src, pos)already behaves inside link labels.Verification
renderInlineand full token streams: 0 differences. A further 4,730-case HTML-focused corpus (all "Raw HTML" spec examples, random strings over<>!?-[]CDATA/abz \n="', random tag concatenations) and 12,000 posMax-restricted variants (inside links, emphasis, blockquotes, tables, images): 0 differences.<>!?-[Aa/plus 250,000 random strings, 1,040,507<positions in total, found no position where the quick-reject fired but the regex would have matched.spec.mdandREADME.mdrender at 0.99–1.00× (min of 20 reps, interleaved).Tests (
tests/test_html_inline.py)Deterministic rather than timed, so they are immune to coverage and PyPy slowdowns (a first version used the global timeout as a guard and timed out on the PyPy job):
<a>,</a>,<!-->,<!--->,<!---->,<??>,<!a>,<[CDATA[]]>) still passes through unchanged;HTML_TAG_RE.matchis asserted to be invoked zero times (before the fix: once per opener) and the output is the escaped text;