perf(metrics/tokens): drop per-leaf ancestor walk (DoS)#1063
Merged
Conversation
`Tokens::compute` decided whether a leaf sat inside a comment by walking that leaf's ancestor chain. `Node::parent` is itself O(depth) in tree-sitter, so the metric cost O(leaves x depth^2): a 2 KB file of nested parentheses took ~19 s and a 4 KB one over two minutes, while parsing the same files stayed flat. That made it an unauthenticated CPU exhaustion vector against `bca-web`, whose parse deadline frees the client but cannot cancel the blocking task, and a way to stall `bca check` in CI. The walker now propagates comment membership down the traversal instead of rediscovering it per leaf. `Walk` carries an `in_comment` flag alongside the nesting level; each node ORs in its own `is_comment` on arrival and tags its children with the result, which by induction is exactly the old predicate "the node itself or any ancestor is a comment". Token counts are unchanged, including the case that motivated the walk: a Rust doc comment's inner leaves (`//`, `outer_doc_comment_marker`, `doc_comment`) are not comment kinds themselves and are still excluded. Measured at nesting depths 1000 / 2000 / 4000, `tokens` now costs 4 / 5 / 6 ms and the same files analyse end-to-end in 74 ms / 285 ms / 1.1 s, against 19 s and a >120 s timeout before. `is_comment` is now computed once per node and shared with suppression, which previously called it separately. It is passed to `apply_comment_suppression` before `subtree_in_comment` is bound, so the two adjacent same-typed flags cannot be transposed at that call site -- passing the inclusive one would re-apply a suppression marker once per descendant leaf. `push_children` became generic over its tag so `ops` keeps a plain `usize`. Threading the flag pushed `compute_per_node` to 8 arguments, over clippy's limit. The three per-node flags are bundled into `NodeFacts` rather than allowed through: they are same-typed booleans about one node, so a struct also removes the transposition hazard AGENTS.md warns about. That dropped the function to 6 arguments, which retired its `nargs` baseline entry and lowered its halstead effort ~7%. `metrics_inner` grew 12% in halstead effort past its recorded value; the baseline is refreshed here per the same-PR discipline. `Node::parent` now documents its O(depth) cost at the definition. It was the only accessor in `node.rs` without a cost note, while its neighbours gained one after #217/#521 -- and it is the one that reads like O(1). Nesting-heavy input is faster but still superlinear overall, so this narrows rather than closes the exposure: `cognitive`'s nesting-map lookup dominates the shape measured here, `Loc::count_specific_ancestors` is superlinear on nested declarations, and Elixir's `is_inside_quote_block` is O(depth^2) per call behind no metric selection at all. All tracked in #1062. Fixes #1052
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1063 +/- ##
=======================================
Coverage 98.03% 98.04%
=======================================
Files 267 267
Lines 66021 66073 +52
Branches 65591 65643 +52
=======================================
+ Hits 64726 64778 +52
Misses 854 854
Partials 441 441
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Tokens::computedecided whether a leaf sat inside a comment by walking thatleaf's ancestor chain:
Node::parentis itselfO(depth)— tree-sitter stores no parent pointer, sots_node_parentrestarts at the root and descends — which made the metricO(leaves × depth²).Parsing the same files stayed flat (
bca dump75 ms,bca ops9 ms at depth1000), so this was ours, not tree-sitter's. It made a few kilobytes of legal
nested source an unauthenticated CPU-exhaustion vector against
bca-web—whose parse deadline frees the client but cannot cancel the
spawn_blockingtask — and a way to stall
bca checkin CI.Fix
The walker propagates comment membership down the traversal instead of each
leaf rediscovering it.
Walkcarries anin_commentflag beside the nestinglevel; each node ORs in its own
is_commenton arrival and tags its childrenwith the result. By induction that is exactly the old predicate, so token
counts are unchanged — including the case that motivated the walk, where a
comment's inner leaves are not themselves comment kinds:
is_commentis now computed once per node and shared with suppression, whichpreviously called it separately.
push_childrenbecame generic over its tag soopskeeps a plainusize.Scope: this narrows the DoS, it does not close it
Per-metric isolation at depth 4000 after the fix —
tokensis flat, andcognitiveis now the dominant term on this shape:Node::parentis used per-node in three other places, tracked in #1062:cognitive::get_nesting_from_map— oneparent()per node, dominates thenested-paren shape measured above.
Loc::count_specific_ancestors(C-family, Java, C#, Go, Groovy, Objective-C)— fires on
Declarationnodes, so nested parens leave--metrics slocflat at 4-8 ms; nested declarations are superlinear (35 KB → 153 ms).
elixir_is_inside_quote_block—O(depth²)per call, reached frompromotes_to_func_space_with_code, which the walker calls for every nodebehind no metric selection, so it cannot be deselected.
The CHANGELOG says this plainly rather than implying the problem is closed, and
cross-references from
### Securitysince the entry describes aremotely-triggerable DoS.
Testing
tokens_deep_nesting_is_tractable— restricted toMetric::Tokensso itmeasures the metric it guards (with all metrics enabled, ~97% of its runtime
was cognitive; the module went 1.23 s → 0.01 s). Asserts the count by formula
(each paren pair contributes exactly two leaves) and a wall-clock bound,
so a reintroduced walk fails rather than hanging CI.
rust_tokens_comment_excluded_at_depth— anchored (baseline == 111,derived as 4 + 2 + 100 + 5) so an
in_commentwired to a constant cannotpass it.
Test-via-revert per
.claude/rules/testing.md, using a precise inverse edit(the work was uncommitted, so a file-level restore would have wiped it): the
deep-nesting test hangs past 60 s against the old ancestor-walk body and was
killed at 90 s;
rust_tokens_comment_excluded_at_depthpasses against both,which is its job as the behaviour-preservation guard.
All 3,067 lib tests pass, token counts unchanged workspace-wide, no snapshot
drift.
Incidental
Threading the flag pushed
compute_per_nodeto 8 arguments, over clippy'slimit. Bundling the three per-node flags into
NodeFacts— same-typed booleansabout one node, so this also removes the transposition hazard AGENTS.md warns
about — took it to 6, which retired its
nargsbaseline entry and loweredits halstead effort ~7%.
metrics_innergrew 12% in halstead effort past itsrecorded value; the baseline is refreshed in the same commit per the same-PR
discipline. Net baseline entries 176 → 175.
Node::parentnow documents itsO(depth)cost at the definition — it was theonly accessor in
node.rswithout a cost note, while its neighbours gained oneafter #217/#521, and it is the one that reads like
O(1).Review
A high-effort review over this branch found 10 issues, all fixed before this
was pushed. Two were real defects introduced by the change itself, both
tending to quietly undo it:
Walk::in_comment's doc stated the opposite of the value it carries (itexcludes the node itself), in wording identical to
NodeFacts::in_comment,which includes it. Trusting it would make the
|| is_commentat the readsite look redundant.
apply_comment_suppressiontook a loose positionalboolone line from itsinverse-meaning twin. Passing the wrong one compiles silently and re-applies
a suppression marker once per descendant leaf. Fixed by ordering, so the
wrong value is not yet bound at the call site.
Validation
make pre-commitgreen — 3,067 lib tests, clippy clean on default and--all-features, markdown lint, man-page drift, both self-scan tiers.Fixes #1052