Skip to content

perf(metrics/tokens): drop per-leaf ancestor walk (DoS)#1063

Merged
dekobon merged 1 commit into
mainfrom
fix/1052-tokens-ancestor-walk
Jul 25, 2026
Merged

perf(metrics/tokens): drop per-leaf ancestor walk (DoS)#1063
dekobon merged 1 commit into
mainfrom
fix/1052-tokens-ancestor-walk

Conversation

@dekobon

@dekobon dekobon commented Jul 25, 2026

Copy link
Copy Markdown
Owner

Problem

Tokens::compute decided whether a leaf sat inside a comment by walking that
leaf's ancestor chain:

std::iter::successors(Some(*node), Node::parent).any(|n| Self::is_comment(&n))

Node::parent is itself O(depth) — tree-sitter stores no parent pointer, so
ts_node_parent restarts at the root and descends — which made the metric
O(leaves × depth²).

depth input before after tokens only
1000 2.0 KB 19,022 ms 74 ms 4 ms
2000 4.0 KB >120,000 ms (timeout) 285 ms 5 ms
4000 8.0 KB 1,122 ms 6 ms

Parsing the same files stayed flat (bca dump 75 ms, bca ops 9 ms at depth
1000), so this was ours, not tree-sitter's. It made a few kilobytes of legal
nested source an unauthenticated CPU-exhaustion vector against bca-web
whose parse deadline frees the client but cannot cancel the spawn_blocking
task — and a way to stall bca check in CI.

Fix

The walker propagates comment membership down the traversal instead of each
leaf rediscovering it. Walk carries an in_comment flag beside the nesting
level; each node ORs in its own is_comment on arrival and tags its children
with the result. By induction that is exactly the old predicate, so token
counts are unchanged
— including the case that motivated the walk, where a
comment's inner leaves are not themselves comment kinds:

line_comment                 <- the only node `is_comment` matches
├─ //                        <- leaf, not a comment kind
├─ outer_doc_comment_marker
│  ╰─ /                      <- leaf, not a comment kind
╰─ doc_comment               <- leaf, not a comment kind

is_comment is now computed once per node and shared with suppression, which
previously called it separately. push_children became generic over its tag so
ops keeps a plain usize.

Scope: this narrows the DoS, it does not close it

Per-metric isolation at depth 4000 after the fix — tokens is flat, and
cognitive is now the dominant term on this shape:

metric time
cognitive 1080 ms
every other metric ~5 ms

Node::parent is used per-node in three other places, tracked in #1062:

  • cognitive::get_nesting_from_map — one parent() per node, dominates the
    nested-paren shape measured above.
  • Loc::count_specific_ancestors (C-family, Java, C#, Go, Groovy, Objective-C)
    — fires on Declaration nodes, so nested parens leave --metrics sloc
    flat at 4-8 ms; nested declarations are superlinear (35 KB → 153 ms).
  • elixir_is_inside_quote_blockO(depth²) per call, reached from
    promotes_to_func_space_with_code, which the walker calls for every node
    behind no metric selection
    , so it cannot be deselected.

The CHANGELOG says this plainly rather than implying the problem is closed, and
cross-references from ### Security since the entry describes a
remotely-triggerable DoS.

Testing

  • tokens_deep_nesting_is_tractable — restricted to Metric::Tokens so it
    measures the metric it guards (with all metrics enabled, ~97% of its runtime
    was cognitive; the module went 1.23 s → 0.01 s). Asserts the count by formula
    (each paren pair contributes exactly two leaves) and a wall-clock bound,
    so a reintroduced walk fails rather than hanging CI.
  • rust_tokens_comment_excluded_at_depth — anchored (baseline == 111,
    derived as 4 + 2 + 100 + 5) so an in_comment wired to a constant cannot
    pass it.

Test-via-revert per .claude/rules/testing.md, using a precise inverse edit
(the work was uncommitted, so a file-level restore would have wiped it): the
deep-nesting test hangs past 60 s against the old ancestor-walk body and was
killed at 90 s; rust_tokens_comment_excluded_at_depth passes against both,
which is its job as the behaviour-preservation guard.

All 3,067 lib tests pass, token counts unchanged workspace-wide, no snapshot
drift.

Incidental

Threading the flag pushed compute_per_node to 8 arguments, over clippy's
limit. Bundling the three per-node flags into NodeFacts — same-typed booleans
about one node, so this also removes the transposition hazard AGENTS.md warns
about — took it to 6, which retired its nargs baseline entry and lowered
its halstead effort ~7%. metrics_inner grew 12% in halstead effort past its
recorded value; the baseline is refreshed in the same commit per the same-PR
discipline. Net baseline entries 176 → 175.

Node::parent now documents its O(depth) cost at the definition — it was the
only accessor in node.rs without a cost note, while its neighbours gained one
after #217/#521, and it is the one that reads like O(1).

Review

A high-effort review over this branch found 10 issues, all fixed before this
was pushed. Two were real defects introduced by the change itself, both
tending to quietly undo it:

  • Walk::in_comment's doc stated the opposite of the value it carries (it
    excludes the node itself), in wording identical to NodeFacts::in_comment,
    which includes it. Trusting it would make the || is_comment at the read
    site look redundant.
  • apply_comment_suppression took a loose positional bool one line from its
    inverse-meaning twin. Passing the wrong one compiles silently and re-applies
    a suppression marker once per descendant leaf. Fixed by ordering, so the
    wrong value is not yet bound at the call site.

Validation

make pre-commit green — 3,067 lib tests, clippy clean on default and
--all-features, markdown lint, man-page drift, both self-scan tiers.

Fixes #1052

`Tokens::compute` decided whether a leaf sat inside a comment by walking
that leaf's ancestor chain. `Node::parent` is itself O(depth) in
tree-sitter, so the metric cost O(leaves x depth^2): a 2 KB file of
nested parentheses took ~19 s and a 4 KB one over two minutes, while
parsing the same files stayed flat. That made it an unauthenticated CPU
exhaustion vector against `bca-web`, whose parse deadline frees the
client but cannot cancel the blocking task, and a way to stall
`bca check` in CI.

The walker now propagates comment membership down the traversal instead
of rediscovering it per leaf. `Walk` carries an `in_comment` flag
alongside the nesting level; each node ORs in its own `is_comment` on
arrival and tags its children with the result, which by induction is
exactly the old predicate "the node itself or any ancestor is a
comment". Token counts are unchanged, including the case that motivated
the walk: a Rust doc comment's inner leaves (`//`,
`outer_doc_comment_marker`, `doc_comment`) are not comment kinds
themselves and are still excluded.

Measured at nesting depths 1000 / 2000 / 4000, `tokens` now costs
4 / 5 / 6 ms and the same files analyse end-to-end in 74 ms / 285 ms /
1.1 s, against 19 s and a >120 s timeout before.

`is_comment` is now computed once per node and shared with suppression,
which previously called it separately. It is passed to
`apply_comment_suppression` before `subtree_in_comment` is bound, so the
two adjacent same-typed flags cannot be transposed at that call site --
passing the inclusive one would re-apply a suppression marker once per
descendant leaf. `push_children` became generic over its tag so `ops`
keeps a plain `usize`.

Threading the flag pushed `compute_per_node` to 8 arguments, over
clippy's limit. The three per-node flags are bundled into `NodeFacts`
rather than allowed through: they are same-typed booleans about one
node, so a struct also removes the transposition hazard AGENTS.md warns
about. That dropped the function to 6 arguments, which retired its
`nargs` baseline entry and lowered its halstead effort ~7%.
`metrics_inner` grew 12% in halstead effort past its recorded value; the
baseline is refreshed here per the same-PR discipline.

`Node::parent` now documents its O(depth) cost at the definition. It was
the only accessor in `node.rs` without a cost note, while its neighbours
gained one after #217/#521 -- and it is the one that reads like O(1).

Nesting-heavy input is faster but still superlinear overall, so this
narrows rather than closes the exposure: `cognitive`'s nesting-map
lookup dominates the shape measured here, `Loc::count_specific_ancestors`
is superlinear on nested declarations, and Elixir's
`is_inside_quote_block` is O(depth^2) per call behind no metric
selection at all. All tracked in #1062.

Fixes #1052
@dekobon dekobon added bug Something isn't working security Security-relevant finding rust Pull requests that update rust code web bca-web REST API surface labels Jul 25, 2026
@codecov

codecov Bot commented Jul 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.04%. Comparing base (92680fa) to head (2283b51).

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #1063   +/-   ##
=======================================
  Coverage   98.03%   98.04%           
=======================================
  Files         267      267           
  Lines       66021    66073   +52     
  Branches    65591    65643   +52     
=======================================
+ Hits        64726    64778   +52     
  Misses        854      854           
  Partials      441      441           
Flag Coverage Δ
rust 98.02% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/metrics/tokens.rs 99.39% <100.00%> (+0.05%) ⬆️
src/node.rs 95.51% <ø> (ø)
src/spaces/compute.rs 98.44% <100.00%> (+0.11%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@dekobon
dekobon merged commit fe9f504 into main Jul 25, 2026
39 checks passed
@dekobon
dekobon deleted the fix/1052-tokens-ancestor-walk branch July 25, 2026 03:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working rust Pull requests that update rust code security Security-relevant finding web bca-web REST API surface

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(metrics/tokens): O(leaves x depth^2) ancestor walk is a DoS vector

1 participant