[ethanzhoucool]

Summary

When pasted Python has a syntax error, the adapter salvages the longest prefix that still parses ("Broken code still explains" in the README). _parse_longest_prefix did that by dropping one line per attempt and calling ast.parse again. An error near the top of a large file therefore re-parsed it about once per line, so the work grew with the square of the file length.

Measured locally with Python 3.13, explain() end to end:

input	before	after
197 KB generated Python, unclosed ( on line 1	6.98 s, all in parse	0.004 s
195 KB of stdlib modules, broken on line 1	2.95 s	0.005 s
same files, no error	0.33 s	0.33 s (no change)

/v1/explain/all-levels, the endpoint the web UI calls, runs explain() three times, so a broken 200 KB paste used to cost about 20 s of CPU on the single worker.

Changes

• adapters/python.py: a prefix that still contains the line the SyntaxError was reported on fails the same way. So each failed attempt now jumps to just above the reported line instead of retrying one line shorter. The first attempt starts from the original error's line, which saves a parse. When the error is reported past the end of the prefix (e.g. "expected an indented block" at EOF), it falls back to dropping one line.
• ir.py: Node.walk is now an explicit-stack pre-order traversal instead of a recursive generator. The generator passed every node up through one yield from frame per ancestor, which is O(nodes × depth), and the analysis walks the tree a few dozen times per request. Measured: on deeply nested input (40 lines of 300-term a + a + ..., 48 KB) explain() goes from 7.3 s to 5.4 s. On ordinary code the difference is within noise (0.331 s → 0.326 s on 195 KB of stdlib). I'm including it for the deep case and because it no longer depends on generator recursion depth.

Testing

• pytest on Python 3.13, 3.12 and 3.11: 154 passed, 3 skipped. ruff check explain_my_code tests is clean.
• Equivalence check: I broke stdlib modules at random (inserted a bad line, truncated mid-block, deleted a closing bracket, deleted a line) and compared the old and new salvage with ast.dump on the result. That was 477 broken files on 3.13, 515 on 3.12 and 526 on 3.11. Every one matched. The old salvage took 45–62 s per run and the new one 0.9–1.4 s.
• New tests:
  • test_salvaging_a_large_broken_file_does_not_reparse_it_line_by_line counts ast.parse calls (at most 3 for a 6,000-line file broken on line 1; it fails on main). It counts calls rather than timing them, so it can't flake.
  • test_broken_python_keeps_everything_above_the_error covers 5 error shapes, including an unterminated string and a missing body at EOF. It passes before and after, to pin the behaviour.
  • test_walk_is_pre_order checks the new walk against a recursive reference.

Notes

The remaining cost on deep input is ast.unparse. _repr() unparses a node's whole subtree and only then truncates it to 24–64 characters, so a long a + b + c + ... chain is quadratic. Fixing that properly means a length-bounded unparser, and the stdlib's unparser is private API that moved in 3.14 (ast._Unparser became _ast_unparse.Unparser). I left it out rather than build on it.

Merge note: #2 also touches _parse_longest_prefix, making it skip prefixes that raise MemoryError/RecursionError. Whichever PR lands second gets a small conflict there. To resolve it, keep this PR's loop and add except (MemoryError, RecursionError): end -= 1 after the SyntaxError branch.

https://claude.ai/code/session_01UigC7feaY8NkAy9oay3GDW

---

Generated by Claude Code