Replace docparse with a tree-sitter based parser

[rumblefrog]

Why

The upstream SourcePawn docparse tool that libalternator bound to over FFI is deprecated. It is also failing in practice:

• It can't parse several current SourceMod includes: handles.inc, float.inc, entitylump.inc and tf2_stocks.inc. Their pages on sourcemod.dev are stale.
• Its Rust post-processing panics on current sourcemod.inc (typedef Address = int64 in parse_type_signature), so the core bundle can't be regenerated at all.
• String defaults with escapes produced invalid JSON.
• Building it needs a SourcePawn checkout, AMBuild and bindgen/libclang, and that no longer works with current compilers without patching.

What

libalternator now parses includes in Rust. The public API (alternator::consume) is unchanged, so chumbucket needs no changes.

1. Preprocessor (preprocess.rs): evaluates #if/#elseif/#else/#endif (with defined, macro values and arithmetic), records #defines and honours #endinput. Inactive code, directive lines and function bodies are blanked with spaces, so byte offsets and line numbers stay identical to the source.
2. Parsing: tree-sitter-sourcepawn parses the blanked source.
3. Extraction (extract.rs): natives, forwards and stocks; methodmaps (including properties, constructors, destructors and aliases); enums and constants; enum structs and legacy structs; typedefs and typesets (including legacy functag/funcenum); and defines.
4. Comment attachment (comments.rs): comments attach to declarations using the same front/tail rules as the SourcePawn lexer docparse relied on, quirks included.
5. Rendering (expr.rs): types, defaults, dimensions and enum values are rendered in docparse's exact format. That means int& vs Handle &h, 1 << 0 with parentheses dropped, decimal integers, %f floats, and same-file macro expansion. Existing bundles therefore don't churn.
6. Error policy: syntax errors inside declarations fail the file, as before, so walking history never mistakes a failed parse for deleted symbols. Errors in function bodies and in legacy globals are tolerated.

Doc comments are still parsed by spdcp, now 0.5.0, which includes the fixes from rumblefrog/sp-dcp#1. parse_type_signature now returns Option instead of panicking.

Tests

• libalternator/tests/corpus: 221 upstream .inc/.sp files from SourceMod, sm-json, ripext, SourceBans++, TTT, the IncludeLibrary collection and others. They're pinned and fetched by tests/update-corpus.sh, with licenses listed in SOURCES.md. Three sources from the manifests are left out because they have no license.
• tests/corpus.rs: snapshot test against tests/expected/*.json. Update the snapshots with ALTERNATOR_BLESS=1.
• tests/extract.rs and unit tests cover the preprocessor, comment attachment, expressions and each declaration kind.

Parity with docparse

I built the old docparse locally and compared outputs on the 189 corpus files it (and its post-processing) could handle. Every symbol, kind, type, decl, default, value and comment location matches. The only differences are intentional doc comment fixes:

• trailing newline after **/: 34 tags
• cut-off character before */, e.g. /** Ungag*/ became Unga: 6
• block comments lost after a // Section label: 3 symbols
• @ param typos in cURL.inc: previously dropped entirely

Files the old pipeline couldn't handle now parse. Of the 221 files, only cURL_header.inc still fails: it uses function-like macros inside an enum, which docparse couldn't handle either. That failure is recorded as a snapshot.

End to end

I ran chumbucket generate to continue the published core.bundle from its recorded commit to current SourceMod master, which took 7 seconds. Every change traces to a real upstream edit:

• 48 symbols added, e.g. the Handle methodmap and WriteFileAsync
• 5 symbols removed upstream, e.g. IsValidHandle
• updated docs, including new @notes such as on OnMapStart

CI

• The SourcePawn/AMBuild job is gone. CI now runs cargo test -p alternator -p schema and builds chumbucket on Linux and Windows, on current action versions.
• tokio is updated within 1.x (1.15 → 1.53). The Windows build failed compiling ntapi 0.3.6, pulled in by mio 0.7, which current Rust rejects. mio 1.x no longer uses it.
• edge_worker is no longer built: its pinned wasm-bindgen doesn't compile on current Rust, and that is unrelated to this change.

#29 (@note)

• The published bundle already contains the @note for CloseHandle (116 of the 119 notes in SourceMod). The notes are missing from the site because the frontend only renders brief, param:*, return and error. That is fixed in sourcemod-dev/sourcemod-dev#20, which is merged.
• On the parser side, a @note on its own line (text on the following lines) used to be dropped and its text merged into the previous tag. That is fixed in spdcp 0.5.
• Files docparse failed on, which have many notes, are parsed again.

Fixes #29

🤖 Generated with Claude Code

https://claude.ai/code/session_011dNCUV71HWmyb9MwBD4xhC