Milestone 3: lexer — scan.l port, base_yylex filter, Scan/SplitWithScanner/HashXXH3_64

[kyleconroy]

Lands milestone 3 of PLAN.md: the lexer, with its oracle-backed acceptance gate met in full — all 43,373 scan-suite cases and all 8 split_scanner cases pass byte-identically against the pinned-oracle goldens, and are harvested out of the metadata.json todo sidecars.

What's in here

internal/lexer — the scan.l port

A pull-based scanner (Scanner.Next, one token per call like core_yylex) covering all twelve flex start conditions: standard/extended/dollar-quoted/Unicode strings (with quote-continuation across newlines via the <xqs> lookahead), quoted identifiers, bit/hex strings, and nested C comments — plus the operator rule (trailing +/- stripping, embedded /*/-- truncation, "operator too long"), keyword lookup against the generated tables, and SQL_COMMENT/C_COMMENT tokens per libpg_query's patches. Every production carries an attribution comment naming its scan.l rule.

The fourteen overlapping numeric-literal rules (decinteger … real_junk) interact purely through flex's longest-match/rule-order discipline, including backtracking inside the *_junk rules (e.g. 1_2$ is trailing junk via the 1 + _2$ split); numbers.go reproduces this by computing every candidate rule's match length and picking the winner.

Scanner errors carry the reference's exact messages, scan.l filename, scanner_yyerror/core_yylex attribution, and character-based (not byte) cursor positions (pg_mbstrlen_with_len semantics); the "at or near" text runs to the end of the current match, mirroring flex's hold-char NUL.

base_yylex filter (internal/lexer/filter.go)

The token-merge layer from parser.c: NOT_LA/NULLS_LA/WITH_LA/WITHOUT_LA/FORMAT_LA merges, comment-token dropping, and UIDENT/USCONST + UESCAPE resolution with a full str_udeescape port (surrogate pairs, escape-character validation, exact error cursors). Not exercised by this milestone's corpus (the parser consumes it from milestone 4), so it ships with unit tests — including one pinning a real pinned-oracle quirk verified against pg_query_go v6.2.2: a comment between a merge pair blocks the merge (the lookahead is a raw core_yylex call), so SELECT 1 WHERE 1 NOT /* c */ IN (2) is a syntax error in libpg_query where vanilla PostgreSQL accepts it.

internal/xxh3

Pure-Go XXH3_64bits_withSeed (scalar paths, the only entry point the API needs), verified against pg_query_go's three published vectors plus 126 vectors generated from the pinned cgo oracle covering every length class (0 / 1–3 / 4–8 / 9–16 / 17–128 / 129–240 / hashLong incl. multi-block) with zero, small, and full-64-bit seeds.

Public API wiring

parser.ScanToProtobuf (raw core-scanner tokens, Version 170007, End offsets per pg_query_scan.c's yyllocend rules), parser.SplitWithScanner (pg_query_split.c semantics: top-level ; terminators, keyword-bearing statements only, paren tracking), and parser.HashXXH3_64. Their not-implemented stubs and test guards are retired; the ported pg_query_go TestScan smoke test now runs, and new compat tests pin a full token stream, split behavior, and the published hash vectors.

Validation

• go test ./... green, including a -race run over the corpus
• Scan + split_scanner suites at zero todos; the sidecar diff (−44k todo entries) is the harvest from go test ./parser -run TestCorpus -check-parse
• No goldens edited by hand; PLAN.md milestone 3 marked landed with as-built notes

cmd/next-test now points at the parse suite — milestone 4 (expressions + SELECT) is next.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AJQ7Jk1QEPsT5r7sFcfzEx

---

Generated by Claude Code