Milestones 10 and 11: Summary and PL/pgSQL — every entry point implemented, corpus at 100%

.github/workflows/ci.yml

@@ -39,5 +39,5 @@ jobs:
           go-version-file: go.mod
       - name: cmd/generate output is committed and current
         run: |
-          go run ./cmd/generate -aliases -keywords
+          go run ./cmd/generate -aliases -keywords -plpgsql
           git diff --exit-code

CLAUDE.md

@@ -23,7 +23,8 @@ the main module; the only module allowed to link pg_query_go (cgo) is
 - **Expected outputs are never edited by hand.** Goldens come from
   `cmd/regenerate` (the pinned oracle) or not at all.
 - Generated files (`ast/pg_query.pb.go`, `aliases.go`,
-  `internal/lexer/keywords.go`) are only ever written by `cmd/generate`.
+  `internal/lexer/keywords.go`, `internal/plpgsql/tables.go`) are only ever
+  written by `cmd/generate`.
 - The public API mirrors pg_query_go exactly; consumers must migrate by
   changing one import path. Never rename, add, or remove exported symbols
   without checking pg_query_go v6.2.2.

PLAN.md

@@ -417,12 +417,19 @@ gate; the corpus harness exists before the first line of the lexer.
    reference's parse → deparse → reparse roundtrip criterion, since the
    goldens are the oracle's own deparse output). See "As-built notes
    (milestones 8–9)".
-10. **Summary.** Port the summary API (classification + smart truncation).
-11. **PL/pgSQL.** Port `pl_gram.y` + `pl_comp` subset + the JSON serializer
-    behind `ParsePlPgSqlToJSON`. Deliberately last: self-contained, and the
-    only piece whose deferral wouldn't block sqlc. If v1.0 ships without it,
-    the function returns a clear unimplemented error — flagged for an
-    explicit scope call rather than silently dropped.
+10. **Summary.** ✅ *Landed 2026-08-17.* Port the summary API
+    (classification + smart truncation). **Gate:** the deferred summary
+    corpus extracted and green — met: all 43,992 summary and 43,345
+    summary_truncate cases pass (the regress tiers untruncated and at limit
+    100, plus libpg_query's summary test-call inputs at their upstream
+    limits). See "As-built notes (milestones 10–11)".
+11. **PL/pgSQL.** ✅ *Landed 2026-08-17.* Port `pl_gram.y` + `pl_comp`
+    subset + the JSON serializer behind `ParsePlPgSqlToJSON`. **Gate:** the
+    deferred `plpgsql_regress/` + `plpgsql_samples` corpus green — met: all
+    690 plpgsql cases pass, and the plpgsql_regress files also feed the
+    standard SQL suites (all passing on arrival). Every corpus suite's todo
+    list is empty: 2,067 files / 308,561 cases across eleven suites. See
+    "As-built notes (milestones 10–11)".
 12. **Hardening.** `cmd/difftest` mutation fuzzing vs the live oracle as a
     scheduled CI job; `go test -race` over the parallel corpus run;
     benchmarks vs pg_query_go (cgo) and wasilibs (wasm) — expect wins from
@@ -460,7 +467,10 @@ Measured at the pin, where the plan's estimates differ:
 - Deferred, deliberately: the **sqlc endtoend tier** (needs per-directory
   engine classification in the sqlc repo; import it alongside milestone 4),
   **`plpgsql_regress/`** (milestone 11), and **summary** golden extraction
-  (milestone 10). The stretch tarball tier remains stretch.
+  (milestone 10). The stretch tarball tier remains stretch. *(The plpgsql
+  and summary tiers landed with milestones 10–11: the corpus is now 2,067
+  files / 308,561 cases across eleven suites; only the sqlc endtoend and
+  stretch tiers remain out.)*
 
 ### As-built notes (milestone 3)
 
@@ -648,6 +658,70 @@ Measured at the pin, where the plan's estimates differ:
   unimplemented entry points are `Summary` (milestone 10) and
   `ParsePlPgSqlToJSON` (milestone 11).
 
+### As-built notes (milestones 10–11)
+
+- The deferred corpus tiers landed first, as their own commit: `summary`
+  (regress + plpgsql_regress + libpg_query's summary test-call inputs,
+  untruncated), `summary_truncate` (the same regress statements at limit
+  100 — the limit rides along as a `-- truncate_limit: N` directive line in
+  the case input, stripped by both sides — plus the upstream truncate tests
+  at their exact limits and every summary input at limit 50), and `plpgsql`
+  (`plpgsql_regress/` + `plpgsql_samples.sql`, one `ParsePlPgSqlToJSON`
+  golden per statement). libpg_query's summary tests are call sites, not
+  `tests[]` tables, so `cmd/regenerate` grew a call extractor
+  (`summary("...", 0, limit)` with C line splices removed); inputs are
+  transcribed mechanically, expectations always from the oracle. The
+  plpgsql_regress files also feed the five standard SQL suites — all of
+  those passed the existing implementation on arrival.
+- Summary (`internal/summary`) is three ports over one shared
+  `raw_expression_tree_walker` (now `internal/rawwalk`, PostgreSQL 17's
+  nodeFuncs.c gated by `pg_query_raw_tree_walker_supports`): the
+  table/alias/CTE/function/filter-column walk with its quirks preserved
+  (WHERE clauses walked twice, so functions there are recorded once per
+  pass; the filter-column pass aborts at sub-SELECTs; `MERGE ... USING`
+  sources are never tables), the statement-type walk (insertion-ordered
+  set), and deparse-driven truncation. Truncation ordering needed the exact
+  `pg_qsort` algorithm again — `list_sort` maps to `pg_qsort` at the pin,
+  and equal (depth, length) pairs are ordered by the partition scheme, not
+  the comparator. List nodes count as depth levels, matching `WALK` on a
+  List field. The `"…"` dummy-node replacements and the multibyte fallback
+  chop (`pg_mbcharcliplen` over UTF-8) are byte-exact ports.
+- PL/pgSQL (`internal/plpgsql`) ports what libpg_query *builds*, not
+  vanilla PostgreSQL: `extract_source.rb` mocks `parse_datatype` (type text
+  kept verbatim; only RECORD/REFCURSOR/CURSOR/TEXT recognized, by
+  length-limited case-insensitive prefix), stubs the `%TYPE`/`%ROWTYPE`
+  lookups to NULL (so `foo%rowtype` falls through to the verbatim-text
+  path), replaces `make_return_stmt` (no return-type checks; RETURN never
+  captures a varno), and fails `function_parse_error_transpose`
+  unconditionally — so compile errors carry `compilation of PL/pgSQL
+  function "f" near line N` context and no cursor, with filenames as the
+  vendored build's `__FILE__` (`pl_gram.y`,
+  `src_pl_plpgsql_src_pl_scanner.c`, ...).
+- The pl scanner layers on the milestone-3 lexer: the core scanner runs
+  with the PL reserved keyword list, modeled by re-classifying the SQL
+  lexer's output (SQL keywords become plain identifiers unless
+  PL-reserved; the `n'...'` NCHAR special falls back to IDENT `"n"`), plus
+  the pushback stack, `A.B.C` datum composition, and the
+  statement-start/unreserved-keyword dance. `pl_gram.y` became recursive
+  descent with statement linenos computed at bison's reduce points —
+  `plpgsql_latest_lineno` is stateful and feeds the error context, so call
+  order is part of conformance.
+- `check_sql_expr` needed the core grammar's PL/pgSQL raw-parse modes:
+  `parse.ParseWithMode` adds `RAW_PARSE_PLPGSQL_EXPR` (the `PLpgSQL_Expr`
+  production — everything that can follow SELECT, minus SELECT) and
+  `ASSIGN1..3` (`PLAssignStmt`). Embedded-SQL errors keep the core
+  message/filename/funcname but lose their cursor
+  (`plpgsql_sql_error_callback` flushes the external position).
+- `cmd/generate -plpgsql` emits `internal/plpgsql/tables.go` from the newly
+  vendored `pl_reserved_kwlist.h`/`pl_unreserved_kwlist.h`/`plerrcodes.h`
+  (condition names with their duplicate counts — duplicate names produce
+  condition chains of that length).
+- Corpus effect: all 43,992 summary, 43,345 summary_truncate, and 690
+  plpgsql todos graduated; every suite's todo list is empty (2,067 files /
+  308,561 cases). pg_query_go's `summary_test.go` is ported verbatim and
+  `TestParsePlPgSQL` now runs green; no entry point returns
+  not-implemented anymore.
+
 ## Regeneration (the PostgreSQL-upgrade story)
 
 Everything derived is derived by committed tooling from the pin:

README.md

@@ -22,13 +22,15 @@ for the development loop.
 | Milestone | State |
 |---|---|
 | 1. Scaffolding + codegen | ✅ module, licenses, CI, vendored pin, `ast/` protobuf types, aliases, keyword tables, API surface, `parser.Error`; pg_query_go's parse_test tree literals compile unchanged |
-| 2. Corpus + oracle | ✅ cgo oracle (`oracle/`), `cmd/regenerate`, 1,525 golden files / 217k cases across 8 suites, byte-reproducible; harness + `cmd/next-test` running everything as todo |
-| 3. Lexer | — |
-| 4. Expressions + SELECT | — |
-| 5–7. DML, DDL, utility | — |
-| 8. Normalize + fingerprint | — |
-| 9. Deparse | — |
-| 10–13. Summary, PL/pgSQL, hardening, sqlc integration | — |
+| 2. Corpus + oracle | ✅ cgo oracle (`oracle/`), `cmd/regenerate`, byte-reproducible goldens; harness + `cmd/next-test` |
+| 3. Lexer | ✅ `Scan`, `SplitWithScanner`, `HashXXH3_64`; token streams byte-identical across the corpus |
+| 4. Expressions + SELECT | ✅ `Parse`/`ParseToJSON` for the SELECT family |
+| 5–7. DML, DDL, utility | ✅ the full statement grammar; `SplitWithParser`, `IsUtilityStmt`; parse corpus at 100% |
+| 8. Normalize + fingerprint | ✅ `Normalize`, `NormalizeUtility`, `Fingerprint` |
+| 9. Deparse | ✅ `Deparse`, byte-equal to the oracle across the corpus |
+| 10. Summary | ✅ `Summary` (classification + smart truncation) |
+| 11. PL/pgSQL | ✅ `ParsePlPgSqlToJSON` (pl_gram + pl_comp subset + JSON dump) |
+| 12–13. Hardening, sqlc integration | — |
 
-Until the parser milestones land, every public entry point returns (or
-panics with, for `HashXXH3_64`) a clear not-implemented error.
+Every public entry point is implemented; the corpus — 2,067 golden files /
+308,561 cases across eleven suites — passes with an empty todo list.

cmd/difftodo/main.go

@@ -23,7 +23,8 @@ func main() {
 	}
 	suite := "parse"
 	for _, s := range []string{"parse", "scan", "normalize", "normalize_utility",
-		"fingerprint", "deparse", "split_scanner", "split_parser"} {
+		"fingerprint", "deparse", "split_scanner", "split_parser",
+		"summary", "summary_truncate", "plpgsql"} {
 		if strings.Contains(path, "/"+s+"/") {
 			suite = s
 		}
@@ -76,6 +77,42 @@ func evaluate(suite, input string) string {
 		if err == nil {
 			out, err = pg_query.Deparse(tree)
 		}
+	case "summary", "summary_truncate":
+		limit, sql := -1, input
+		if suite == "summary_truncate" {
+			limit, sql, err = testfile.SplitTruncateLimit(input)
+			if err != nil {
+				return "BAD CASE: " + err.Error()
+			}
+		}
+		var res *pg_query.SummaryResult
+		res, err = pg_query.Summary(sql, limit)
+		if err == nil {
+			e := testfile.SummaryExpectation{
+				Aliases:        res.Aliases,
+				CteNames:       res.CteNames,
+				StatementTypes: res.StatementTypes,
+				TruncatedQuery: res.TruncatedQuery,
+			}
+			for _, t := range res.Tables {
+				e.Tables = append(e.Tables, testfile.SummaryTable{
+					Name: t.Name, SchemaName: t.SchemaName, TableName: t.TableName, Context: t.Context.String(),
+				})
+			}
+			for _, f := range res.Functions {
+				e.Functions = append(e.Functions, testfile.SummaryFunction{
+					Name: f.Name, FunctionName: f.FunctionName, SchemaName: f.SchemaName, Context: f.Context.String(),
+				})
+			}
+			for _, f := range res.FilterColumns {
+				e.FilterColumns = append(e.FilterColumns, testfile.SummaryFilterColumn{
+					SchemaName: f.SchemaName, TableName: f.TableName, Column: f.Column,
+				})
+			}
+			out = testfile.RenderSummary(e)
+		}
+	case "plpgsql":
+		out, err = pg_query.ParsePlPgSqlToJSON(input)
 	default:
 		panic("difftodo: unsupported suite " + suite)
 	}

cmd/generate/main.go

@@ -21,10 +21,11 @@ func main() {
 	proto := flag.Bool("proto", false, "regenerate ast/pg_query.pb.go (needs protoc + protoc-gen-go)")
 	aliases := flag.Bool("aliases", false, "regenerate aliases.go")
 	keywords := flag.Bool("keywords", false, "regenerate internal/lexer/keywords.go")
+	plpgsql := flag.Bool("plpgsql", false, "regenerate internal/plpgsql/tables.go")
 	flag.Parse()
 
-	if !*proto && !*aliases && !*keywords {
-		*aliases, *keywords = true, true
+	if !*proto && !*aliases && !*keywords && !*plpgsql {
+		*aliases, *keywords, *plpgsql = true, true, true
 	}
 
 	if *proto {
@@ -47,6 +48,12 @@ func main() {
 		}
 		fmt.Println("wrote internal/lexer/keywords.go")
 	}
+	if *plpgsql {
+		if err := generatePlpgsql("internal/reference", "internal/plpgsql/tables.go"); err != nil {
+			fatal(err)
+		}
+		fmt.Println("wrote internal/plpgsql/tables.go")
+	}
 }
 
 func run(name string, args ...string) {

cmd/generate/plpgsql.go

@@ -0,0 +1,97 @@
+package main
+
+import (
+	"fmt"
+	"go/format"
+	"os"
+	"regexp"
+	"sort"
+	"strings"
+)
+
+// generatePlpgsql emits internal/plpgsql/tables.go from the pinned
+// pl_reserved_kwlist.h / pl_unreserved_kwlist.h (keyword → token maps, the
+// data pl_scanner.c compiles in) and plerrcodes.h (the exception condition
+// name table pl_comp.c's exception_label_map includes; only the name and its
+// number of entries matter — dump_condition serializes condname alone, and
+// duplicate names produce chains of that length).
+func generatePlpgsql(refDir, outPath string) error {
+	kwRe := regexp.MustCompile(`PG_KEYWORD\("([a-z_]+)",\s*(K_[A-Z_]+)\)`)
+	readKeywords := func(file string) ([][2]string, error) {
+		src, err := os.ReadFile(refDir + "/" + file)
+		if err != nil {
+			return nil, err
+		}
+		var out [][2]string
+		for _, m := range kwRe.FindAllStringSubmatch(string(src), -1) {
+			out = append(out, [2]string{m[1], m[2]})
+		}
+		if len(out) == 0 {
+			return nil, fmt.Errorf("%s: no PG_KEYWORD entries found", file)
+		}
+		return out, nil
+	}
+
+	reserved, err := readKeywords("pl_reserved_kwlist.h")
+	if err != nil {
+		return err
+	}
+	unreserved, err := readKeywords("pl_unreserved_kwlist.h")
+	if err != nil {
+		return err
+	}
+
+	errSrc, err := os.ReadFile(refDir + "/plerrcodes.h")
+	if err != nil {
+		return err
+	}
+	condRe := regexp.MustCompile(`"([a-z_]+)",\s*ERRCODE_[A-Z_0-9]+`)
+	condCount := map[string]int{}
+	var condOrder []string
+	for _, m := range condRe.FindAllStringSubmatch(string(errSrc), -1) {
+		if condCount[m[1]] == 0 {
+			condOrder = append(condOrder, m[1])
+		}
+		condCount[m[1]]++
+	}
+	if len(condOrder) == 0 {
+		return fmt.Errorf("plerrcodes.h: no condition entries found")
+	}
+	sort.Strings(condOrder)
+
+	var b strings.Builder
+	b.WriteString("// Code generated by cmd/generate -plpgsql from internal/reference/\n")
+	b.WriteString("// pl_reserved_kwlist.h, pl_unreserved_kwlist.h, and plerrcodes.h\n")
+	b.WriteString("// (libpg_query 17-6.2.2). DO NOT EDIT.\n\npackage plpgsql\n\n")
+
+	b.WriteString("// reservedKeywords is pl_reserved_kwlist.h: these are passed to the core\n")
+	b.WriteString("// scanner's keyword list, so they win over any variable name.\n")
+	b.WriteString("var reservedKeywords = map[string]plToken{\n")
+	for _, kw := range reserved {
+		fmt.Fprintf(&b, "\t%q: %s,\n", kw[0], kw[1])
+	}
+	b.WriteString("}\n\n")
+
+	b.WriteString("// unreservedKeywords is pl_unreserved_kwlist.h: checked only where the\n")
+	b.WriteString("// grammar wants a keyword and no variable matched.\n")
+	b.WriteString("var unreservedKeywords = map[string]plToken{\n")
+	for _, kw := range unreserved {
+		fmt.Fprintf(&b, "\t%q: %s,\n", kw[0], kw[1])
+	}
+	b.WriteString("}\n\n")
+
+	b.WriteString("// errConditions maps exception condition names to their number of\n")
+	b.WriteString("// entries in exception_label_map (plpgsql_parse_err_condition returns a\n")
+	b.WriteString("// chain with one node per entry).\n")
+	b.WriteString("var errConditions = map[string]int{\n")
+	for _, name := range condOrder {
+		fmt.Fprintf(&b, "\t%q: %d,\n", name, condCount[name])
+	}
+	b.WriteString("}\n")
+
+	src, err := format.Source([]byte(b.String()))
+	if err != nil {
+		return fmt.Errorf("gofmt plpgsql tables: %w", err)
+	}
+	return os.WriteFile(outPath, src, 0o644)
+}

cmd/next-test/main.go

@@ -22,6 +22,7 @@ import (
 var suiteOrder = []string{
 	"scan", "split_scanner", "parse", "deparse",
 	"normalize", "normalize_utility", "fingerprint", "split_parser",
+	"summary", "summary_truncate", "plpgsql",
 }
 
 func main() {